Eye-hand coordination is crucial for everyday visuo-haptic object-manipulation. Noninformative vision has been reported to improve haptic spatial tasks relying on world-based reference frames. The current study investigated whether the degree of visuo-haptic congruity systematically affects haptic task performance. Congruent and parametrically varied incongruent visual orientation cues were presented while participants manually explored the orientation of a reference bar stimulus. Participants were asked to haptically match this reference orientation by turning a test bar either to a parallel or mirrored orientation, depending on the instruction. While parallel matching can only be performed correctly in a world-based frame, mirror matching (in the mid-sagittal plane) can also be achieved in a body-centered frame. We revealed that visuo-haptic incongruence affected parallel but not mirror matching responses in size and direction. Parallel matching did not improve when congruent visual orientation cues were provided throughout a run, and mirror matching even deteriorated. These results show that there is no positive effect of visual input on haptic performance per se. Tasks, which favor a body-centered frame are immune to incongruent visual input, while such input parametrically modulates performance on world-based haptic tasks.