The relative pitch of harmonic complex sounds, such as instrumental sounds, may be perceived by decoding either the fundamental pitch (f0) or the spectral pitch (fSP) of the stimuli. We classified a large cohort of 420 subjects including symphony orchestra musicians to be either f0 or fSP listeners, depending on the dominant perceptual mode. In a subgroup of 87 subjects, MRI (magnetic resonance imaging) and magnetoencephalography studies demonstrated a strong neural basis for both types of pitch perception irrespective of musical aptitude. Compared with f0 listeners, fSP listeners possessed a pronounced rightward, rather than leftward, asymmetry of gray matter volume and P50m activity within the pitch-sensitive lateral Heschl's gyrus. Our data link relative hemispheric lateralization with perceptual stimulus properties, whereas the absolute size of the Heschl's gyrus depends on musical aptitude.