Objective: To investigate the influence of deep white matter hyperintensities (DWMH) and periventricular white matter hyperintensities (PVWMH) on progression of cognitive decline in non-demented elderly people. Methods: All data come from the nested MRI sub-study of the PROspective Study of Pravastatin in the Elderly at Risk (PROSPER). We performed a 3 year follow up study on 554 subjects of the PROSPER study using both repeated magnetic resonance imaging and cognitive testing. Cognitive decline and its dependency on WMH severity was assessed using linear regression models adjusted for sex, age, education, treatment group, and test version when applicable. Results: We found that the volume of PVWMH at baseline was longitudinally associated with reduced mental processing speed (p = 0.0075). In addition, we found that the progression in PVWMH volume paralleled the decline in mental processing speed (p = 0.024). In contrast, neither presence nor progression of DWMH was associated with change in performance on any of the cognitive tests. Conclusion: PVWMH should not be considered benign but probably underlie impairment in cognitive processing speed.