In general, no disproportionate detrimental effects of irrelevant background speech on cognition are found in aging individuals, although this is predicted by the inhibitory view of aging. This may be due to the nature of the primary task (most studies involve a verbal learning task) or the cognitive level at which irrelevant speech interferes with this task. In this study, the irrelevant speech effect on a numeric working memory task was investigated among 20 young (M = 21.8 years) and 20 older (M = 68.1 years) native Dutch individuals. Level of interference (LOI) was manipulated by presenting white noise (no interference), Russian words (low interference), Dutch words (phonological interference), and Dutch numbers (semantic interference) in the background. Results showed that reaction time increases as a function of LOI relative to silence, whereas accuracy remains unaffected. However, no interaction between LOI and age group was found, which suggests that the elderly were not disproportionately affected by an increased level of interference. These results are discussed in the light of the inhibitory view of aging.