High Levels of impulsivity have adverse effects on performance in cognitive tasks, particularly in those tasks that require high attention investment. Furthermore, both animal and human research has indicated that reduced brain serotonin (5-HT) function is associated with increases in impulsive behaviour or decreased inhibition ability, but the effects of 5-HT challenge have not yet been investigated in subjects vulnerable to impulsivity. The present study aimed to investigate whether subjects with high trait impulsivity perform worse than low impulsive subjects in a task switching paradigm in which they have to rapidly shift their attention between two response rules, and to investigate the influence of a 5-HT enhancing diet. Healthy subjects with high (n = 19) and low (n = 18) trait impulsivity scores participated in a double-blind placebo-controlled study. ALL subjects performed the attention switch task in the morning following breakfast containing either tryptophan-rich alpha-lactalbumin (4.8 g/100 g TRP) or placebo protein (1.4 g/100 g TRP). Whereas there were no baseline differences between high and tow impulsive subjects in task switching abilities, high impulsive subjects made significantly more switch errors and responded slower after dietary 5-HT stimulation, whereas no dietary effects were found on task switching performance in Low-impulsive subjects. The deterioration in task switching performance induced by the 5-HT enhancing diet in high impulsive subjects was suggested to be established by general arousal/attention-reducing effects of 5-HT, which might have a larger impact in high impulsive subjects due to either different brain circuitry involved in task switching in this group or tower baseline arousal levels.