Acute tryptophan depletion (ATD), a well-recognized method to lower central serotonin levels, was used to examine the effects of lower central serotonin levels on memory function in healthy males. Functional Magnetic Resonance Imaging (fMRI) was used to examine changes in brain activation during the encoding and the retrieval phase of a visual verbal episodic memory task. ATD led to more positively rated words in the encoding phase and to poorer recognition of these positively rated words in the retrieval phase. Furthermore, encoding was accompanied by enhanced brain activation in occipital, middle and superior frontal, anterior and posterior cingulate and striatal areas. Retrieval attempt was accompanied by enhanced activation in the cuneus, inferior occipital gyrus and inferior and middle frontal areas. Retrieval success was accompanied by activation in an extensive network including frontal, parietal, temporal, cingulate, striatal and cerebellar areas. In the encoding phase ATD attenuated activation in the right hippocampus and ATD did not affect brain activity in the retrieval phase. These results show that serotonin is important in long term memory processes, and that serotonin acts on the encoding phase and not on the retrieval phase.