The evidence for an association between occupational asbestos exposure and esophageal, gastric and colorectal cancer is limited. We studied this association specifically addressing risk differences between relatively low and high exposure, risk associated with cancer subtypes, the influence of potential confounders and the interaction between asbestos and smoking in relation to cancer risk. Using the Netherlands Cohort Study (n = 58,279 men, aged 55-69 years at baseline), asbestos exposure was estimated by linkage to a job-exposure matrix. After 17.3 years of follow-up, 187 esophageal, 486 gastric and 1,724 colorectal cancer cases were available for analysis. The models adjusted for age and family history of cancer showed that mainly (prolonged) exposure to high levels of asbestos was statistically significantly associated with risk of esophageal adenocarcinoma (EAC), total and distal colon cancer and rectal cancer. For overall gastric cancer and gastric non-cardia adenocarcinoma (GNCA), also exposure to lower levels of asbestos was associated. Additional adjustment for lifestyle confounders, especially smoking status, yielded non-significant associations with overall gastric cancer and GNCA in the multivariable-adjusted model, except for the prolonged highly exposed subjects (tertile 3 vs. never: HR 2.67, 95% Cl: 1.11-6.44 and HR 3.35, 95% Cl: 1.33-8.44, respectively). No statistically significant additive or multiplicative interaction between asbestos and smoking was observed for any of the studied cancers. This prospective population-based study showed that (prolonged) high asbestos exposure was associated with overall gastric cancer, EAC, GNCA, total and distal colon cancer and rectal cancer.