Are all shortcuts in encoder-decoder networks beneficial for CT denoising?

Denoising of CT scans has attracted the attention of many researchers in the medical image analysis domain. Encoder-decoder networks are deep learning neural networks that have become common for image denoising in recent years. Shortcuts between the encoder and decoder layers are crucial for some image-to-image translation tasks. However, are all shortcuts necessary for CT denoising? To answer this question, we set up two encoder-decoder networks representing two popular architectures and then progressively removed shortcuts from the networks from shallow to deep (forward removal) and from deep to shallow (backward removal). We used two unrelated datasets with different noise levels to test the denoising performance of these networks using two metrics, namely root mean square error and content loss. The results show that while more than half of the shortcuts are still indispensable for CT scan denoising, removing certain shortcuts leads to performance improvement for denoising. Both shallow and deep shortcuts might be removed, thus retaining sparse connections, especially when the noise level is high. Backward removal seems to have a better performance than forward removal, which means deep shortcuts have priority to be removed. Finally, we propose a hypothesis to explain this phenomenon and validate it in the experiments.