We study a noisy oscillator with pulse delayed feedback, theoretically and in an electronic experimental implementation. Without noise, this system has multiple stable periodic regimes. We consider two types of noise: (i) phase noise acting on the oscillator state variable and (ii) stochastic fluctuations of the coupling delay. For both types of stochastic perturbations the system hops between the deterministic regimes, but it shows dramatically different scaling properties for different types of noise. The robustness to conventional phase noise increases with coupling strength. However for stochastic variations in the coupling delay, the lifetimes decrease exponentially with the coupling strength. We provide an analytic explanation for these scaling properties in a linearized model. Our findings thus indicate that the robustness of a system to stochastic perturbations strongly depends on the nature of these perturbations.