How can we show that weight decay produces a positive effect on the performance of a neural network? Specifically, a network that classifies images? One way would be to add noise to the images in the dataset, and compare the performance of training with weight decay and without. What kind of noise would be best for images? Could I also occlude part of the image? Or select a subset of the dataset that represents non standard images (many extraneous details) that would hurt the performance but could be improved by using weight decay?
How can we show that weight decay produces a positive effect on the performance of a neural network?
Train a network with weight decay, and the same network without weight decay and measure the difference between the two methods.
The alternatives that you propose are not testing weight decay directly, they're testing the effect of noise and weight decay.