My grandma walks, but doesn't climb. Some grandmas do. One grandma was famous for climbing Kilimanjaro.
That dormant volcano is big. It is 16,000 feet above its base. (Don't hate my imperial units.) It also has glaciers on the top, sometimes.
If you climb on a year where there is no glacier, and you get to the top, is it the same top as if there was a glacier? The altitude is different. The path you have to take is different. What if you go to the top when the glacier thickness is larger? Does that make it more of an accomplishment? About 35,000 people attempt to climb it every year, but only about 16,000 succeed.
So I would explain the control of weights (aka minimizing model complexity) to my grandma, as follows:
Grandma, your brain is an amazing thinker whether or not you know it. If I ask you how many of the 16,000 who think they reached the top actually did so, you would say "all of them".
If I put sensors in shoes of all the 30,000 climbers, and measured height above sea-level, then some of those folks didn't get as high as others, and might not qualify. When I do that I am going to a constant model - I am saying if height is not equal to some percentile of measured max heights then it is not the top. Some people jump at the top. Some people just cross the line and sit down.
I could add latitude and longitude to the sensor, and fit some higher order equations and maybe I could get a better fit, and have more folks in, maybe even exactly 45% of the total folks who attempt it.
So let's say next year is a "big glacier" year or a "no glacier" year because some volcano really transforms the albedo of the earth. If I take my complex and exacting model from this year and apply it to the folks who climb next year the model is going to have strange results. Maybe everyone will "pass" or even be too high to pass. Maybe nobody at all will pass, and it will think nobody actually completed the climb. Especially when the model is complex it will tend to not generalize well. It may exactly fit this year's "training" data, but when new data comes it behaves poorly.
When you limit the complexity of the model, then you can usually have better generalization without over-fitting. Using simpler models, ones that are more built to accommodate real-world variation, tends to give better results, all else being equal.
Now you have a fixed network topology, so you are saying "my parameter count is fixed" - I can't have variation in model complexity. Nonsense. Measure the entropy in the weights. When the entropy is higher it means some coefficients carry substantially more "informativeness" than others. If you have very low entropy it means that in general the coefficients carry similar levels of "informativeness". Informativeness is not necessarily a good thing. In a democracy you want all people to be equal, and things like George Orwell "more equal than others" is a measure of failures of the system. If you don't have a great reason for it, you want weights to be pretty similar to each other.
On a personal note: instead of using voodoo or heuristics, I prefer things like "information criteria" because they allow me to get reliable and consistent results. AIC, AICc, and BIC are some common and useful starting points. Repeating the analysis to determine stability of the solution, or range of information criteria results is a common approach. One might look at putting a ceiling on the entropy in the weights.