12

Consider a neural network model with 100 weights. If we think only about getting the sign of the weights right and don't worry for the moment about their magnitude. There are 2^100 combinations of the signs of these weights, which is a very large number. If we sample 60 random weight vectors, we will have seen only an minuscule proportion of that space, ...


9

One limitation of random search is that searching over a large space is extremely challenging; even a small difference can spoil the result. Émile Borel's 1913 article "Mécanique Statistique et Irréversibilité" stated if a million monkeys spent ten hours a day at a typewriter, it's extremely unlikely that the quality of their writing would equal a ...


4

Suppose we want to answer your question with a 1000 character answer. One approach could be to sample 60 1000-tuples of characters, punctuation marks, and whitespace. With 95% probability, one of these will be within the most useful 5% of all possible Stack Exchange answers within this character limit. Basically the problem as you point out is that being ...


Only top voted, non community-wiki answers of a minimum length are eligible