2
$\begingroup$

Research work that have solved problems using neural networks, simply state the structure of their networks in their research papers. No explanation is generally given about what led them to that structure (for example, AlphaGo's neural networks).

It can't be trial and error, since with huge networks the search space is huge; one can't simply try a lot of options and get to the correct structure.

And, it can't be fully deterministic either; some trial and error is involved, but not much. I've been taught in college that experts on neural networks make educated guesses of what to try.

I want to know what those educated guesses are and how is it driven. There must be some base principle/intuition or some data analysis on the given dataset, that can help us make educated guesses of our desired structure.

Please answer with small examples if possible. An example could be on a small enough dataset to make your point, or some problem that you solved using neural network (and knew how to design the network), or some research material where the actual process is explained. If these principles can be explained using XOR (classification) data, then please do so on that example..

One general idea is that instead of random weights, initialize the network weights using some domain knowledge about the problem. How is this done specifically?

Any more such ideas?

Improve this question
$\endgroup$
4
  • $\begingroup$ This is a good question, but it is awfully broad. It isn't clear how well someone will be able to give you a comprehensive answer. You might consider editing this to make it more focused & concrete. Seeing how this might be done in a particular situation may help you understand how the process could be applied even in a different situation. $\endgroup$
    – gung - Reinstate Monica
    Commented Apr 13, 2016 at 15:16
  • $\begingroup$ @gung Yeah, its a pretty broad question. I think the intuition (for experts), comes from their personal experience of having worked on neural nets a lot. The best way to answer this question would be to quote examples, with a dataset. Maybe this question can be answered with something as simple as the XOR data, but I can't be sure if that is too narrow. Hence leaving it open for people to cite examples.. $\endgroup$
    – sanjeev mk
    Commented Apr 13, 2016 at 15:32
  • $\begingroup$ If people think it's too broad, this will likely be closed. You may want to narrow this & make it more concrete. You can also ask several related questions, each pertaining to a different situation, & link between them for context. $\endgroup$
    – gung - Reinstate Monica
    Commented Apr 13, 2016 at 15:34
  • $\begingroup$ I've incorporated a sub-question on weights initialization in the question now.. $\endgroup$
    – sanjeev mk
    Commented Apr 13, 2016 at 15:38

1 Answer 1

Reset to default
0
$\begingroup$

This is often referred to as hyperparameter tuning. How to initialize the weights, or what learning rate to use in order to produce the lowest error, are a couple of examples of hyperparameters: basically everything that doesn't involve the weights of the net itself.

Tuning of hyperparameters is usually done by grid search, or randomized search. Sure, domain knowledge is useful, but at the same time can induce bias.

If you need some insights, you can read this paper, which presents randomized hyperparameter search but also explains how hyperparameters are tuned with carefully crafted grid search: http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.