After reading a lot of deep learning papers, a kind of rough feeling is that there exist a lot of tricks in training the network to get the better-than-normal performance. From an industry application perspective, it is very hard to develop this kind of tricks except those elite research groups in big tech companies, e.g., google or facebook. Then what is the best way to apply deep learning algorithms in practice. Any thought and suggestions will be highly appreciated.
True, some details used for improving performance are considered as tricks and you won't always know if these tricks yield the same improvement for your data and your network.
Some things that you will definitely need:
- Data, lots of it
- GPUs will let you run experiments faster and try out more things in a shorter time span.
- Learning curve analysis. In the end it comes down to performance on the test set, but looking at the both train and test metrics you can identify reasons for bad performance. Strong bias? Overfitting from too many hidden nodes?
- The activation function. I don't think it counts as a trick to know which kind of activation function you need. ReLU have a critical charactersitic in that they do not saturate like sigmoids and tanh. A neuron with ReLU will longer have probability-like output, but you don't need this for neurons in mid-level layers anyway. The advantag you get is mitigating the vanishing or exploding of gradients and speed up convergence.
- Regularization. Might apply as tricks, but if you're using any of the mainstream deep learning libraries you can get off-the-shelf implementations for regularization via dropout.
- Data augmentation. You're basically expanding your dataset synthetically without the added cost of manual annotation. The key is to augment the data with transformations that actuall make sense. So that the network gets to see variants of the data it may encounter in the test phase or when it gets deployed into the product. For visual data it horizontal flipping is trivial and adds a lot of gain. Jitter is probably dependent on the type of data and how noisy it is.
- Diving into hyperparameter exploration can be frustrating. Start off with small networks and simple training procedures. Smaller networks are faster to train. Add more layers when you see signs of overfitting.
- Good initialization. Random intitialization are appropriate for gauging the network's ability to converge but will not necessarily lead t optimal performance. At the same time, just keeping on iterating might lead to the network overfitting to the training data. If possible use a pre-trained network that has already learned a representation and fine tune it to your dataset. Unsupervised pre-training is another way to go and can allow the supervised training procedure to start from a far more promising position in weight space.
- Scrutinize tricks. Understand what the trick really does. A paper describing a small detail that was used in improving the performance of a network will focus on that new aspect. The paper could be part of a sequence of projects that the authors have been working on. The context of the trick may not always be clear right away but for the authors it's not a trick but a technique that solves a problem they had. Sometimes a technique comes out and is treated as a trick and later someone will analyze its impact and describe its function. For example that this trick is equivalent to L2 regularization which more people are familiar with. We can the decide if we should try out this new technique or stick with the L2 regularization that we already know about. A lot of these tricks try to solve problems in deep learning, like risk of overfitting, costly computations, over parameterization and highly redundant weights. It's worth taking the time to understand what these tricks really do. By understanding the problem they try to solve we can judge the applicability of different tricks and pick the one that works well with constraints we may have (e.g. little computing power, small dataset)
Here is an interesting book Neural Networks: Tricks of the Trade, an updated 2012 version of the book. Lots of articles by some of the pioneers of neural networks.
ypx beautifully touched on a lot of practical issues with training, so to touch upon the other issues you raised: a lot of the elite industrial labs still publish their results. For example Microsoft Research's team just won ImageNet 2015 and they released a technical report describing their new deep net module: Deep Residual Learning for Image Recognition, Google's team published their Inception architecture as well, Going Deeper with Convolutions. To a non-trivial degree there is still a culture in machine learning (for now) of sharing the big innovations. Possibly because the key is access to the data. Google and Facebook simply have access to data that we do not. Hard to say how much credit goes to raw algorithmic innovation and how much goes to massive amounts of data.
With regards to what will happen in the future? Hard to say. It's an issue that lots of people have raised given how valuable these data driven companies have become and how competitive the market is. But for now, I think there is a good enough balance of what industrial research labs share and don't share. I understand them not sharing their exact code implementation. But they do share some very novel innovations.
Find researchers who publish important results and read, read, read. I believe in Yann LeCun's AMA on Reddit he mentioned that he is a voracious reader. I believe this is the most important thing. And to the extent that it is practical, try to recreate their benchmarks, or apply their method to a dataset that is within your budget.
I think regardless of where you are or what your station in life is, this is the best way to stay sharp and continue to develop your skills. Be a voracious reader and implement things and build intuition. I personally do not have the resources to participate in ImageNet competitions, but reading all the top performing ImageNet group's articles has helped me tremendously.