I'm following deeplearning.ai's videos on Coursera. In one of the videos, Prof Ng mentions:
So a couple of points to notice. First is that, depending on whether you have high bias or high variance, the set of things you should try could be quite different. So I'll usually use the training dev set to try to diagnose if you have a bias or variance problem, and then use that to select the appropriate subset of things to try. So for example, if you actually have a high bias problem, getting more training data is actually not going to help. Or at least it's not the most efficient thing to do. So being clear on how much of a bias problem or variance problem or both can help you focus on selecting the most useful things to try. Second, in the earlier era of machine learning, there used to be a lot of discussion on what is called the bias variance tradeoff. And the reason for that was that, for a lot of the things you could try, you could increase bias and reduce variance, or reduce bias and increase variance. But back in the pre-deep learning era, we didn't have many tools, we didn't have as many tools that just reduce bias or that just reduce variance without hurting the other one. But in the modern deep learning, big data era, so long as you can keep training a bigger network, and so long as you can keep getting more data, which isn't always the case for either of these, but if that's the case, then getting a bigger network almost always just reduces your bias without necessarily hurting your variance, so long as you regularize appropriately. And getting more data pretty much always reduces your variance and doesn't hurt your bias much. So what's really happened is that, with these two steps, the ability to train, pick a network, or get more data, we now have tools to drive down bias and just drive down bias, or drive down variance and just drive down variance, without really hurting the other thing that much. And I think this has been one of the big reasons that deep learning has been so useful for supervised learning, that there's much less of this tradeoff where you have to carefully balance bias and variance, but sometimes you just have more options for reducing bias or reducing variance without necessarily increasing the other one. And, in fact, [inaudible] you have a well regularized network. We'll talk about regularization starting from the next video. Training a bigger network almost never hurts. And the main cost of training a neural network that's too big is just computational time, so long as you're regularizing.
I don't understand this at all. My understanding is if one increases, the other should definitely decrease. Particularly:
I understand training a bigger network ensures learning more complex features and hence, reduces bias. But won't a more complex function have higher variance? Why should the variance remain the same?
I understand that getting more data reduces a smaller model's ability to capture the distribution and hence more data leads to high bias and must hence lead to lower variance. Why should the bias remain the same?
I may be looking into it too much. I am guessing he actually means that the tradeoff still exists, but since we have more access to data and computation, this tradeoff is less of a concern and either one can be corrected more easily. Is my understanding correct? Thanks in advance.
