I'm a newbie and learning ML. I've a doubt, normally we know we should increase the size of training dataset or should add more data to reduce variance (fairly understood why). Now variance has inverse relationship with bias, so it means when we're adding more data, we're reducing variance - or we're increasing bias. Then, why this is not possible to reduce bias by reducing the number of training samples. Could someone please explain me.
1
-
4$\begingroup$ Bias-variance "tradeoff" is a misnomer. When we add bias in order to reduce the variance, we hope that doing so decreases the mean squared error, not that it maintains the MSE. Let's say that we have a model with no bias and a variance of $1$. Then $MSE = 1$. If we decide to use a biased model that has a bias of $0.5$, we want to variance to drop below $0.75$ in order to have $0.5^2 + \text{variance} < 1$. It could be, however, that we only drop our variance to $0.8$, meaning that our MSE is $0.5^2 + 0.8 = 1.05$, which means that we are doing worse in terms of mean squared error. $\endgroup$– DaveCommented Jun 30, 2022 at 13:18
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
1
Now variance has an inverse relationship with bias
Not necessary. A picture is worth a thousand words, so let me use the image below. (Check also the Intuitive explanation of the bias-variance tradeoff? thread.)
Imagine your model is an oracle that perfectly predicts the target, it will have no bias and no variance.
Then, why is not possible to reduce bias by reducing the number of training samples.
Imagine a model that always predicts the same constant (say, $42$), it will be biased regardless of how much data would you use because the result is independent of the data. The example is abstract, but not as abstract as you may think, for example, this would be the case for a Bayesian model with a very strong prior, or using an incorrect model for the job (e.g. image classification using a model that was designed for natural language processing), such models are doomed to make bad predictions regardless of the data.
-
$\begingroup$ understood... in case of the model that predict constant 42... what will be the variance ? variance also should remain constant. but can we determine if it is high or low ? $\endgroup$ Commented Jun 30, 2022 at 20:10
$\begingroup$
$\endgroup$
2
You can only speak of 'inverse relation' when you change model complexity (including, to some extent, feature selection vs feature addition).
- As the number of samples grows, variance drops, bias is unchanged.
- Increasing the model complexity decreases bias but increases variance.
- Dropping irrelevant features decreases variance, adding relevant features decreases bias.
-
$\begingroup$ why so ? for example in a regression model, as the variance drops, there must be some change in prediction values. So the MSE will also be changed. Then bias should also be changed as there is change in MSE ? $\endgroup$ Commented Jun 30, 2022 at 20:06
-
$\begingroup$ Omitting the irreducible noise, MSE breaks down into mean square of (true value - average prediction) [bias^2] and mean square of (prediction - average prediction) [variance]. Predictions with the same expectation can possess all sorts of variances. $\endgroup$– dx2-66Commented Jul 1, 2022 at 11:08
$\begingroup$
$\endgroup$
The variance bias relationship is a causal relationship.
For certain types of bias the variance can reduce when the bias increases. That means that the other way around does not work. If you reduce/increase the variance (by some other means than bias, for instance sample size) then you do not increase/reduce the bias.
A clear example is a shrinkage estimator for estimating the mean of a normal distribution. The typical unbiased estimator is the sample mean $\bar{x}$, which has a variance $\frac{\sigma^2}{n}$. We could also use a biased estimate by multiplying the sample mean with a factor $c \cdot \bar{x}$, which has a variance $c^2\frac{\sigma^2}{n}$. By changing the bias $c$ we can decrease the variance, but by changing the variance, for instance by having different $n$, we do not change $c$.
In a more complex situation we could have an interaction between bias and variance. In the previous example the bias is dependent on $c$. There are settings in which we change this parameter based on the observed variance. But often the relationship is in the other direction. The bias is optimised to reduce the mean squared error, and if we have less variance, then we need less bias.
answered Jul 1, 2022 at 17:17