Questions tagged [bias-variance-tradeoff]
In predictive modeling, unbiased models can have higher variance, & thus be less accurate. Modelers may prefer some bias to maximize accuracy. Use this tag also for questions about the bias-variance decomposition.
243
questions
0
votes
1
answer
38
views
Practical usage of the bias variance tradeoff
I understand the bias-variance tradeoff. But, I have never come across a scenario where that has changed anything in the modelling process. Is there any practical scenario that you have encountered ...
- 2,263
0
votes
0
answers
10
views
What is the relationship between bias-variance and sensitivity-specificity for novelty detection?
An over or under-parameterized binary classification model (- vs +) tends to over or under-fit (bias-variance tradeoff). This leads to errors during prediction on unseen data. Depending on if ...
- 53
1
vote
1
answer
45
views
Understanding the computation of sample bias and variance
I believe I am confused in some fundamental way about the bias-variance tradeoff and I am trying to clear up my confusion. Sorry for a bit of a preliminary rambling -- I wanted to put my ...
- 11
3
votes
1
answer
45
views
Prediction intervals and bias-variance tradeoff
I was looking for literature which connects prediction intervals with the bias-variance trade-off.
Obviously both concepts deal with describing a mean squared deviation: the bias variance tradeoff ...
- 603
0
votes
1
answer
45
views
Understanding bias-variance tradeoff decomposition
This is the formula :
$E[(Y−f^2)]=σ^2 +Bias^2[f^]+Var[f^]$
What i cant understand , the expectation is over the training Set for a fixed $x_0$ , thus the
$E(ϵ^2) =E(ϵ(x_0)^2) = ϵ(x_0)^2$ and not $...
0
votes
0
answers
22
views
How do bias-effect and variance-effect reduce to bias squared and variance? [duplicate]
I'm currently reading James1997 - Generalizations of the Bias/Variance Decomposition for Prediction error.
My ultimate goal is to see how, in the special case of squared loss, bias-effect and variance-...
- 321
1
vote
1
answer
19
views
How to combine a noisy (but unbiased) estimate with a precise (but possibly biased) estimate in A/B tests?
Suppose I want to estimate some set of unknown quantities $\theta_1$, …, $\theta_N$. For each $i \in \{1, …, N\}$, I have two estimators: $\hat{\theta_i}_A$ and $ \hat{\theta_i}_B$. The goal is to ...
- 1,297
5
votes
6
answers
367
views
Bias-Variance tradeoff in prediction versus causal inference
In prediction, accepting a little more bias in exchange for a lot less variance is the very name of the game - we'll chose the model with minimal test MSE without regard for its composition (bias ...
- 4,166
0
votes
1
answer
65
views
How to avoid bias/avoid overfitting when choosing a machine learning model? [closed]
My typical workflow in the past, when creating machine learning models, has been to do the following:
Decide on some candidate model families for the task at hand.
Divide dataset into train and test ...
anon
2
votes
1
answer
74
views
Is truncated mean a biased estimator
We have data $X_1, \dots, X_n$ which are i.i.d copies of $X$. Where we denote $\mathbb{E}[X] = \mu$, and $X$ has finite variance.
We define the truncated sample mean:
$\begin{align}
\hat{\mu}^{\...
- 319
2
votes
1
answer
104
views
Proving upper bound for truncated difference
Let $X$ and $Y$ be real valued random variables. And define a truncation operator as:
$\begin{align}
X(\tau) = (|X| \wedge \tau) \; \text{sign}(X), \quad \tau > 0
\end{align}$
Now, I am not ...
- 319
1
vote
0
answers
29
views
Bias and variance for quantile estimates
Is bias variance tradeoff a thing for quartile regression? Can I assume the error for quantile estimation follows a certain distribution (e.g., estimated quantile - true quantile follows normal ...
1
vote
1
answer
104
views
Proving upper bound for Bias of truncated sample mean
We have data $X_1, \dots, X_n$ which are i.i.d copies of $X$. Where we denote $\mathbb{E}[X] = \mu$, and $X$ has finite variance.
We define the truncated sample mean:
$\begin{align}
\hat{\mu}^{\...
- 319
1
vote
0
answers
57
views
Propensity score matching with replacement - OK to trim excess control group matches to same treatment subject?
My team is conducting propensity score matching with 1:1 nearest neighbor replacement for a case-control healthcare study.
While we're obtaining match rates of 80-90% with good covariate balance, we ...
- 5,442
0
votes
0
answers
19
views
Fitting the conditional expectation?
Say we want to fit some model to predict $\mathbf{E}(A | B)$, which is the expected value for some distribution (ex. Poisson).
What would be the benefit/loss of fitting this vs. computing $\mathbf{E}(...
- 269
2
votes
1
answer
64
views
What is fixed and what varies in the bias-variance decomposition?
I am reading about the bias-variance decomposition from An Introduction to Statistical Learning with Applications in R (Second edition at page 34). It states that $$Y = f(X) + \epsilon$$
where the ...
- 422
2
votes
1
answer
335
views
Philosophical insight of Bias Variance Decomposition
As we know that we can perform a Bias Variance decomposition of an Estimator with MSE as loss function and it will look like below:
$$\operatorname{MSE}(\hat{\theta}) = \operatorname{tr}(\operatorname{...
- 31
2
votes
2
answers
52
views
Why do you overfit if you train a linear regression model on a dataset that doesn't have enough datapoints?
First of all, definitionally speaking, linear regressions tend to underfit (have high bias, low variance).
Additionally, just intuitively speaking, it seems like a linear regression would underfit in ...
- 704
0
votes
0
answers
25
views
Sampling distribution, bias and variance of cross-validation methods (particularly LOOCV)
(TL;DR version below) If my understanding is correct, bias/variance are measures of goodness of fit of a statistical estimator w.r.t. the sampling distribution. So if I have a statistic $t(X)$ that ...
- 63
2
votes
1
answer
81
views
bias-variance decomposition for OLS
Section 7.3 of Elements of Statistical Learning (2nd edition) gives the bias-variance decomposition for OLS prediction first for a single input $x_0$, and then averaged over a set of inputs $x_1, \...
- 143
2
votes
0
answers
36
views
Minimize MSE for only some parameters
Suppose I have a structural model parameterized by some $\theta = (\beta_i)_{i=1}^n$, but I am only interested in obtaining an unbiased/consistent/low variance estimator for $\beta_1$. For example, ...
- 121
0
votes
0
answers
99
views
More training data decreases variance
A similar question: Does more training data help lower the bias of a high bias model?
The answer mentions
Bias, is defined as $\operatorname{Bias}[\hat{f}(x)]=\mathrm{E}[\hat{f}(x)]-f(x)$ and thus ...
- 143
0
votes
0
answers
23
views
Linear Regression, predictions get worse with more parameters
I would like to get a better intuition for the Overfitting and Underfitting occuring in Machine Learning. My intuition before going into this experiment was, if I have $3$ data points with $2$ ...
- 505
0
votes
0
answers
75
views
Understanding Plots Related to the Variance-Bias Tradeoff and Plotting them in R
I am recreating some of the work by the authors from Introduction to Statistical Learning (2nd Edition) by Hastie et at. This text can be found here:
https://hastie.su.domains/ISLR2/ISLRv2_website.pdf
...
- 205
2
votes
1
answer
263
views
What is an intuitive way to think about why high variance in predictions is associated with overfitting of a model?
I read that for linear models, when more variables are added to the regression, typically the bias of the predictions decreases and the variance increases. That is:
Too few covariates yields high ...
- 3,452
3
votes
2
answers
163
views
What is the relationship between noise reduction and dimension reduction?
My understanding is that unsupervised methods like PCA, autoencoders and K-means shape a data space such that the modified representation of the data either nicely separates different families of data ...
- 53
3
votes
3
answers
1k
views
If we reduce size of training dataset does it decreases bias?
I'm a newbie and learning ML. I've a doubt, normally we know we should increase the size of training dataset or should add more data to reduce variance (fairly understood why). Now variance has ...
- 133
2
votes
0
answers
182
views
Is deep double descent important in practical contemporary CNNs?
Deep double descent is an empirically observed phenomenon that happens with contemporary neural networks. Its essence is that often, increasing the model complexity first leads to the test loss ...
- 162
9
votes
4
answers
1k
views
Bias Variance tradeoff in neural networks
Large neural networks have low bias and high variance. Training on large datasets greatly reduces the variance allowing them to fit complicated functions. My question is why they seem to have much ...
- 197
2
votes
2
answers
193
views
Robust distance weighted mean
Given a data sample $\{x_i\}_1^n$, instead of hard omitting outliers by e.g. trimming, one can form a weighted average where we soft penalize observations out in the tails.
\begin{align}
\mu = \frac{...
0
votes
1
answer
47
views
Welford vs Bayes?
To incrementally estimate the mean and standard deviation of some data one can use an algorithm such as Welford’s algorithm or Bayesian updating by using the likelihood, a conjugate prior and ...
- 151
2
votes
1
answer
75
views
Cross-validation: error estimation and bias
When obtaining the error estimation of a model over a dataset using k-fold cross-validation, lower values of the error estimation necessarily imply a lower bias? Are both concepts, error estimation ...
- 65
0
votes
0
answers
59
views
bias and variance decomposition derivation trick
I am following the derivation from here. My question is about the first trick, where the author claims that:
\begin{equation}
\begin{aligned}
E_{\mathbf{x}, y, D}\left[\left[h_{D}(\mathbf{x})-y\right]^...
1
vote
0
answers
344
views
When do control variables increase precision?
Suppose we're interested in the effect $\beta$ of a treatment $D$. To increase the precision of our estimate (ie., reduce the variance of $\hat{\beta}$), we can include a control variable $X$ that ...
- 31
1
vote
1
answer
128
views
Reasons to prefer low bias with higher variance over the alternative (and vice versa)
I am trying to understand the bias-variance tradeoff in practice. I have read several related questions and answers, but still have a few questions:
Assume we are estimating a structural equation ...
- 95
1
vote
0
answers
38
views
Could I pick the best model based on the bias-variance tradeoff?
Usually would I pick the best performing model according to accuracy, or another evaluation method, on the validation dataset.
But is this viable to chose the best model according to bias-variance ...
- 153
1
vote
0
answers
19
views
does assumptions effect the bias or variance?
in machine learning text it is often said that assumptions affect bias like the following text from Kevin Murphy:
"Given the large variety of models in the literature, it is natural to wonder ...
- 21
6
votes
2
answers
690
views
Does bias eventually increase with model complexity?
Does bias eventually increase with model complexity?
Reasoning behind the question:
If I understand it correctly, "bias" measures the discrepancy between the expected value of our model's ($...
- 433
7
votes
4
answers
2k
views
Why do we say that the model has a high variance when variance is actually the measure of spread of the data and not some property of the model?
I am trying to understand the difference between bias-variance and overfitting-underfitting. If a modal overfits the data it means that it will not generalize well on new data because it over learns ...
2
votes
1
answer
1k
views
Poor model performance on certain out-of-sample data
We're noticing poor model performance on certain out of sample products.
We have trained a ML model on about 2000 different products in a few markets.
Our predictors include
a) product ...
- 528
0
votes
1
answer
124
views
Bias-variance trade-off in linear regression [closed]
As it’s understood, in the bias-variance trade-off, variance refers to overfitting of the model and it examines the variability of output predictions.
Suppose we have a simple dataset with one ...
- 31
1
vote
0
answers
25
views
Variance analysis on boosting approachs, Is there any guarantee that boosting will not worse the weak learner variance or even get it better?
I'm looking for a theorical justification why boosting does work in pratice, I'm almost sure that this reduces the bias of their weak learners (assuming all weak learners have the same bias), but I ...
- 1,007
0
votes
0
answers
42
views
Checking understanding about a derivation of bias and variance in the context of generalization in course notes
Sorry for the long image. This derivation of bias and variance was given in publicly available course notes (here) on pages 3 and 4.
I understand the first derivation. They showed that y* was the best ...
- 207
5
votes
1
answer
389
views
Regularization and Shrinkage : Theoretical Advantages vs. Empirical Advantages [duplicate]
I have the following question about the theoretical advantages vs. the empirical advantages of regularization (i.e. shrinkage).
As far as I understand, this is the general idea behind regularization: ...
2
votes
0
answers
277
views
Apart from the Bias-Variance "Decomposition" - is there a Bias-Variance "Proof"?
I am sure at some point, many of us have come across the "Bias-Variance Tradeoff" : The "error" of any "estimator" (e.g an estimator can be considered as a linear ...
4
votes
2
answers
967
views
How does repeated k-fold cross validation identify model instability?
In these threads 1,2,3, cbeleites mentions that in a single k-fold cross validation you cannot tell whether the variance is caused by model instability or using a different test set. Hence, one can ...
- 319
2
votes
0
answers
389
views
Bias-variance trade-off between LDA and QDA w.r.t. dimensionality
Consider the bias-variance trade-off between linear discriminant analysis (LDA) and quadratic discriminant analysis (QDA). Switching from QDA to LDA will generally yield a reduction in variance. The ...
- 63.6k
1
vote
0
answers
65
views
Bias vs. variance
I have a question about bias/variance trade-off for different competing models. Say one has estimated model A and model B and calculated their respective train and test error. How does one yield an ...
- 37
1
vote
1
answer
885
views
Lasso vs Ridge Regression
My question relates on the Ridge vs Lasso Regression. I know the difference in the cost function (ridge penalizes sum of quadratic coefficients, lasso penalizes sum of absolute value of coefficients). ...
- 37
0
votes
1
answer
672
views
Bias-Variance tradeoff with Clustering algorithms
I'm investigating the bias-variance tradeoff in well-known machine learning algorithms. However, I'm not sure this concept applies in the case of unsupervised methods such as clustering algorithms.
Is ...
