Skip to main content

Questions tagged [bias-variance-tradeoff]

In predictive modeling, unbiased models can have higher variance, & thus be less accurate. Modelers may prefer some bias to maximize accuracy. Use this tag also for questions about the bias-variance decomposition.

88 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
6 votes
1 answer
1k views

What do Cross Validation results actually tell about Bias and Variance?

I am trying to get a deeper understanding of the common ML pipelines and I have some doubts regarding Cross Validation, why do we really use it and what does it really tell us about Bias and Variance. ...
rusiano's user avatar
  • 566
5 votes
2 answers
2k views

Reconstruction Error: Principal component analysis vs Probabilistic prinicpal component analysis

I am working through the book "Machine Learning: A Probabilistic Perspective". After introducing PCA and Probabilistic PCA, the following graphic is shown (the upper two graphics ...
user avatar
5 votes
0 answers
355 views

Is there a theoretical reason why simple models perform better than complex models on time series forecasting tasks?

Empirically, simple forecasting methods such as damped trend exponential smoothing, STL, or even random walks typically outperform more complex models such as higher order ARIMA models or ML based ...
Skander H.'s user avatar
4 votes
0 answers
86 views

Can the calibration-discrimination decomposition of Brier score be viewed as the bias-variance decomposition of mean squared error?

The mean squared error has a famous decomposition into bias and variance. $$ \text{MSE} = \text{bias}^2 + \text{var} $$ Brier score is also a mean squared error calculation, and Brier score has a ...
Dave's user avatar
  • 63.6k
4 votes
0 answers
32 views

Variance analysis on boosting approachs, Is there any guarantee that boosting will not worse the weak learner variance or even get it better?

I'm looking for a theorical justification why boosting does work in pratice, I'm almost sure that this reduces the bias of their weak learners (assuming all weak learners have the same bias), but I ...
Davi Américo's user avatar
4 votes
0 answers
512 views

Does Bias-Variance Tradeoff always exist?

I'm following deeplearning.ai's videos on Coursera. In one of the videos, Prof Ng mentions: So a couple of points to notice. First is that, depending on whether you have high bias or high variance, ...
Nitin's user avatar
  • 349
3 votes
1 answer
90 views

Prediction intervals and bias-variance tradeoff

I was looking for literature which connects prediction intervals with the bias-variance trade-off. Obviously both concepts deal with describing a mean squared deviation: the bias variance tradeoff ...
Ggjj11's user avatar
  • 1,343
3 votes
0 answers
408 views

Apart from the Bias-Variance "Decomposition" - is there a Bias-Variance "Proof"?

I am sure at some point, many of us have come across the "Bias-Variance Tradeoff" : The "error" of any "estimator" (e.g an estimator can be considered as a linear ...
stats_noob's user avatar
3 votes
0 answers
47 views

Who first discovered the decomposition of mean squared error?

Can someone here provide the earliest known reference to the decomposition of MSE into variance and bias squared?
user120911's user avatar
3 votes
0 answers
419 views

What is "Entropic Capacity"?

I found this term on the Keras blog website, quoted below Your main focus for fighting overfitting should be the entropic capacity of your model --how much information your model is allowed to store. ...
Shuklaswag's user avatar
3 votes
0 answers
182 views

Curse of dimensionality with 1-NN in an specific example of Elements of Statistical Learning

I'm reading "The Elements of Statistical Learning", in Fig. 2.8 there is a simulation example highlighting curse of dimensionality with $1$-NN (Nearest Neighbors). In this example the variance should ...
ahstat's user avatar
  • 1,250
2 votes
1 answer
84 views

What is fixed and what varies in the bias-variance decomposition?

I am reading about the bias-variance decomposition from An Introduction to Statistical Learning with Applications in R (Second edition at page 34). It states that $$Y = f(X) + \epsilon$$ where the ...
ado sar's user avatar
  • 471
2 votes
0 answers
41 views

Minimize MSE for only some parameters

Suppose I have a structural model parameterized by some $\theta = (\beta_i)_{i=1}^n$, but I am only interested in obtaining an unbiased/consistent/low variance estimator for $\beta_1$. For example, ...
Daniel's user avatar
  • 121
2 votes
0 answers
260 views

Is deep double descent important in practical contemporary CNNs?

Deep double descent is an empirically observed phenomenon that happens with contemporary neural networks. Its essence is that often, increasing the model complexity first leads to the test loss ...
CrabMan's user avatar
  • 172
2 votes
1 answer
77 views

Cross-validation: error estimation and bias

When obtaining the error estimation of a model over a dataset using k-fold cross-validation, lower values of the error estimation necessarily imply a lower bias? Are both concepts, error estimation ...
dreamco9's user avatar
2 votes
0 answers
509 views

Bias-variance trade-off between LDA and QDA w.r.t. dimensionality

Consider the bias-variance trade-off between linear discriminant analysis (LDA) and quadratic discriminant analysis (QDA). Switching from QDA to LDA will generally yield a reduction in variance. The ...
Richard Hardy's user avatar
2 votes
0 answers
246 views

Bias, variance, train and test/validation error + total error

I'm trying to make sense out of the following two plots: Source: https://www.linkedin.com/pulse/bias-variance-tradeoff-machine-learning-satya-mallick/ Source: https://towardsdatascience.com/the-bias-...
Ben's user avatar
  • 3,473
2 votes
0 answers
548 views

Machine Learning with few observations

Is common to say that Machine Learning techniques represent are purely data driven methods, and them are effective only if we have a large amount of data. I focused here on supervised/predictive ...
markowitz's user avatar
  • 5,689
2 votes
0 answers
35 views

What is the relationship (if any) between the number of input features and underfitting/ overfitting in ML/DL models?

Firstly below are a few points on this topic based on my understanding: Overfitting occurs when a particular model is too complex for the given data. This results in the model memorizing the data ...
DrSpill's user avatar
  • 121
2 votes
0 answers
134 views

What is the Bias Variance Tradeoff from a Bayesian perspective

How do Bayesian's treat the Bias Variance Tradeoff? Typically the Bias Variance Tradeoff is expressed as $Bias^2 + Variance + irreducible\_error$, however wouldn't choosing a prior introduce bias and ...
J Doe's user avatar
  • 372
2 votes
0 answers
315 views

Does there exist a Bayesian analysis of bias-variance decomposition of an estimator?

I was wondering if anyone could spare a moment to help with the answers to the following questions. Suppose we have an estimator $\hat{\theta}:\mathbb{R}^{d}\rightarrow\mathbb{R}$ such that the ...
Kirk Walla's user avatar
2 votes
0 answers
45 views

Understanding the trade-off between bias and variance in machine learning prediction using the math formula

About machine learning prediction, I would like to understand the trade-off between the bias and variance but using the mathematical formula. We have some train data X and y target variable. To be ...
Fam's user avatar
  • 1,007
2 votes
0 answers
88 views

Bias Variance Decomposition 2.7 in Elements of Statistical Inference

I try to derive 2.7 from the book. I expose my demonstration $E_\tau[(y_0-\hat{y}_0)^2]=E_\tau[y_0^2]-2E_{\tau}[y_{0}\hat{y_{0}}]+E_{\tau}[\hat{y_{0}}^{2}]$ $= y_{...
kiriloff's user avatar
  • 643
2 votes
0 answers
221 views

Is there any case in which unbiased but larger MSE estimator preferred to biased and smaller MSE one?

Let saying we are interested in a population mean $\mu$ and we have two estimators $\hat{\mu}_{n}^b$ and $\hat{\mu}_{n}^{u}$ defined on $n$ samples such that $\hat{\mu}_{n}^b$ : biased (i.e, $\mathbb{...
JaeHyeok Shin's user avatar
2 votes
0 answers
949 views

Cross validation and the Bias Variance trade-off

So I know that there have been a lot of questions about this topic but I try to understand it from a bit more theoretical/mathematical point of view. I have some basic questions of how cross-...
guest1's user avatar
  • 799
2 votes
0 answers
183 views

How to evaluate the bias-variance tradeoff?

How can you evaluate the bias-variance tradeoff by looking at the train error and at the test error?
Claire's user avatar
  • 63
2 votes
0 answers
460 views

Effect of sample size and f(x) on bias and variance components of the error

I was wondering how sample size and the underlying function (linear, logistic, loglinear etc.) affect the bias and variance component of the error. Is it correct to say that for a perfectly linear ...
Chen Shen's user avatar
1 vote
0 answers
18 views

Optimal estimate under altered MSE loss function

Suppose I am interested in estimating $\theta \in \mathbb{R}$ and I observe a noisy data point $\tilde{\theta}=\theta + N(0,\sigma^2)$ where $\sigma^2$ is known. I am interested in constructing an ...
econ_enthusiast's user avatar
1 vote
0 answers
59 views

Who was the first to notice that the bias can be decomposed into model bias and estimation bias?

As the title says, who was the first to notice that the bias can be decomposed into model bias and estimation bias? For reference, I'm talking about the quantities here at page 224 eq. (7.14) https://...
rick's user avatar
  • 11
1 vote
0 answers
25 views

Bias-Variance Tradeoff, computing bias theoretically

Bias, in machine learning, is mathematically defined as $f-E(\hat{f})$, where $f$ is the true model and $\hat{f}$ is the estimate. I was wondering how we can compute theoretically $E(\hat{f})$, given ...
cgo's user avatar
  • 9,167
1 vote
1 answer
66 views

Practical usage of the bias variance tradeoff

I understand the bias-variance tradeoff. But, I have never come across a scenario where that has changed anything in the modelling process. Is there any practical scenario that you have encountered ...
figs_and_nuts's user avatar
1 vote
0 answers
53 views

Bias and variance for quantile estimates

Is bias variance tradeoff a thing for quartile regression? Can I assume the error for quantile estimation follows a certain distribution (e.g., estimated quantile - true quantile follows normal ...
notfunnyatall's user avatar
1 vote
0 answers
97 views

Propensity score matching with replacement - OK to trim excess control group matches to same treatment subject?

My team is conducting propensity score matching with 1:1 nearest neighbor replacement for a case-control healthcare study. While we're obtaining match rates of 80-90% with good covariate balance, we ...
RobertF's user avatar
  • 6,124
1 vote
0 answers
470 views

When do control variables increase precision?

Suppose we're interested in the effect $\beta$ of a treatment $D$. To increase the precision of our estimate (ie., reduce the variance of $\hat{\beta}$), we can include a control variable $X$ that ...
Macaulay's user avatar
1 vote
0 answers
38 views

Could I pick the best model based on the bias-variance tradeoff?

Usually would I pick the best performing model according to accuracy, or another evaluation method, on the validation dataset. But is this viable to chose the best model according to bias-variance ...
Adrian Evensen's user avatar
1 vote
0 answers
22 views

does assumptions effect the bias or variance?

in machine learning text it is often said that assumptions affect bias like the following text from Kevin Murphy: "Given the large variety of models in the literature, it is natural to wonder ...
john's user avatar
  • 21
1 vote
0 answers
72 views

Bias vs. variance

I have a question about bias/variance trade-off for different competing models. Say one has estimated model A and model B and calculated their respective train and test error. How does one yield an ...
alphaH's user avatar
  • 37
1 vote
1 answer
282 views

Bias variance tradeoff in Gaussian process regression

Can someone explain in simple terms the bias and variance tradeoff of Gaussian Process regression?
mathlete42's user avatar
1 vote
0 answers
22 views

Is an increase in variability of the per-sample loss values a sign of overfitting?

I trained a couple of vanilla U-Nets at depths of 1, 2 and 3 (by depth I refer to down max-pooling steps) and different numbers of filters. Running on the validation data I get the following graphs ...
packoman's user avatar
  • 131
1 vote
0 answers
20 views

How does the splitting point in cross-validation affect the performance estimate?

I want to use simple cross-validation to estimate the performance of a model. There are different ways how I can split the data into a training and test dataset. Let's consider three cases: a) 50% ...
Funkwecker's user avatar
  • 3,092
1 vote
3 answers
2k views

Understanding bias and variance for different models over same dataset

Consider we have 1-D data generated by a polynomial of degree 5. Which will of thhe following give higher / lower bias and higher / lower variance? Regression with linear basis functions Regression ...
RajS's user avatar
  • 151
1 vote
0 answers
190 views

Variance-bias tradeoff problem and how Bayesian and non-Bayesian approaches perform in a big data setting

When it comes to deal with the variance-bias tradeoff issue, I assume that bias is automatically induced in the Bayesian approach just from using a prior, while non-Bayesian approaches use math to ...
jpcgandre's user avatar
  • 413
1 vote
0 answers
128 views

Relationship between Bias/Variance and Covariates in Ridge/Lasso Regression

Suppose I add irrelevant (i.e. no explanatory power) regressors to a ridge/lasso regression. Does this impact the model bias/variance? In the case of OLS, the model bias remains unchanged, while the ...
shenflow's user avatar
  • 1,119
1 vote
0 answers
44 views

If base classifier is stable then error of ensemble is caused by bias in base classifier. Why?

I'm reading the book- Intro to Data Mining by Pang-Ning Tan. Under "Bagging" it's written: If a base classifier is stable, i.e., robust to minor perturbations in the training set, then the ...
DiamondDust's user avatar
1 vote
1 answer
109 views

Role of misspecification by biased data in the generalization error

I am confused with the role that model misspecification plays in the generalization error, in particular when the misspecification is due to a biased (non representative) training dataset. To clarify ...
synack's user avatar
  • 371
1 vote
0 answers
18 views

Description of how the count of binary patches and their size is affected by noise?

In acoustics, signals can be represented as a matrix $M$ in time, frequency, and amplitude. Obviously the signal we want to describe is always superposed over other noise, $N$: One way to analyze ...
Halyn Betchkal's user avatar
1 vote
0 answers
22 views

Relation between nonlinearity and hypothesis set in the view of bias variance trade off

I understand bias variance Trade off pretty well in terms of linear regression. As when the functions gets more wiggly (higher degrees) then variance on test set increases. Makes sense. However, in a ...
Salih's user avatar
  • 287
1 vote
0 answers
33 views

What does the famous bias-variance figure actually represent?

Below figure is generally used to explain bias-variance tradeoff. But something which is not clear and not explained anywhere is: What does the dots represent ? Do they represent: 1. predictions on ...
mach's user avatar
  • 1,815
1 vote
1 answer
374 views

How to measure variance that is due to random noise in training data

I would like to measure the variance of a binary classification model (deep neural network). Say the performance metric of choice is f1-score. There are two sources of variance that I can think about: ...
hyperio's user avatar
  • 125
1 vote
0 answers
41 views

Why do correlated decision trees increase bias?

To decrease variance in decision trees often, bagging is used to create several decision trees. This has better accuracy than one tree since the decrease in variance outweighs the increase in bias. ...
Tibo Geysen's user avatar