Skip to main content

Questions tagged [bias-variance-tradeoff]

In predictive modeling, unbiased models can have higher variance, & thus be less accurate. Modelers may prefer some bias to maximize accuracy. Use this tag also for questions about the bias-variance decomposition.

Filter by
Sorted by
Tagged with
0 votes
0 answers
27 views

time series squared forecast evaluation

I have a time series with very weak autocorrelations- mostly unforecastable. However, its squared values have stronger autocorrelations. Something like this: ...
dayum's user avatar
  • 643
1 vote
0 answers
18 views

Optimal estimate under altered MSE loss function

Suppose I am interested in estimating $\theta \in \mathbb{R}$ and I observe a noisy data point $\tilde{\theta}=\theta + N(0,\sigma^2)$ where $\sigma^2$ is known. I am interested in constructing an ...
econ_enthusiast's user avatar
0 votes
1 answer
21 views

Does the intuitive sense of overfitting in this mechanism design context exemplify bias-variance tradeoff?

Suppose the (we can say unanimous) preference of each individual in a society is to select roads for travel by placing 95% weight on the objective of minimizing travel time, and the remaining 5% ...
user10478's user avatar
  • 123
5 votes
2 answers
281 views

Expected loss function from bias variance trade off (integral help)

I have a hard time understanding this formula. It's from bias-variance trade-off proof. and the expected loss function is as follows: $$L(\hat f) := \mathbb E_D\mathbb E_{(x,y)}[(y-\hat f(x))^2]=\...
Taewooo Kim's user avatar
0 votes
0 answers
18 views

Variance Bias Tradeoff decomposition of Linear Regression with a twist

Normally, for a linear regression problem with fixed observations, we have the variance and bias tradeoff as: $$Var(Y) + Bias^2 (\hat{\beta_x}) + Var(\hat{\beta_x})$$. My question is what happens to ...
Averi Tan's user avatar
3 votes
1 answer
49 views

What's the relationship between "bias-variance tradeoff" and "consistent model selection"?

I'm very confused about the relationship between "bias-variance tradeoff" and "consistent model selection". Based on my current interpretation, the ultimate goal of taking care of ...
ExcitedSnail's user avatar
  • 2,914
0 votes
0 answers
15 views

Formally state the bias-variance tradeoff & intuition [duplicate]

The following states the bias-variance dilemma formally: $$\hat{\epsilon}_i = E[\epsilon^2] + \left[f - E[\hat{f}] \right]^2 + E\left[ \left(\hat{f} - E[\hat{f}] \right)^2 \right]$$ where this is ...
Marlon Brando's user avatar
1 vote
1 answer
54 views

Does the Bias of the Model Only Depend on Model Class?

In machine learning with statistical approach, does the bias of the model solely depend on the selection of the model class without considering the training data? There is a claim regarding the bias-...
sharp_flyingrain's user avatar
4 votes
1 answer
96 views

Mean Squared Error for point estimation

I am attempting to understand Mean Squared Error when evaluating point estimators for particular parameters of interest. The book we are reading for class states the following: The mean squared error (...
Harry Lofi's user avatar
0 votes
0 answers
40 views

Derivation of bias variance trade-off with or without conditional expectation?

I found this nice lecture here where the bias variance trade-off is explained using conditional expectation - using e.g. $E_{y|X}[...]$ In this lecture here I found another proof of the formula ...
Christoph's user avatar
  • 209
1 vote
0 answers
59 views

Who was the first to notice that the bias can be decomposed into model bias and estimation bias?

As the title says, who was the first to notice that the bias can be decomposed into model bias and estimation bias? For reference, I'm talking about the quantities here at page 224 eq. (7.14) https://...
rick's user avatar
  • 11
1 vote
0 answers
25 views

Bias-Variance Tradeoff, computing bias theoretically

Bias, in machine learning, is mathematically defined as $f-E(\hat{f})$, where $f$ is the true model and $\hat{f}$ is the estimate. I was wondering how we can compute theoretically $E(\hat{f})$, given ...
cgo's user avatar
  • 9,177
3 votes
1 answer
33 views

Explanation for the success of bagging

I'm reading Machine Learning - A First Course for Engineers and Scientists. On page 168 they give a rough explanation for why bagging works. I'm a little confused by their explanation. They consider ...
oweydd's user avatar
  • 225
6 votes
1 answer
51 views

Bias-variance trade-off for a specific fitted model vs. a class of models: terminology

Consider a data generating process $$Y=f(X)+\varepsilon$$ where $\varepsilon$ is independent of $x$ with $\mathbb E(\varepsilon)=0$ and $\text{Var}(\varepsilon)=\sigma^2_\varepsilon$. According to ...
Richard Hardy's user avatar
7 votes
1 answer
507 views

Understanding importance sampling in Monte Carlo integration

Introduction I'm studying importance sampling and I'm trying to figure out by myself, with a couple of examples, what are the main benefits with respect to standard Monte Carlo integration. I'm not ...
matteogost's user avatar
0 votes
0 answers
21 views

in-sample error and the optimism

I'm currently reading p228 of The Element of Statistical Learning, which covers training error, in-sample error, and optimism. Let me quote some of the textbook contents as follows. The $Y^{0}$ ...
jason 1's user avatar
  • 311
3 votes
0 answers
70 views

Why is the variance smaller for the same coefficient in a reduced regression model vs. full regression model? [duplicate]

Let's say we have two estimators for $\beta$. $\beta$ denotes all a full set of coefficients, one for each covariate in a dataframe. $\beta$ can be split into $\beta_p$ and $\beta_r$, where $p$ ...
Estimate the estimators's user avatar
0 votes
0 answers
9 views

How does the training set size affect the uncertainty (variance) of performance estimation?

I am reading this paper which discusses the factors that affect the uncertainty (variance) in the performance estimation of a learner. The authors say (p. 2, "The monotonicity of the learning ...
ado sar's user avatar
  • 471
4 votes
0 answers
86 views

Can the calibration-discrimination decomposition of Brier score be viewed as the bias-variance decomposition of mean squared error?

The mean squared error has a famous decomposition into bias and variance. $$ \text{MSE} = \text{bias}^2 + \text{var} $$ Brier score is also a mean squared error calculation, and Brier score has a ...
Dave's user avatar
  • 63.8k
1 vote
1 answer
67 views

Practical usage of the bias variance tradeoff

I understand the bias-variance tradeoff. But, I have never come across a scenario where that has changed anything in the modelling process. Is there any practical scenario that you have encountered ...
figs_and_nuts's user avatar
0 votes
0 answers
38 views

What is the relationship between bias-variance and sensitivity-specificity for novelty detection?

An over or under-parameterized binary classification model (- vs +) tends to over or under-fit (bias-variance tradeoff). This leads to errors during prediction on unseen data. Depending on if ...
Douw Marx's user avatar
1 vote
1 answer
90 views

Understanding the computation of sample bias and variance

I believe I am confused in some fundamental way about the bias-variance tradeoff and I am trying to clear up my confusion. Sorry for a bit of a preliminary rambling -- I wanted to put my ...
user394866's user avatar
3 votes
1 answer
90 views

Prediction intervals and bias-variance tradeoff

I was looking for literature which connects prediction intervals with the bias-variance trade-off. Obviously both concepts deal with describing a mean squared deviation: the bias variance tradeoff ...
Ggjj11's user avatar
  • 1,343
0 votes
1 answer
59 views

Understanding bias-variance tradeoff decomposition

This is the formula : $E[(Y−f^2)]=σ^2 +Bias^2[f^]+Var[f^]$ What i cant understand , the expectation is over the training Set for a fixed $x_0$ , thus the $E(ϵ^2) =E(ϵ(x_0)^2) = ϵ(x_0)^2$ and not $...
Hocine Islam GUIA's user avatar
0 votes
0 answers
35 views

How do bias-effect and variance-effect reduce to bias squared and variance? [duplicate]

I'm currently reading James1997 - Generalizations of the Bias/Variance Decomposition for Prediction error. My ultimate goal is to see how, in the special case of squared loss, bias-effect and variance-...
ngmir's user avatar
  • 339
1 vote
1 answer
24 views

How to combine a noisy (but unbiased) estimate with a precise (but possibly biased) estimate in A/B tests?

Suppose I want to estimate some set of unknown quantities $\theta_1$, …, $\theta_N$. For each $i \in \{1, …, N\}$, I have two estimators: $\hat{\theta_i}_A$ and $ \hat{\theta_i}_B$. The goal is to ...
frelk's user avatar
  • 1,357
9 votes
7 answers
742 views

Bias-Variance tradeoff in prediction versus causal inference

In prediction, accepting a little more bias in exchange for a lot less variance is the very name of the game - we'll chose the model with minimal test MSE without regard for its composition (bias ...
ColorStatistics's user avatar
0 votes
1 answer
138 views

How to avoid bias/avoid overfitting when choosing a machine learning model? [closed]

My typical workflow in the past, when creating machine learning models, has been to do the following: Decide on some candidate model families for the task at hand. Divide dataset into train and test ...
user avatar
2 votes
1 answer
118 views

Is truncated mean a biased estimator

We have data $X_1, \dots, X_n$ which are i.i.d copies of $X$. Where we denote $\mathbb{E}[X] = \mu$, and $X$ has finite variance. We define the truncated sample mean: $\begin{align} \hat{\mu}^{\...
Dylan Dijk's user avatar
2 votes
1 answer
114 views

Proving upper bound for truncated difference

Let $X$ and $Y$ be real valued random variables. And define a truncation operator as: $\begin{align} X(\tau) = (|X| \wedge \tau) \; \text{sign}(X), \quad \tau > 0 \end{align}$ Now, I am not ...
Dylan Dijk's user avatar
1 vote
0 answers
54 views

Bias and variance for quantile estimates

Is bias variance tradeoff a thing for quartile regression? Can I assume the error for quantile estimation follows a certain distribution (e.g., estimated quantile - true quantile follows normal ...
notfunnyatall's user avatar
1 vote
1 answer
128 views

Proving upper bound for Bias of truncated sample mean

We have data $X_1, \dots, X_n$ which are i.i.d copies of $X$. Where we denote $\mathbb{E}[X] = \mu$, and $X$ has finite variance. We define the truncated sample mean: $\begin{align} \hat{\mu}^{\...
Dylan Dijk's user avatar
1 vote
0 answers
97 views

Propensity score matching with replacement - OK to trim excess control group matches to same treatment subject?

My team is conducting propensity score matching with 1:1 nearest neighbor replacement for a case-control healthcare study. While we're obtaining match rates of 80-90% with good covariate balance, we ...
RobertF's user avatar
  • 6,124
0 votes
0 answers
29 views

Fitting the conditional expectation?

Say we want to fit some model to predict $\mathbf{E}(A | B)$, which is the expected value for some distribution (ex. Poisson). What would be the benefit/loss of fitting this vs. computing $\mathbf{E}(...
Victor M's user avatar
  • 309
2 votes
1 answer
84 views

What is fixed and what varies in the bias-variance decomposition?

I am reading about the bias-variance decomposition from An Introduction to Statistical Learning with Applications in R (Second edition at page 34). It states that $$Y = f(X) + \epsilon$$ where the ...
ado sar's user avatar
  • 471
2 votes
1 answer
365 views

Philosophical insight of Bias Variance Decomposition

As we know that we can perform a Bias Variance decomposition of an Estimator with MSE as loss function and it will look like below: $$\operatorname{MSE}(\hat{\theta}) = \operatorname{tr}(\operatorname{...
Rehan Guha's user avatar
2 votes
2 answers
79 views

Why do you overfit if you train a linear regression model on a dataset that doesn't have enough datapoints?

First of all, definitionally speaking, linear regressions tend to underfit (have high bias, low variance). Additionally, just intuitively speaking, it seems like a linear regression would underfit in ...
Katsu's user avatar
  • 981
0 votes
0 answers
26 views

Sampling distribution, bias and variance of cross-validation methods (particularly LOOCV)

(TL;DR version below) If my understanding is correct, bias/variance are measures of goodness of fit of a statistical estimator w.r.t. the sampling distribution. So if I have a statistic $t(X)$ that ...
statkun's user avatar
  • 63
3 votes
1 answer
108 views

bias-variance decomposition for OLS

Section 7.3 of Elements of Statistical Learning (2nd edition) gives the bias-variance decomposition for OLS prediction first for a single input $x_0$, and then averaged over a set of inputs $x_1, \...
hessian's user avatar
  • 153
2 votes
0 answers
41 views

Minimize MSE for only some parameters

Suppose I have a structural model parameterized by some $\theta = (\beta_i)_{i=1}^n$, but I am only interested in obtaining an unbiased/consistent/low variance estimator for $\beta_1$. For example, ...
Daniel's user avatar
  • 121
2 votes
1 answer
531 views

What is an intuitive way to think about why high variance in predictions is associated with overfitting of a model?

I read that for linear models, when more variables are added to the regression, typically the bias of the predictions decreases and the variance increases. That is: Too few covariates yields high ...
user321627's user avatar
  • 4,386
3 votes
2 answers
306 views

What is the relationship between noise reduction and dimension reduction?

My understanding is that unsupervised methods like PCA, autoencoders and K-means shape a data space such that the modified representation of the data either nicely separates different families of data ...
Douw Marx's user avatar
3 votes
3 answers
2k views

If we reduce size of training dataset does it decreases bias?

I'm a newbie and learning ML. I've a doubt, normally we know we should increase the size of training dataset or should add more data to reduce variance (fairly understood why). Now variance has ...
iamawesome's user avatar
2 votes
0 answers
261 views

Is deep double descent important in practical contemporary CNNs?

Deep double descent is an empirically observed phenomenon that happens with contemporary neural networks. Its essence is that often, increasing the model complexity first leads to the test loss ...
CrabMan's user avatar
  • 172
10 votes
4 answers
2k views

Bias Variance tradeoff in neural networks

Large neural networks have low bias and high variance. Training on large datasets greatly reduces the variance allowing them to fit complicated functions. My question is why they seem to have much ...
efthimio's user avatar
  • 207
2 votes
2 answers
374 views

Robust distance weighted mean

Given a data sample $\{x_i\}_1^n$, instead of hard omitting outliers by e.g. trimming, one can form a weighted average where we soft penalize observations out in the tails. \begin{align} \mu = \frac{...
Pontus Hultkrantz's user avatar
0 votes
1 answer
63 views

Welford vs Bayes?

To incrementally estimate the mean and standard deviation of some data one can use an algorithm such as Welford’s algorithm or Bayesian updating by using the likelihood, a conjugate prior and ...
user1134616's user avatar
2 votes
1 answer
78 views

Cross-validation: error estimation and bias

When obtaining the error estimation of a model over a dataset using k-fold cross-validation, lower values of the error estimation necessarily imply a lower bias? Are both concepts, error estimation ...
dreamco9's user avatar
0 votes
0 answers
73 views

bias and variance decomposition derivation trick

I am following the derivation from here. My question is about the first trick, where the author claims that: \begin{equation} \begin{aligned} E_{\mathbf{x}, y, D}\left[\left[h_{D}(\mathbf{x})-y\right]^...
Emanuel Huber's user avatar
1 vote
0 answers
473 views

When do control variables increase precision?

Suppose we're interested in the effect $\beta$ of a treatment $D$. To increase the precision of our estimate (ie., reduce the variance of $\hat{\beta}$), we can include a control variable $X$ that ...
Macaulay's user avatar

1
2 3 4 5 6