Questions tagged [bias-variance-tradeoff]

In predictive modeling, unbiased models can have higher variance, & thus be less accurate. Modelers may prefer some bias to maximize accuracy. Use this tag also for questions about the bias-variance decomposition.

Filter by
Sorted by
Tagged with
1
vote
0answers
46 views

Bias-variance decomposition for time series

Suppose you have a time series $y_t$. What is the bias-variance decomposition of $$E[[y_{t+1} - \hat{y}_{t+1}]^2|y_1, \dots, y_t] $$ where $\hat{y}_{t+1}$ is some forecast of $y_t$. I tried to ...
1
vote
1answer
78 views

Bias-Variance Trade Off with Cauchy Estimator

I'm having a look at the bias and standard error of a set of estimators. I expected to see the trade off when varying the parameter of the estimator, but I see that both the bias and the variance ...
0
votes
0answers
20 views

How to interprete the derivative of the bias with respect to the MSE-bound for biased estimators?

In my lecture notes I stumbled upon the following theorem. Let $f_\theta(x)$ be fisher-regular. If our estimator $\hat{\theta}$ has Bias $B(\theta) = E_\theta(\hat{\theta})-\theta$ than it holds: $...
1
vote
0answers
14 views

Properties of the mean of a predictor function

I am learning about the bias-variance tradeoff, and my question is about the properties of the function $E_{D_n}(\hat f(x)) $ which appears inside both the $variance$ and $bias^2$ terms. I am looking ...
0
votes
0answers
10 views

Measuring the variance of an unsupervised model

I have a combined model that consists of two unsupervised models, an auto-encoder and a K-means clustering model. The combined model is showing variance in its prediction and I was not sure how to ...
1
vote
3answers
167 views

What is the random variable when we talk about high variance model or high bias model?

I have read about what a high variance and high bias model is and everywhere the emphasis is more on the consequences of either. I am confused as to what the random variable is when we are talking ...
3
votes
0answers
61 views

Does Bias-Variance Tradeoff always exist?

I'm following deeplearning.ai's videos on Coursera. In one of the videos, Prof Ng mentions: So a couple of points to notice. First is that, depending on whether you have high bias or high variance, ...
2
votes
1answer
32 views

Maximum Likelihood: Bias and Efficiency

I have a basic question but I somehow can't wrap my head around it. It is said that Maximum Likelihood Estimation is unbiased and efficient in large samples. I am also aware that usually there is ...
0
votes
1answer
163 views

Why is there a bias variance tradeoff? A counterexample

Suppose that $$y=f(x)+\epsilon$$ Where $\epsilon$ has mean $0$ and variance $\sigma^2_e$, independent of $x$. Here is the composition of the mean-squared error into bias and variance: $$\begin{...
1
vote
0answers
39 views

what's in a name: Bias? [duplicate]

I'm studying machine learning and one of the key points I make sure is to understand the origin and the reasoning why such a name or term is used. It could be a rich and insightful dive into the ...
1
vote
0answers
59 views

Violation of the IID assumption in Gradient Boosting

Generally, machine learning methods make little to no statistical assumptions. However, a key assumption they do make is that the data are IID. What are the implications of a violation of the ...
1
vote
0answers
27 views

Bias Variance Decomposition

In "Elements of Statistical Learning" by Hastie et al. I am confused about the notation they use. (p.24 Second edition) For $Y=f(X)=e^{-8\| X\|^2}$ they use the 1-NN approximation to predict $y_0$ at ...
2
votes
0answers
29 views

Understanding the trade-off between bias and variance in machine learning prediction using the math formula

About machine learning prediction, I would like to understand the trade-off between the bias and variance but using the mathematical formula. We have some train data X and y target variable. To be ...
2
votes
1answer
60 views

Why does LOOCV produce correlated estimates?

(Before the edit) Why do the authors of Introduction to Statistical Learning state that: ..the test error estimate resulting from LOOCV tends to have higher variance than does the test error ...
0
votes
0answers
32 views

Best sampling method within the normal family

Suppose that we want to make the best Bayesian inference about some value $\mu$ we have some normal prior about it. I.e. $\mu\sim N(\mu_0, \sigma_0^2)$ with known parameters. To do so, we can choose ...
0
votes
0answers
44 views

Is it possible to have overfitting due to high bias?

Overfitting is usually associated with high variance, whereas underfitting is associated with high bias. But one of my professors at uni mentioned that overfitting might be caused by high variance and/...
1
vote
2answers
77 views

Does the MSE values of regression coefficients sum up to the MSE value of the regression model in which the regression coefficients are included?

I think either i dont understand something or i try to mix something that are different things. The mse value of a regression coefficients tells me how good i estimated the coefficent. Does it mean ...
5
votes
1answer
102 views

When does the underfitted regression model have more precise coefficient estimates?

Say we have a full regression model \begin{align*} \mathbf{y} &= \mathbf{X} \boldsymbol{\beta} + \boldsymbol{\epsilon}\\ &= \mathbf{X}_p \boldsymbol{\beta}_p + \mathbf{X}_r \boldsymbol{\beta}...
0
votes
0answers
21 views

Why i get the same MSE value for two least square models that differ in one explanatory variable?

I have two ols-regression models that just differ in one variable. It means that one model have the same variables like the other plus an explanatory variable more. I estimated both models on a train ...
0
votes
1answer
96 views

How to calculate Bias and Variance to get the MSE value step by step?

I want to compute my MSE value for a forecast step by step for test set. For me the Bias is: Bias = mean(predicted values - actual values) Variance = mean((predicted values- actual values)^2) ...
0
votes
0answers
27 views

Machine Learning Model Evaluation

If a model is overfitted that means decent gap between training curve and testing/validation curve but achieves good precision and recall score,does that still indicate that the model is decent?
1
vote
0answers
50 views

Does bias in regression coefficients affect the prediction?

Goal is to create ols model for out of sample prediction for log(wages). Theory say I could have a sample selection bias. So I choose the heckit method to correct for it. The correction term lambda (...
1
vote
0answers
29 views

Do unbiased regression coefficents yield better prediction?

I ask myself if a have a omitted variables bias in my regression modell the coefficients of the model are biased so the mse growth because this coefficents are biased right? So does it mean if i ...
0
votes
0answers
27 views

I understand over-fitting in RandomForest algorithm is taken care by bootstrapping & bagging but what happens if we prune the trees and apply bagging? [duplicate]

According to Random Forest algorithm, tress are not supposed to be pruned intuitively, don't bagging on pruned models give a better final model than the one without pruning?
2
votes
0answers
62 views

Bias Variance Decomposition 2.7 in Elements of Statistical Inference

I try to derive 2.7 from the book. I expose my demonstration $E_\tau[(y_0-\hat{y}_0)^2]=E_\tau[y_0^2]-2E_{\tau}[y_{0}\hat{y_{0}}]+E_{\tau}[\hat{y_{0}}^{2}]$ $= y_{...
0
votes
0answers
25 views

Variance-bias trade off in classification and regression trees

I know what the variance/bias trade off is when I am talking about regression problems. Also in this context i understand the technical derivation. But I don't have an idea of the variance-bias trade ...
3
votes
1answer
85 views

Mean Squared Error as quantifier of the Bias-Variance tradeoff

I have acquired the impression that many of the people doing statistical work, will prefer a biased estimator $\hat b$ to an unbiased one $\hat \beta$, if the former has lower Mean Squared Error. This ...
3
votes
2answers
142 views

Where does linear regression fit into the bias-variance tradeoff?

In ISL, the concept of the bias-variance tradeoff is presented with the rule of thumb that simple models will have high bias and that complex models will have high variance. Given this idea, I would ...
2
votes
2answers
246 views

Variance/bias trade off regularisation penalty - why does it take this form?

In machine learning, if we estimate weights using a loss function $$L(W) = ||Y-F_W(X)||^2$$ (where $W$ is a weight matrix) we may add a "regularisation penalty" to control for the "variance/bias ...
3
votes
2answers
381 views

Bias-variance tradeoff associated with cross validation methods

I was reading about the bias-variance tradeoff associated with cross validation methods on James et al, Introduction to Statistical Learning (Page 183-184). When we perform LOOCV, we are in effect ...
0
votes
0answers
24 views

Issues when converting panel to cross-section? (i.e. $(y_{it},x_{it}) \rightarrow (y_{i},x_{i})$)

Let's say we've got a panel data set of individuals $i$ over time $t$, with their inputs $x_{it}$ and outputs $y_{it}$. We want to relate inputs to outputs, but the problem is that the timing of ...
1
vote
0answers
135 views

Bias-Variance decomposition for non-squared loss

While the Bias-Variance decomposition of the squared loss is part of any introductory ML class, I am curious to know if similar decompositions can be done for other loss functions, e.g., cross entropy?...
3
votes
1answer
140 views

bias variance tradeoff — properties that do not follow

Going through this lecture note on bias-variance trade-off, I didn't follow the latter part of this paragraph. It shows the common situation in practice that (1) for simple models, the bias ...
3
votes
1answer
194 views

Frequentist vs. Bayesian bias-variance decomposition

Iv'e read the answer to this related question and still have some issues. Suppose that given some data $X$, we want an estimator $\hat{\theta}$ for some parameter $\theta$. A common approach is to ...
2
votes
0answers
60 views

Is there any case in which unbiased but larger MSE estimator preferred to biased and smaller MSE one?

Let saying we are interested in a population mean $\mu$ and we have two estimators $\hat{\mu}_{n}^b$ and $\hat{\mu}_{n}^{u}$ defined on $n$ samples such that $\hat{\mu}_{n}^b$ : biased (i.e, $\mathbb{...
0
votes
1answer
26 views

Is variance the only cause of overfitting in any Machine Learning Algorithm?

I am used to associating high variance with overfitting - I don't know how else to think of the physical manifestation of variance wrt an algorithm. Also, the only cause of overfitting seems to be ...
3
votes
1answer
1k views

How can we explain the fact that “Bagging reduces the variance while retaining the bias” mathematically?

I am able to understand the intution behind saying that "Bagging reduces the variance while retaining the bias". What is the mathematically principle behind this intution? I checked with few experts ...
1
vote
1answer
209 views

Bias-variance decomposition of MSE

I'm trying to decompose the MSE into the bias and variance terms and have done the following: $$MSE = \frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y_i})^2$$ $$E(MSE) = E\left[\frac{1}{n}\sum_{i=1}^{n} (y_i - ...
1
vote
0answers
307 views

Cross validation and the Bias Variance trade-off

So I know that there have been a lot of questions about this topic but I try to understand it from a bit more theoretical/mathematical point of view. I have some basic questions of how cross-...
2
votes
0answers
54 views

Bias variance tradeoff when the estimated target function is also random

I'm interested to understand "bias variance tradeoff" notion in a different setting than usually presented. In a setting where target $f$ (see the map $f$ below) is a random map rather than ...
2
votes
1answer
45 views

What to do if recombination of independent variables cause multicollinearity issue?

Let's say you use a regression that has either: 1) interaction variables or 2) polynomials. When using those features you may run into multicollinearity issues. Do you know how to resolve this issue?...
1
vote
0answers
34 views

Estimate of population mean that minimises squared error

When reading Susan Athey's lecture slides here, I was confused by her claim that the population mean estimate for a given sample that minimises squared prediction error was not the sample mean, but ...
2
votes
1answer
129 views

Tradeoffs of robust mean measures (trimmed, Huber, cosh, etc)

After recently having delved into the world of robust measures (for location, mean being the classical case), I have had difficulty understanding robust measures' core dynamic. Basically, what are ...
4
votes
1answer
723 views

Why does bagging increase bias?

In machine learning, why does bagging increase bias? I've read that using less data would lead to a worse estimate of the parameters, but isn't the expected value of the parameter constant regardless ...
1
vote
2answers
385 views

multicollinearity resulting in high variance

Section 8.7.1 of Elements of Statistical Learning talks about high variance in a classification tree due to high correlation between features. What is the intuition behind this? I would think that ...
1
vote
0answers
36 views

What is the relation of bias in the Bias-Variance trade-off to underfitting

In machine learning literature, people often talk about the bias-variance trade-off as well as overfitting and underfitting in the same paragraph. However, in these contexts, bias and variance comes ...
1
vote
0answers
91 views

regression model can't achieve low bias and low variance at the same time

I am running a RandomForestRegressor model on a dataset and it seems it can NOT achieve low bias and low variance at the same time. So I suspected that the input (independent) variables are not ...
5
votes
0answers
146 views

Is there a theoretical reason why simple models perform better than complex models on time series forecasting tasks?

Empirically, simple forecasting methods such as damped trend exponential smoothing, STL, or even random walks typically outperform more complex models such as higher order ARIMA models or ML based ...
1
vote
0answers
97 views

Can Bias-Variance Decomposition be applied to find MSE of a linear model?

I'm trying to apply the Bias-Variance Decomposition to the produced estimates of a linear model. X = [0,1,2,3,4,5,6,7,8,9] θ = [2,15,27,36,48,57,63,73,80,95] Linear model = 5 + 10x Estimated ...
2
votes
0answers
225 views

What is “Entropic Capacity”?

I found this term on the Keras blog website, quoted below Your main focus for fighting overfitting should be the entropic capacity of your model --how much information your model is allowed to ...