Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [bias-variance-tradeoff]

In predictive modeling, unbiased models can have higher variance, & thus be less accurate. Modelers may prefer some bias to maximize accuracy. Use this tag also for questions about the bias-variance decomposition.

0
votes
0answers
21 views

Machine Learning Model Evaluation

If a model is overfitted that means decent gap between training curve and testing/validation curve but achieves good precision and recall score,does that still indicate that the model is decent?
1
vote
0answers
30 views

Does bias in regression coefficients affect the prediction?

Goal is to create ols model for out of sample prediction for log(wages). Theory say I could have a sample selection bias. So I choose the heckit method to correct for it. The correction term lambda (...
1
vote
0answers
26 views

Do unbiased regression coefficents yield better prediction?

I ask myself if a have a omitted variables bias in my regression modell the coefficients of the model are biased so the mse growth because this coefficents are biased right? So does it mean if i ...
0
votes
0answers
20 views

I understand over-fitting in RandomForest algorithm is taken care by bootstrapping & bagging but what happens if we prune the trees and apply bagging? [duplicate]

According to Random Forest algorithm, tress are not supposed to be pruned intuitively, don't bagging on pruned models give a better final model than the one without pruning?
2
votes
0answers
51 views

Bias Variance Decomposition 2.7 in Elements of Statistical Inference

I try to derive 2.7 from the book. I expose my demonstration $E_\tau[(y_0-\hat{y}_0)^2]=E_\tau[y_0^2]-2E_{\tau}[y_{0}\hat{y_{0}}]+E_{\tau}[\hat{y_{0}}^{2}]$ $= y_{...
0
votes
0answers
16 views

Variance-bias trade off in classification and regression trees

I know what the variance/bias trade off is when I am talking about regression problems. Also in this context i understand the technical derivation. But I don't have an idea of the variance-bias trade ...
2
votes
0answers
23 views

Mean Squared Error as quantifier of the Bias-Variance tradeoff

I have acquired the impression that many of the people doing statistical work, will prefer a biased estimator $\hat b$ to an unbiased one $\hat \beta$, if the former has lower Mean Squared Error. This ...
3
votes
2answers
54 views

Where does linear regression fit into the bias-variance tradeoff?

In ISL, the concept of the bias-variance tradeoff is presented with the rule of thumb that simple models will have high bias and that complex models will have high variance. Given this idea, I would ...
0
votes
2answers
83 views

Variance/bias trade off regularisation penalty - why does it take this form?

In machine learning, if we estimate weights using a loss function $$L(W) = ||Y-F_W(X)||^2$$ (where $W$ is a weight matrix) we may add a "regularisation penalty" to control for the "variance/bias ...
3
votes
2answers
89 views

Bias-variance tradeoff associated with cross validation methods

I was reading about the bias-variance tradeoff associated with cross validation methods on James et al, Introduction to Statistical Learning (Page 183-184). When we perform LOOCV, we are in effect ...
0
votes
0answers
22 views

Issues when converting panel to cross-section? (i.e. $(y_{it},x_{it}) \rightarrow (y_{i},x_{i})$)

Let's say we've got a panel data set of individuals $i$ over time $t$, with their inputs $x_{it}$ and outputs $y_{it}$. We want to relate inputs to outputs, but the problem is that the timing of ...
1
vote
0answers
46 views

Bias-Variance decomposition for non-squared loss

While the Bias-Variance decomposition of the squared loss is part of any introductory ML class, I am curious to know if similar decompositions can be done for other loss functions, e.g., cross entropy?...
2
votes
1answer
99 views

bias variance tradeoff — properties that do not follow

Going through this lecture note on bias-variance trade-off, I didn't follow the latter part of this paragraph. It shows the common situation in practice that (1) for simple models, the bias ...
2
votes
1answer
98 views

Frequentist vs. Bayesian bias-variance decomposition

Iv'e read the answer to this related question and still have some issues. Suppose that given some data $X$, we want an estimator $\hat{\theta}$ for some parameter $\theta$. A common approach is to ...
2
votes
0answers
43 views

Is there any case in which unbiased but larger MSE estimator preferred to biased and smaller MSE one?

Let saying we are interested in a population mean $\mu$ and we have two estimators $\hat{\mu}_{n}^b$ and $\hat{\mu}_{n}^{u}$ defined on $n$ samples such that $\hat{\mu}_{n}^b$ : biased (i.e, $\mathbb{...
0
votes
1answer
18 views

Is variance the only cause of overfitting in any Machine Learning Algorithm?

I am used to associating high variance with overfitting - I don't know how else to think of the physical manifestation of variance wrt an algorithm. Also, the only cause of overfitting seems to be ...
2
votes
1answer
214 views

How can we explain the fact that “Bagging reduces the variance while retaining the bias” mathematically?

I am able to understand the intution behind saying that "Bagging reduces the variance while retaining the bias". What is the mathematically principle behind this intution? I checked with few experts ...
1
vote
1answer
146 views

Bias-variance decomposition of MSE

I'm trying to decompose the MSE into the bias and variance terms and have done the following: $$MSE = \frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y_i})^2$$ $$E(MSE) = E\left[\frac{1}{n}\sum_{i=1}^{n} (y_i - ...
0
votes
0answers
37 views

What machine model can balance bias and variance trade-off at testing time?

I wonder if there is a machine learning method can be trained to converge, and then balance the bias-variance trade-off during testing time with some hyperparameters? For example, K nearest neighbor ...
1
vote
0answers
132 views

Cross validation and the Bias Variance trade-off

So I know that there have been a lot of questions about this topic but I try to understand it from a bit more theoretical/mathematical point of view. I have some basic questions of how cross-...
2
votes
0answers
46 views

Bias variance tradeoff when the estimated target function is also random

I'm interested to understand "bias variance tradeoff" notion in a different setting than usually presented. In a setting where target $f$ (see the map $f$ below) is a random map rather than ...
0
votes
0answers
12 views

Isn't the bias-variance trade-off of an estimator sort of a “loose” analogy for what we're doing in machine learning?

I feel like the analogy between the bias-variance trade-off of an estimator and what we typically think of as the bias variance trade-off in machine learning is actually sort of loose. In machine ...
1
vote
1answer
43 views

What to do if recombination of independent variables cause multicollinearity issue?

Let's say you use a regression that has either: 1) interaction variables or 2) polynomials. When using those features you may run into multicollinearity issues. Do you know how to resolve this issue?...
1
vote
0answers
31 views

Estimate of population mean that minimises squared error

When reading Susan Athey's lecture slides here, I was confused by her claim that the population mean estimate for a given sample that minimises squared prediction error was not the sample mean, but ...
2
votes
1answer
67 views

Tradeoffs of robust mean measures (trimmed, Huber, cosh, etc)

After recently having delved into the world of robust measures (for location, mean being the classical case), I have had difficulty understanding robust measures' core dynamic. Basically, what are ...
3
votes
1answer
240 views

Why does bagging increase bias?

In machine learning, why does bagging increase bias? I've read that using less data would lead to a worse estimate of the parameters, but isn't the expected value of the parameter constant regardless ...
1
vote
2answers
110 views

multicollinearity resulting in high variance

Section 8.7.1 of Elements of Statistical Learning talks about high variance in a classification tree due to high correlation between features. What is the intuition behind this? I would think that ...
1
vote
0answers
33 views

What is the relation of bias in the Bias-Variance trade-off to underfitting

In machine learning literature, people often talk about the bias-variance trade-off as well as overfitting and underfitting in the same paragraph. However, in these contexts, bias and variance comes ...
1
vote
0answers
47 views

Bias-Variance terminology for loss functions in ML vs cross-validation — different things?

I am a bit confused about the use of variance and bias across the machine learning and statistical learning literature. In particular, the bias-variance trade-off arises from the fact that one can ...
1
vote
0answers
41 views

regression model can't achieve low bias and low variance at the same time

I am running a RandomForestRegressor model on a dataset and it seems it can NOT achieve low bias and low variance at the same time. So I suspected that the input (independent) variables are not ...
4
votes
0answers
95 views

Is there a theoretical reason why simple models perform better than complex models on time series forecasting tasks?

Empirically, simple forecasting methods such as damped trend exponential smoothing, STL, or even random walks typically outperform more complex models such as higher order ARIMA models or ML based ...
1
vote
0answers
84 views

Can Bias-Variance Decomposition be applied to find MSE of a linear model?

I'm trying to apply the Bias-Variance Decomposition to the produced estimates of a linear model. X = [0,1,2,3,4,5,6,7,8,9] θ = [2,15,27,36,48,57,63,73,80,95] Linear model = 5 + 10x Estimated ...
1
vote
0answers
135 views

What is “Entropic Capacity”?

I found this term on the Keras blog website, quoted below Your main focus for fighting overfitting should be the entropic capacity of your model --how much information your model is allowed to ...
3
votes
2answers
967 views

Does more training data help lower the bias of a high bias model?

It is clear that more training data will help lower the variance of a high variance model since there will be less overfitting if the learning algorithm is exposed to more data samples. However, ...
2
votes
1answer
60 views

High complexity random forest always performs best on test data

(I am new to machine learning so please bare with me) I am using Random Forest Regression algorithm but I am seeing interesting results. I randomly split data into validation set, test set, and ...
1
vote
0answers
46 views

Bias-Variance Tradeoff when using Oversampling Technique

Oversampling techniques (e.g. SMOTE) are often used when target values are not approximately equally represented. How does this technique affect bias and variance of the predictive model that is ...
2
votes
1answer
56 views

Are the bias term b and the Bias in “Bias–variance tradeoff” the same thing?

In ADALINE algorithm , with $y=x*w+b$, where $x$ is the feature vector of a sample and $w$ is the weight vector, the update rule (SGD) for the bias $b$ is: $b \leftarrow b + \eta(o - y)$. With ...
1
vote
0answers
380 views

Relation between train and test MSE in case of low Variance

maybe that is a dublicate, but I cant find an answer on my specific question. I am thinking about the bias-variance trade-off. It is known that, if I have a large variance but low bias I am ...
13
votes
2answers
2k views

Why is best subset selection not favored in comparison to lasso?

I'm reading about best subset selection in the Elements of statistical learning book. If I have 3 predictors $x_1,x_2,x_3$, I create $2^3=8$ subsets: Subset with no predictors subset with predictor $...
1
vote
1answer
293 views

Bias-variance decomposition of squared error

In the equation for expected test error we are summing the variance of the estimated function's bias and the variance of its error. I am not quite sure why we are also summing the variance of the ...
4
votes
1answer
1k views

When is it better to use Multiple Linear Regression instead of Polynomial Regression?

In the course I've just learnt Multiple Linear Regression and Polynomial Regression. Why would you ever use Multiple Linear Regression when Polynomial Regression will always fit the data better?
2
votes
1answer
41 views

What is better? x-times repeated n-fold cross validation or repeated (xn)-fold cross validation?

x-times repeated n-fold cross validation has a higher bias than basic n-fold cross validation (without repetitions). If I want to reduce variance but maintain a low bias, isn't it probably better to ...
2
votes
1answer
102 views

Bias-Variance Decomposition Analysis

I understand how the bias-variance decomposition was done, but I'm not sure what the author means when he says "Unless the nearest neighbor is at 0, $\hat{y}_{o}$ will be smaller than f(0) in this ...
3
votes
1answer
274 views

Interpreting bias variance with accuracy in learning curves

I plotted the accuracy of a Decision Tree model with varying depths. I see that as the depth increases, the delta between the training and test set starts to increase to a point where they never ...
0
votes
0answers
94 views

Bias variance dilemma (derivation in Haykin's Neural Networks)

I have a question regarding a certain derivation of the bias variance dilemma. Generally, I guess I have understood the derivation in, e.g., Geman's Paper, or in books like Bishop's Pattern ...
3
votes
2answers
239 views

Bagging vs pasting: bias-variance tradeoff

In the Hands-On ML with Scikit-Learn book, it states that, ...bagging ends up with a slightly higher bias than pasting, but... the ensemble's variance is reduced. I am a bit confused about this ...
1
vote
1answer
54 views

Variance-bias trade-off equation

If we consider the variance bias trade off equation, stated as: $$\newcommand{\Var}{{\rm Var}} E(y_0-\hat{f}(x_0))^2=\Var(\hat{f}(x_0))+({\rm Bias}(\hat{f}(x_0)))^2+\Var(\varepsilon) $$ We assume ...
1
vote
0answers
47 views

How does using control variables in experiments affect the bias-variance trade-off?

Following up on the post, Using control variables in experiments?, I wonder how this connects with the usual bias-variance trade-off? That is, to minimise the MSE one must trade off the gain from ...
7
votes
3answers
2k views

If Mean Squared Error = Variance + Bias^2. Then How can the Mean Squared Error be lower than the Variance

I was reading the Introduction to Statistical Learning. Here it is shown that:- In a later example, the train and test MSE are plotted. I wanted to know if both the bias^2 and variance are positive ...
0
votes
0answers
92 views

Bias-variance decomposition for a statistical test

A statistical estimator can be characterized – among others – by its bias and variance. These are different aspects, but if one is willing to accept a certain loss function (risk) than they can merged;...