Questions tagged [bias-variance-tradeoff]

In predictive modeling, unbiased models can have higher variance, & thus be less accurate. Modelers may prefer some bias to maximize accuracy. Use this tag also for questions about the bias-variance decomposition.

Filter by
Sorted by
Tagged with
0
votes
0answers
28 views

Best sampling method within the normal family

Suppose that we want to make the best Bayesian inference about some value $\mu$ we have some normal prior about it. I.e. $\mu\sim N(\mu_0, \sigma_0^2)$ with known parameters. To do so, we can choose ...
0
votes
0answers
20 views

Determinstic and stochastic sources in the Bias-Variance decomposition? [on hold]

The proof of the Bias-Variance decomposition states $\mathbb{E}[f(x)] = f(x)$ since $f$ is deterministic but I'm finding hard to create a situation where $f(x)$ is determinstic and $\hat{f}(x)$ is ...
0
votes
0answers
40 views

Is it possible to have overfitting due to high bias?

Overfitting is usually associated with high variance, whereas underfitting is associated with high bias. But one of my professors at uni mentioned that overfitting might be caused by high variance and/...
1
vote
2answers
40 views

Does the MSE values of regression coefficients sum up to the MSE value of the regression model in which the regression coefficients are included?

I think either i dont understand something or i try to mix something that are different things. The mse value of a regression coefficients tells me how good i estimated the coefficent. Does it mean ...
5
votes
1answer
94 views

When does the underfitted regression model have more precise coefficient estimates?

Say we have a full regression model \begin{align*} \mathbf{y} &= \mathbf{X} \boldsymbol{\beta} + \boldsymbol{\epsilon}\\ &= \mathbf{X}_p \boldsymbol{\beta}_p + \mathbf{X}_r \boldsymbol{\beta}...
0
votes
0answers
18 views

Why i get the same MSE value for two least square models that differ in one explanatory variable?

I have two ols-regression models that just differ in one variable. It means that one model have the same variables like the other plus an explanatory variable more. I estimated both models on a train ...
0
votes
1answer
24 views

How to calculate Bias and Variance to get the MSE value step by step?

I want to compute my MSE value for a forecast step by step for test set. For me the Bias is: Bias = mean(predicted values - actual values) Variance = mean((predicted values- actual values)^2) ...
0
votes
0answers
27 views

Machine Learning Model Evaluation

If a model is overfitted that means decent gap between training curve and testing/validation curve but achieves good precision and recall score,does that still indicate that the model is decent?
1
vote
0answers
35 views

Does bias in regression coefficients affect the prediction?

Goal is to create ols model for out of sample prediction for log(wages). Theory say I could have a sample selection bias. So I choose the heckit method to correct for it. The correction term lambda (...
1
vote
0answers
27 views

Do unbiased regression coefficents yield better prediction?

I ask myself if a have a omitted variables bias in my regression modell the coefficients of the model are biased so the mse growth because this coefficents are biased right? So does it mean if i ...
0
votes
0answers
23 views

I understand over-fitting in RandomForest algorithm is taken care by bootstrapping & bagging but what happens if we prune the trees and apply bagging? [duplicate]

According to Random Forest algorithm, tress are not supposed to be pruned intuitively, don't bagging on pruned models give a better final model than the one without pruning?
2
votes
0answers
53 views

Bias Variance Decomposition 2.7 in Elements of Statistical Inference

I try to derive 2.7 from the book. I expose my demonstration $E_\tau[(y_0-\hat{y}_0)^2]=E_\tau[y_0^2]-2E_{\tau}[y_{0}\hat{y_{0}}]+E_{\tau}[\hat{y_{0}}^{2}]$ $= y_{...
0
votes
0answers
20 views

Variance-bias trade off in classification and regression trees

I know what the variance/bias trade off is when I am talking about regression problems. Also in this context i understand the technical derivation. But I don't have an idea of the variance-bias trade ...
2
votes
0answers
30 views

Mean Squared Error as quantifier of the Bias-Variance tradeoff

I have acquired the impression that many of the people doing statistical work, will prefer a biased estimator $\hat b$ to an unbiased one $\hat \beta$, if the former has lower Mean Squared Error. This ...
3
votes
2answers
70 views

Where does linear regression fit into the bias-variance tradeoff?

In ISL, the concept of the bias-variance tradeoff is presented with the rule of thumb that simple models will have high bias and that complex models will have high variance. Given this idea, I would ...
0
votes
2answers
107 views

Variance/bias trade off regularisation penalty - why does it take this form?

In machine learning, if we estimate weights using a loss function $$L(W) = ||Y-F_W(X)||^2$$ (where $W$ is a weight matrix) we may add a "regularisation penalty" to control for the "variance/bias ...
3
votes
2answers
162 views

Bias-variance tradeoff associated with cross validation methods

I was reading about the bias-variance tradeoff associated with cross validation methods on James et al, Introduction to Statistical Learning (Page 183-184). When we perform LOOCV, we are in effect ...
0
votes
0answers
23 views

Issues when converting panel to cross-section? (i.e. $(y_{it},x_{it}) \rightarrow (y_{i},x_{i})$)

Let's say we've got a panel data set of individuals $i$ over time $t$, with their inputs $x_{it}$ and outputs $y_{it}$. We want to relate inputs to outputs, but the problem is that the timing of ...
1
vote
0answers
67 views

Bias-Variance decomposition for non-squared loss

While the Bias-Variance decomposition of the squared loss is part of any introductory ML class, I am curious to know if similar decompositions can be done for other loss functions, e.g., cross entropy?...
2
votes
1answer
104 views

bias variance tradeoff — properties that do not follow

Going through this lecture note on bias-variance trade-off, I didn't follow the latter part of this paragraph. It shows the common situation in practice that (1) for simple models, the bias ...
3
votes
1answer
121 views

Frequentist vs. Bayesian bias-variance decomposition

Iv'e read the answer to this related question and still have some issues. Suppose that given some data $X$, we want an estimator $\hat{\theta}$ for some parameter $\theta$. A common approach is to ...
2
votes
0answers
47 views

Is there any case in which unbiased but larger MSE estimator preferred to biased and smaller MSE one?

Let saying we are interested in a population mean $\mu$ and we have two estimators $\hat{\mu}_{n}^b$ and $\hat{\mu}_{n}^{u}$ defined on $n$ samples such that $\hat{\mu}_{n}^b$ : biased (i.e, $\mathbb{...
0
votes
1answer
21 views

Is variance the only cause of overfitting in any Machine Learning Algorithm?

I am used to associating high variance with overfitting - I don't know how else to think of the physical manifestation of variance wrt an algorithm. Also, the only cause of overfitting seems to be ...
3
votes
1answer
369 views

How can we explain the fact that “Bagging reduces the variance while retaining the bias” mathematically?

I am able to understand the intution behind saying that "Bagging reduces the variance while retaining the bias". What is the mathematically principle behind this intution? I checked with few experts ...
1
vote
1answer
160 views

Bias-variance decomposition of MSE

I'm trying to decompose the MSE into the bias and variance terms and have done the following: $$MSE = \frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y_i})^2$$ $$E(MSE) = E\left[\frac{1}{n}\sum_{i=1}^{n} (y_i - ...
0
votes
0answers
49 views

What machine model can balance bias and variance trade-off at testing time?

I wonder if there is a machine learning method can be trained to converge, and then balance the bias-variance trade-off during testing time with some hyperparameters? For example, K nearest neighbor ...
1
vote
0answers
149 views

Cross validation and the Bias Variance trade-off

So I know that there have been a lot of questions about this topic but I try to understand it from a bit more theoretical/mathematical point of view. I have some basic questions of how cross-...
2
votes
0answers
49 views

Bias variance tradeoff when the estimated target function is also random

I'm interested to understand "bias variance tradeoff" notion in a different setting than usually presented. In a setting where target $f$ (see the map $f$ below) is a random map rather than ...
0
votes
0answers
13 views

Isn't the bias-variance trade-off of an estimator sort of a “loose” analogy for what we're doing in machine learning?

I feel like the analogy between the bias-variance trade-off of an estimator and what we typically think of as the bias variance trade-off in machine learning is actually sort of loose. In machine ...
1
vote
1answer
44 views

What to do if recombination of independent variables cause multicollinearity issue?

Let's say you use a regression that has either: 1) interaction variables or 2) polynomials. When using those features you may run into multicollinearity issues. Do you know how to resolve this issue?...
1
vote
0answers
31 views

Estimate of population mean that minimises squared error

When reading Susan Athey's lecture slides here, I was confused by her claim that the population mean estimate for a given sample that minimises squared prediction error was not the sample mean, but ...
2
votes
1answer
78 views

Tradeoffs of robust mean measures (trimmed, Huber, cosh, etc)

After recently having delved into the world of robust measures (for location, mean being the classical case), I have had difficulty understanding robust measures' core dynamic. Basically, what are ...
3
votes
1answer
301 views

Why does bagging increase bias?

In machine learning, why does bagging increase bias? I've read that using less data would lead to a worse estimate of the parameters, but isn't the expected value of the parameter constant regardless ...
1
vote
2answers
141 views

multicollinearity resulting in high variance

Section 8.7.1 of Elements of Statistical Learning talks about high variance in a classification tree due to high correlation between features. What is the intuition behind this? I would think that ...
1
vote
0answers
34 views

What is the relation of bias in the Bias-Variance trade-off to underfitting

In machine learning literature, people often talk about the bias-variance trade-off as well as overfitting and underfitting in the same paragraph. However, in these contexts, bias and variance comes ...
1
vote
0answers
50 views

Bias-Variance terminology for loss functions in ML vs cross-validation — different things?

I am a bit confused about the use of variance and bias across the machine learning and statistical learning literature. In particular, the bias-variance trade-off arises from the fact that one can ...
1
vote
0answers
51 views

regression model can't achieve low bias and low variance at the same time

I am running a RandomForestRegressor model on a dataset and it seems it can NOT achieve low bias and low variance at the same time. So I suspected that the input (independent) variables are not ...
5
votes
0answers
108 views

Is there a theoretical reason why simple models perform better than complex models on time series forecasting tasks?

Empirically, simple forecasting methods such as damped trend exponential smoothing, STL, or even random walks typically outperform more complex models such as higher order ARIMA models or ML based ...
1
vote
0answers
87 views

Can Bias-Variance Decomposition be applied to find MSE of a linear model?

I'm trying to apply the Bias-Variance Decomposition to the produced estimates of a linear model. X = [0,1,2,3,4,5,6,7,8,9] θ = [2,15,27,36,48,57,63,73,80,95] Linear model = 5 + 10x Estimated ...
2
votes
0answers
155 views

What is “Entropic Capacity”?

I found this term on the Keras blog website, quoted below Your main focus for fighting overfitting should be the entropic capacity of your model --how much information your model is allowed to ...
3
votes
2answers
1k views

Does more training data help lower the bias of a high bias model?

It is clear that more training data will help lower the variance of a high variance model since there will be less overfitting if the learning algorithm is exposed to more data samples. However, ...
2
votes
1answer
67 views

High complexity random forest always performs best on test data

(I am new to machine learning so please bare with me) I am using Random Forest Regression algorithm but I am seeing interesting results. I randomly split data into validation set, test set, and ...
1
vote
0answers
50 views

Bias-Variance Tradeoff when using Oversampling Technique

Oversampling techniques (e.g. SMOTE) are often used when target values are not approximately equally represented. How does this technique affect bias and variance of the predictive model that is ...
2
votes
1answer
59 views

Are the bias term b and the Bias in “Bias–variance tradeoff” the same thing?

In ADALINE algorithm , with $y=x*w+b$, where $x$ is the feature vector of a sample and $w$ is the weight vector, the update rule (SGD) for the bias $b$ is: $b \leftarrow b + \eta(o - y)$. With ...
1
vote
0answers
406 views

Relation between train and test MSE in case of low Variance

maybe that is a dublicate, but I cant find an answer on my specific question. I am thinking about the bias-variance trade-off. It is known that, if I have a large variance but low bias I am ...
13
votes
2answers
2k views

Why is best subset selection not favored in comparison to lasso?

I'm reading about best subset selection in the Elements of statistical learning book. If I have 3 predictors $x_1,x_2,x_3$, I create $2^3=8$ subsets: Subset with no predictors subset with predictor $...
1
vote
1answer
309 views

Bias-variance decomposition of squared error

In the equation for expected test error we are summing the variance of the estimated function's bias and the variance of its error. I am not quite sure why we are also summing the variance of the ...
4
votes
1answer
1k views

When is it better to use Multiple Linear Regression instead of Polynomial Regression?

In the course I've just learnt Multiple Linear Regression and Polynomial Regression. Why would you ever use Multiple Linear Regression when Polynomial Regression will always fit the data better?
2
votes
1answer
42 views

What is better? x-times repeated n-fold cross validation or repeated (xn)-fold cross validation?

x-times repeated n-fold cross validation has a higher bias than basic n-fold cross validation (without repetitions). If I want to reduce variance but maintain a low bias, isn't it probably better to ...
2
votes
1answer
108 views

Bias-Variance Decomposition Analysis

I understand how the bias-variance decomposition was done, but I'm not sure what the author means when he says "Unless the nearest neighbor is at 0, $\hat{y}_{o}$ will be smaller than f(0) in this ...