Questions tagged [bias-variance-tradeoff]

In predictive modeling, unbiased models can have higher variance, & thus be less accurate. Modelers may prefer some bias to maximize accuracy. Use this tag also for questions about the bias-variance decomposition.

Filter by
Sorted by
Tagged with
2 votes
2 answers
25 views

If we reduce size of training dataset does it decreases bias?

I'm a newbie and learning ML. I've a doubt, normally we know we should increase the size of training dataset or should add more data to reduce variance (fairly understood why). Now variance has ...
user avatar
0 votes
0 answers
20 views

Is deep double descent important in practical contemporary CNNs?

Deep double descent is an empirically observed phenomenon that happens with contemporary neural networks. Its essence is that often, increasing the model complexity first leads to the test loss ...
user avatar
  • 133
3 votes
1 answer
38 views

Bias Variance tradeoff in neural networks

Large neural networks have low bias and high variance. Training on large datasets greatly reduces the variance allowing them to fit complicated functions. My question is why they seem to have much ...
user avatar
  • 168
0 votes
0 answers
23 views

What does the subscript in an Expectation operator mean in the context of the bias-variance tradeoff?

I've been looking at the derivation of the bias-variance decomposition for a few months now and though I understand its implications, I am still confused by the notation used in the literature. ...
user avatar
0 votes
0 answers
11 views

Bias, Variance and Bagging and what it means in relation to representational complexity

Re-looking at some basic ML algorithms bagging comes up along with the Bias-Variance trade off. I am confused on how bagging relates to representational capacity. Based upon the arguments I have read, ...
user avatar
0 votes
1 answer
45 views

Robust distance weighted mean

Given a data sample $\{x_i\}_1^n$, instead of hard omitting outliers by e.g. trimming, one can form a weighted average where we soft penalize observations out in the tails. \begin{align} \mu = \frac{...
user avatar
0 votes
1 answer
37 views

Welford vs Bayes?

To incrementally estimate the mean and standard deviation of some data one can use an algorithm such as Welford’s algorithm or Bayesian updating by using the likelihood, a conjugate prior and ...
user avatar
2 votes
1 answer
43 views

Cross-validation: error estimation and bias

When obtaining the error estimation of a model over a dataset using k-fold cross-validation, lower values of the error estimation necessarily imply a lower bias? Are both concepts, error estimation ...
user avatar
0 votes
0 answers
9 views

Descriptive statistical metrics of RMSE, MAE and R2 on cross-validation and holdout testing

I've tuned 100 models of each one LASSO (Least Absolute and Selector Operator) and Random Forest regression methods. I mean, to predict a hypothetical dependent variable, I've tuned 100 models for ...
user avatar
0 votes
0 answers
41 views

bias and variance decomposition derivation trick

I am following the derivation from here. My question is about the first trick, where the author claims that: \begin{equation} \begin{aligned} E_{\mathbf{x}, y, D}\left[\left[h_{D}(\mathbf{x})-y\right]^...
user avatar
1 vote
0 answers
26 views

When do control variables increase precision?

Suppose we're interested in the effect $\beta$ of a treatment $D$. To increase the precision of our estimate (ie., reduce the variance of $\hat{\beta}$), we can include a control variable $X$ that ...
user avatar
1 vote
1 answer
36 views

Reasons to prefer low bias with higher variance over the alternative (and vice versa)

I am trying to understand the bias-variance tradeoff in practice. I have read several related questions and answers, but still have a few questions: Assume we are estimating a structural equation ...
user avatar
1 vote
0 answers
28 views

Could I pick the best model based on the bias-variance tradeoff?

Usually would I pick the best performing model according to accuracy, or another evaluation method, on the validation dataset. But is this viable to chose the best model according to bias-variance ...
user avatar
1 vote
0 answers
11 views

does assumptions effect the bias or variance?

in machine learning text it is often said that assumptions affect bias like the following text from Kevin Murphy: "Given the large variety of models in the literature, it is natural to wonder ...
user avatar
  • 21
5 votes
2 answers
141 views

Does bias eventually increase with model complexity?

Does bias eventually increase with model complexity? Reasoning behind the question: If I understand it correctly, "bias" measures the discrepancy between the expected value of our model's ($...
user avatar
  • 105
7 votes
4 answers
940 views

Why do we say that the model has a high variance when variance is actually the measure of spread of the data and not some property of the model?

I am trying to understand the difference between bias-variance and overfitting-underfitting. If a modal overfits the data it means that it will not generalize well on new data because it over learns ...
user avatar
1 vote
1 answer
97 views

Poor model performance on certain out-of-sample data

We're noticing poor model performance on certain out of sample products. We have trained a ML model on about 2000 different products in a few markets. Our predictors include a) product ...
user avatar
  • 510
0 votes
1 answer
52 views

Bias-variance trade-off in linear regression [closed]

As it’s understood, in the bias-variance trade-off, variance refers to overfitting of the model and it examines the variability of output predictions. Suppose we have a simple dataset with one ...
user avatar
0 votes
0 answers
13 views

Variance analysis on boosting approachs, Is there any guarantee that boosting will not worse the weak learner variance or even get it better?

I'm looking for a theorical justification why boosting does work in pratice, I'm almost sure that this reduces the bias of their weak learners (assuming all weak learners have the same bias), but I ...
user avatar
0 votes
0 answers
30 views

Checking understanding about a derivation of bias and variance in the context of generalization in course notes

Sorry for the long image. This derivation of bias and variance was given in publicly available course notes (here) on pages 3 and 4. I understand the first derivation. They showed that y* was the best ...
user avatar
4 votes
1 answer
94 views

Regularization and Shrinkage : Theoretical Advantages vs. Empirical Advantages [duplicate]

I have the following question about the theoretical advantages vs. the empirical advantages of regularization (i.e. shrinkage). As far as I understand, this is the general idea behind regularization: ...
user avatar
  • 5,884
0 votes
0 answers
25 views

Does overfitting mattters if you do good on test and backtesting?

Say you build a model performing <grid/random/hyperband/bayesian/anyothermethod> search hiperparameter tuning with K-FOLD crossvalidation, where K>=3. Then you get the best estimator (the one ...
user avatar
  • 101
1 vote
0 answers
100 views

Apart from the Bias-Variance "Decomposition" - is there a Bias-Variance "Proof"?

I am sure at some point, many of us have come across the "Bias-Variance Tradeoff" : The "error" of any "estimator" (e.g an estimator can be considered as a linear ...
user avatar
  • 5,884
3 votes
2 answers
246 views

How does repeated k-fold cross validation identify model instability?

In these threads 1,2,3, cbeleites mentions that in a single k-fold cross validation you cannot tell whether the variance is caused by model instability or using a different test set. Hence, one can ...
user avatar
  • 157
2 votes
0 answers
180 views

Bias-variance trade-off between LDA and QDA w.r.t. dimensionality

Consider the bias-variance trade-off between linear discriminant analysis (LDA) and quadratic discriminant analysis (QDA). Switching from QDA to LDA will generally yield a reduction in variance. The ...
user avatar
1 vote
0 answers
39 views

Bias vs. variance

I have a question about bias/variance trade-off for different competing models. Say one has estimated model A and model B and calculated their respective train and test error. How does one yield an ...
user avatar
  • 27
0 votes
1 answer
362 views

Lasso vs Ridge Regression

My question relates on the Ridge vs Lasso Regression. I know the difference in the cost function (ridge penalizes sum of quadratic coefficients, lasso penalizes sum of absolute value of coefficients). ...
user avatar
  • 27
0 votes
1 answer
259 views

Bias-Variance tradeoff with Clustering algorithms

I'm investigating the bias-variance tradeoff in well-known machine learning algorithms. However, I'm not sure this concept applies in the case of unsupervised methods such as clustering algorithms. Is ...
user avatar
1 vote
1 answer
78 views

Bias variance tradeoff in Gaussian process regression

Can someone explain in simple terms the bias and variance tradeoff of Gaussian Process regression?
user avatar
1 vote
1 answer
52 views

Bias-variance trade-off in case of biased estimators: is the bias zero?

Consider a data generating process (DGP) that is AR(1): $y_t=\varphi_1 y_{t-1}+\varepsilon_t$ with $\varepsilon_t\sim i.i.D(0,\sigma^2)$ for some distribution $D$ with mean zero and variance $\sigma^2$...
user avatar
1 vote
1 answer
30 views

Bias in bias-variance trade-off: how well the model can possibly approximate the DGP?

Consider a data generating process $$Y=f(X)+\varepsilon$$ where $\varepsilon$ is independent of $x$ with $\mathbb E(\varepsilon)=0$ and $\text{Var}(\varepsilon)=\sigma^2_\varepsilon$. According to ...
user avatar
1 vote
1 answer
169 views

why test error and variance has different curve in bias variance trade off graph?

In bias variance trade off graph Bias is the difference between actual and predicted value in training data set so train error (dotted red curve) and bias(red curve ) looks same Variance is the ...
user avatar
  • 135
0 votes
1 answer
56 views

How to find the optimum when using regularization?

Using regularization increases the training error and the validation possibly as well. How do I find the optimum? Still just the optimum of the bias² and variance, like here: Source: https://dziganto....
user avatar
  • 2,207
1 vote
0 answers
25 views

Bias, variance, train and test/validation error + total error

I'm trying to make sense out of the following two plots: Source: https://www.linkedin.com/pulse/bias-variance-tradeoff-machine-learning-satya-mallick/ Source: https://towardsdatascience.com/the-bias-...
user avatar
  • 2,207
0 votes
0 answers
95 views

An Implementation of MSE decomposition to Variance and Bias Squared

Before I describe my question, it is necessary to note a common fact in estimation theory that MSE can be decomposed to Variance and Bias Squared. Depending on whether it is MSE of an estimator or MSE ...
user avatar
  • 21
1 vote
2 answers
271 views

Definition of the bias of an estimator

I'm quite confused about the definition of the bias of an estimator. Suppose we have unknown distribution $P(x, \theta)$, and construct the estimator $\hat{\theta}$ that maps the observed data sample ...
user avatar
  • 11
2 votes
1 answer
54 views

Bias of MLE scales with $1/N$?

I was reading this paper (link) and it gave me some confusion. $P(r|\theta)$ is a distribution that generates sample $r$ based on some Poisson distribution, whose mean and variance are defined as some ...
user avatar
  • 261
1 vote
2 answers
234 views

Why do we prefer unbiased estimators instead of minimizing MSE?

I was thinking about why, usually, $\hat{\sigma}^2=\hat{p}(1-\hat{p})$ is used to estimate the variance in a Bernoulli population instead of $s^2=\hat{p}(1-\hat{p})\frac{n}{n-1}$. $s^2$ is unbiased, ...
user avatar
0 votes
0 answers
10 views

Can I conclude this is overfitting and how can I reduce it?

I try to fit a GAM with mgcv in R for classification. The dependent variabele (WinFlag) is true or false. The independent variables are two continious variabeles (x1 and x2) and one factor variabele (...
user avatar
0 votes
0 answers
57 views

XGBoost Cross Validation - Baseline model performs better on validation data but it performs well below the training performance

I appreciate it if someone guides me on the following situation: I'm trying to decide what parameter set to choose for my XGBClassifier. The dataset has roughly 200'...
user avatar
4 votes
1 answer
260 views

What Cross Validation results actually tell about Bias and Variance?

I am trying to get a deeper understanding of the common ML pipelines and I have some doubts regarding Cross Validation, why do we really use it and what does it really tell us about Bias and Variance. ...
user avatar
  • 86
2 votes
2 answers
129 views

How do I know my model overfits?

We already have multiple questions on overfitting, however, we also observe certain kinds of questions to re-occur regularly that ask how to diagnose overfitting in practice. Let's try giving an ...
user avatar
  • 115k
2 votes
1 answer
97 views

What conditioning assumptions are hidden in the bias-variance tradeoff proof?

I've read through this question and its answers quite a few times. I'm curious to know what conditioning assumptions are used that are hidden from the derivation. Here's what I mean: let's say I have ...
user avatar
  • 3,903
11 votes
1 answer
2k views

Modern machine learning and the bias-variance trade-off

I stumbled upon the following paper Reconciling modern machine learning practice and the bias-variance trade-off and do not completely understand how they justify the double descent risk curve (see ...
user avatar
  • 615
3 votes
1 answer
78 views

Why is REML default if it inflates MSE?

Within the mixed effects model world, REML has become the method of choice in order to correct for the downward bias in variance components. For years, I accepted this rationale without thinking about ...
user avatar
0 votes
1 answer
426 views

Lasso and Ridge Regression: Variance and Bias

I am studying about learning methods in statistics and the regression section explains that the main difference between Lasso and Ridge regressions is the formulation of the regularization term, ...
user avatar
15 votes
2 answers
2k views

Can I (justifiably) train a second model only on the observations that a previous model predicted poorly?

Say I commit the following sins while building a predictive model: I take my dataset and split it into four subsets: Three for training (Train_A, Train_B, and Train_C) and one for validation. I ...
user avatar
  • 155
11 votes
4 answers
2k views

What is meant by Low Bias and High Variance of the Model?

I am new in this field of Machine Learning. From what I get by the definition, Bias: It simply represents how far your model parameters are from true parameters of the underlying population. $$ Bias(\...
user avatar
0 votes
1 answer
34 views

How is the true label 'constant' in the derivation of the bias-variance decomposition

In the derivation of the bias variance decomposition for example on Wikipedia or in this question the following identity is used: $$E[(E[\hat{f}]-\hat{f})(f-E[\hat{f}])]=E[E[\hat{f}]-\hat{f}](f-E[\hat{...
user avatar
  • 111
39 votes
9 answers
7k views

Is overfitting "better" than underfitting?

I've understood the main concepts behind overfitting and underfitting, even though some reasons as to why they occur might not be as clear to me. But what I am wondering is: isn't overfitting "...
user avatar
  • 641

1
2 3 4 5