5
$\begingroup$

I got these results after fitting a model:

enter image description here

In the above picture X axis shows the fitted values and Y axis shows the truth values.

We can see bias although I tuned reg_alpha & reg_lambda (i used XGBoost and also tuned learning_rate, n_estimators, max_depth, min_child_weight, gamma=0.1, etc.)

So how to measure bias? Is this small or large bias? Or can we use the model or try tuning the parameters again?

Improve this question
$\endgroup$
4
  • 7
    $\begingroup$ You cannot measure bias without knowing the truth, as the bias is a measure of deviation of the average result of your model from the truth. It is a conceptual idea, and cannot be measured in practice. $\endgroup$
    – Matthew Drury
    Jun 29, 2017 at 15:08
  • $\begingroup$ In what way do you "see" bias in this plot? Could you be confusing this word with some other statistical phenomenon such as regression to the mean? $\endgroup$
    – whuber ♦
    Jun 29, 2017 at 19:14
  • $\begingroup$ @whuber my english is bad i can not explain my seeing I just see that high values are more above red line and low values more under the red line I hope you can see thr same $\endgroup$
    – Edward
    Jun 29, 2017 at 19:51
  • 2
    $\begingroup$ Yes, that is called regression to the mean. It is measured by the correlation coefficient. (This subtle relationship was discovered by Francis Galton, something he is justifiably famous for.) $\endgroup$
    – whuber ♦
    Jun 29, 2017 at 19:54

3 Answers 3

Reset to default
3
$\begingroup$

You can measure the bias-variance trade-off using k-fold cross validation and applying GridSearch on the parameters. This way you can compare the score across the different tuning options that you specified and choose the model that achieve the higher test score.

high bias vs low bias

I also encountered a useful reference about bias-variance trade-off. In the section 3.3 - Analytical Bias and Variance the author make an analogy with the error calculation formula of a kNN algorithm:

"In the case of k-Nearest Neighbors we can derive an explicit analytical expression for the total error as a summation of bias and variance:"

error calculation analogy http://scott.fortmann-roe.com/docs/BiasVariance.html

Improve this answer
$\endgroup$
2
  • $\begingroup$ of course I've done cross-validation & train test split & tuning parameters by gridseach $\endgroup$
    – Edward
    Jun 29, 2017 at 15:33
  • 1
    $\begingroup$ If you already have done cross-validation & gridsearch, you are probably getting closer to the best result as jibiel said. I suppose what best approximate to what you want to to mean with "bias measure" is the error on the test set. $\endgroup$
    – Giovanni Zapponi
    Jun 29, 2017 at 16:23
1
$\begingroup$

The difference between your predictions and the real values is the absolute error (and not bias). This error is composed of the variance of your model, the bias of the model, and the irreducible error (that you cannot reduce).

I guess that what you want to do is to measure the error and optimize your model to minimize this error. Look for a function in your toolkit to measure the error in your predictions (e.g. MSE). However, if you are already optimizing your model using cross-validation, you are probably getting closer to the best result (it depends on how you are searching your parameters). Maybe you could be more specific about how you are choosing these parameters?

Improve this answer
$\endgroup$
2
  • $\begingroup$ I know how to measure RMSE, MAE or R2. I donn't know how to measure bias)) $\endgroup$
    – Edward
    Jun 29, 2017 at 15:53
  • $\begingroup$ Bias is not any exactly a performance measure. Bias refers to the error that you make when approximating your prediction function with your specific model (in this case an ensemble of trees). Maybe your are confused with some other measure? $\endgroup$
    – jibiel
    Jun 29, 2017 at 16:03
1
$\begingroup$

I wrote an article on this exact subject here. This should give you some clues about how to tweak parameters such as learning rate, lambda, subsample and number of rounds. This is what a bias variance plot looks like for learning rate: enter image description here

As others have mentioned, this is not an analysis you can typically do for real datasets as we have to know about how the data set was generated. Having said that, it useful to observe how these different xgboost parameters interact with a synthetic dataset to understand at a high level how they manipulate the model.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.