1
$\begingroup$

I am trying to model the odds of soccermatches in play, based on the odds at start of the match and possesion during the game. My dataset contains:

Start_odd (x1)  Possesion (x2) Market_odd_observed (y)
0.67            80             0.90
0.45            75             0.63 etc

Start_odd is on a scale of 0-1. Possesion is on a scale of 0-100. Market_odd is on a scale of 0-1.

The GAM-model is fitted using mgcv:

Family: gaussian 
    Link function: identity 

    Formula:
    Market_odd_observed ~ s(Start_odd , k = 20) + s(Possesion , k = 20) + ti(Start_odd , 
        Possesion , k = c(10, 10))

    Parametric coefficients:
                 Estimate Std. Error t value Pr(>|t|)    
    (Intercept) 7.394e-01  4.609e-05   16043   <2e-16 ***
    ---
    Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

    Approximate significance of smooth terms:
                              edf Ref.df      F p-value    
    s(Start_odd )             18.87  19.00 288685  <2e-16 ***
    s(Possesion )             18.95  19.00 190429  <2e-16 ***
    ti(Start_odd ,Possesion ) 69.69  75.33  12433  <2e-16 ***
    ---
    Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

    R-sq.(adj) =  0.947   Deviance explained = 94.7%
    -REML = -1.134e+06  Scale est. = 0.0012332  n = 587663

If I plot the residuals by the dependent variable I still see a pattern in the data with a upward slope:

enter image description here

I conclude that there is some bias in the model. The problem is that I can not include the dependent variable as an interaction term since this is the outcome I try to predict. Is it unusual to look at the residuals grouped by the dependent variable?

I have tried to fit a second gam-model with the predictions from the model above as the input. Unfortunately the RMSE is exacly the same and the pattern is still there.

I have also plotted the residuals by the predictions. In that case the bias is not there as can be seen in this plot:

enter image description here

Is there an alternative method to improve the model?

Next I have fitted a catagorical GAM on the winflag of the match (0 or 1). The results are the same as above.

Next I have plotted the observed odds vs predictions:

enter image description here

And transparant:

enter image description here

Next I have grouped_by the errors by observed market odd:

enter image description here

And grouped by prediction:

enter image description here

I expect it is not related to the use of the GAM since there is similar pattern using a neural network. What could be the explanation that the models do not fit this pattern?

Thanks a lot!

I have added an example to illustrate the answer from Aksakal:

library(tidyverse)
library(ggplot2)
library(mgcv)
library(mlbench)

data("BostonHousing")


gam_y <-
  gam(
    medv ~ s(nox) + s(rm) + s(dis) ++s(tax) + s(ptratio) + s(lstat) ,
    method = "REML",
    data = BostonHousing
  )

y_pred <- predict(gam_y)
predictions <-
  cbind(BostonHousing$medv, y_pred, resi = BostonHousing$medv - y_pred)
predictions <- as.data.frame(predictions)
colnames(predictions)[1] <- "medv"

ggplot(predictions, mapping = aes(x = medv, y = resi)) +
  geom_point(alpha = 100 / 100) +
  geom_smooth(method = lm) +
  labs(y = "residual", x = "house price observed (y)") +
  ggtitle("residuals by y")

ggplot(predictions, mapping = aes(x = y_pred, y = resi)) +
  geom_point(alpha = 100 / 100) +
  geom_smooth(method = lm) +
  labs(y = "residual", x = "house price predicted (y)") +
  ggtitle("residuals by y")

enter image description here

enter image description here

$\endgroup$
6
  • $\begingroup$ it's not just the bias you have a problem with but the variance seems to be varying too. your errors are heteroscedastic. it's quite likely the regression suffers from lack of exogeneity, i.e. $cov[X,e]\ne 0$ $\endgroup$ – Aksakal Jun 4 '20 at 15:36
  • $\begingroup$ You could try whitening the errors en.wikipedia.org/wiki/Whitening_transformation $\endgroup$ – user234562 Jun 4 '20 at 16:03
  • $\begingroup$ @ Aksakal thank you. Maybe the variance varies because y is a probability? Close to 0 and 1 there is less room to vary down- and upwards repspectively than around 0.5. Is this a problem? I read about the exogeneity it can be caused by : 1) an omitted variable 2) simultaneity 3) selection bias. I can imagine I have many omitted variables since there can be many influences on a soccer match. Although this is hard to solve. How did you conclude the regression suffers from a lack of exogeneity? Do you have an advice what I should do? Thanks! $\endgroup$ – user2165379 Jun 4 '20 at 18:31
  • $\begingroup$ @ user332577 thank you. Do I understand correctly that this is a form of preprocessing? initially I did centering and scaling although there was no improvement in the model. $\endgroup$ – user2165379 Jun 4 '20 at 18:34
  • $\begingroup$ the residuals are typically defined as observed minus predicted, not the other way. which explains why you have a bias in errors vs observed plots $\endgroup$ – Aksakal Jun 6 '20 at 16:17
0
$\begingroup$

when you work with probabilities consider cross-entropy loss instead of fitting the least squares of deviations (residuals). the most straightforward application of this is logit regression.

For instance, consider a logit link function in your code to combine GLM with GAM.

On residuals $e=y-\hat y$ (observed minus prdicted not the other way!), when you plot them vs the observed $y$, they'll exhibit negative bias. The reason is that when $y\to 1$, then predictions will tend to be below observed. In an extreme case where observed is 1 unless you have an absolutely perfect predictive power you must have $\hat y<y=1$, therefore on the right end you must have positive bias, then with similar consideration for $y\to 0$ on the left end you must have negative bias in the plot.

Generally, in any model $y=f(x)+\varepsilon$ you have the errors baked in the dependent variables, hence the plots $\varepsilon\sim y=f(x)+\varepsilon$ have the erros both in x- and y-axes, thus making the plots correlated (sloped). It is, therefore, preferable to plot $\varepsilon\sim f(x)$, i.e. error vs predicted not observed.

$\endgroup$
11
  • $\begingroup$ @ Aksakal thank you. Initially I was using logistic regression and a logistic GAM for classification on the winflag of the matches (0 or 1). I have used AUC to measure improvements in accuracy. Although my dataset is too small since this resulted in relative much noise compared to the tiny improvements from other variables (not included here above). For that reason I started to fit the observed market odds to reduce the impact from the randomness in the dataset. Do you advice me to do the Hausman test or are you already sure that there is endogeneity? This test would be tough for me to set up. $\endgroup$ – user2165379 Jun 5 '20 at 18:58
  • $\begingroup$ In this case I have an observed market-odd before kick-off of a match and an observed market-odd during the game. I do not look at the game outcome. Maybe the channel could be that both are 'set' by the same marketforces? Good point, I will do the GAM for classification again (with game outcome!) and check if there is the same pattern as above. Since the results were too noisy I did not check this. I will also exclude startodd to see if the pattern is there. PS: if I exclude the interaction term the effect is even much stronger. I will let you know my findings. Thanks $\endgroup$ – user2165379 Jun 5 '20 at 19:14
  • $\begingroup$ Oh, I'm sorry, assumed you use game outcomes. If you're modeling odds with odds pre game then yes, there's a concern not only for endogeneity, but for unobserved heterogeneity too. There must be some persistence carried unobserved between pre- and during game odds. If you don't model it, then it must leak into yoou estimates as bias $\endgroup$ – Aksakal Jun 5 '20 at 19:23
  • $\begingroup$ Ps: there is nothing wrong with looking at the residuals by the dependent variable? If I read about it, it is mostly done by the independent variables. $\endgroup$ – user2165379 Jun 5 '20 at 19:38
  • $\begingroup$ @ Aksakal I have added plots with residuals vs predicted in the post. It can be seen that the bias is not there in that case! I don't understand this difference. Next I have added the plots from the GAM on game outcome instead of observed market odds. These show the same patterns. I suppose we can conclude there is no channel between pre- and during game odds? (Although the observed market odd is still in the calculation of the residual (predicted probability - observed market odd).) $\endgroup$ – user2165379 Jun 6 '20 at 13:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.