2
$\begingroup$

The model is from sports.

I'm trying to predict the win percentage of each team in the next season based on information available before that season. I have about 240 observations of the numbers of wins in various seasons for various teams, along with the corresponding values of possible predictors. I want to build a multilinear regression model using this data and I want to demonstrate the success or lack of success of whatever model I come up with by showing how my model's predictions compare to those of an "expert" in one or two most recent seasons.

In my model, I have 4 potential predictors X1, X2, X3, X4. Two of them, say X1 and X2, are a priori reasonable approximations of the target variable, though obtained with different assumptions, so I expected and found that they are highly correlated. I've read some on multicollinearity, so I understand this causes a variety of problems when one tries to include them in the same model.

It would make sense to me that the model which includes X1, X3, and X4 and and the model which includes X2, X3, and X4 would be best, assuming that X3 and X4 are significant, etc. but this also seems a bit unsystematic/arbitrary.

I wonder if you could offer some advice on model selection in this situation, possible methods to use, etc. Particularly, I'm confused about what the appropriate way to partition the data is. Obviously if I'm going to be demonstrating the model on a recent season, I should exclude that recent season from the data I train the model on. But should I also set aside some of the data for validation?

As you can probably tell, I'm new to modelling. Edit:

More details, in light of Rolando's answer:

X1 is that team's win percentage the previous season, while X2 is an estimation of next season's winning percentage based on solving the formula for MP/MW for winning percentage and assuming that MP/MW next season will be similar to what it was the previous season. So one estimate is based on an assumption of "continuity" of winning percentage from one season to the next and the other is based an assumption of the "continuity" of this MP/MW quantity from one season to the next, where MP/MW is supposed to be an estimate of the efficiency of the team's organization, basically.

Another question: Supposing you have a target variable Y and some number of predictor variables X1, ..., Xn which are already predictions of Y, it's reasonable to think that we can aggregate these predictions to come up with a better estimate of Y. What are some simple methods of doing this which I should read about? Thanks.

$\endgroup$
1
$\begingroup$

Two of them, say X1 and X2, are a priori reasonable approximations of the target variable, though obtained with different assumptions

This was something of a red flag for me. You'll need to decide how much use it is to predict WPCT using these 2 variables that sound almost like different versions of WPCT itself. You don't want to stray too close to tautology: "teams that are successful tend to win a lot of games." Maybe you can tell us more about how these 2 are constructed. Are they truly alternate indicators of team success? I can imagine variables such as % of games in which team is leading after a certain portion of the game, or number of winning streaks of at least 3 games, that might constitute overly "incestuous" predictors of this type. On the spectrum from exogenous to endogenous, these would be too close to the latter to make for a useful model.

I like where you're going with crossvalidation--I think your plan is sound, though of course one can always get more rigorous, more intricate. If you're interested in more advanced methods you could look up k-fold crossvalidation, jackknifing, or bootstrapping, for instance.

$\endgroup$
2
  • $\begingroup$ I added some to the original post to provide more details. A question, though: what do you mean when you say the plan is sound? The plan just to train it on a set which excludes the set I intend to demonstrate on? Or the idea to make a validation set? $\endgroup$ – mike Mar 3 '12 at 18:37
  • $\begingroup$ I meant the former. Actually I'm rethinking that: @Frank Harrell has argued on this site that this simple partitioning into training and test sets can lead to badly distorted conclusions unless one has very large sample sizes. He was dealing with cases involving dozens of predictors, whereas you have just a few, so I suppose you need an in-between solution. Depending on your tolerance for error in estimating parameters, you may need to try a crossvalidation method that uses a good deal more than 2 subsamples, and maybe even hundreds--as via bootstrapping. $\endgroup$ – rolando2 Mar 3 '12 at 21:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.