How to find how independent variables affect dependent variables

Question

I'm a bit of a novice at maths and am trying to get my head around a problem.

I have 3 independent variables which affect 1 dependent variable. I want to create a 4D model which will give me the 4th dimension when I give it an (x, y, z) triplet.

I am programming in Java and already have a regression function which will take a set of independent variables and give me coefficients of those independent variables which best fit the data supplied.

What I am trying to figure out is which independent variables to use.

I have tried various cubic functions, with independent variables something like: {1 + x + y + z + xx + xy + xz + yy + yz + zz + xxx + xxy + xxz + yyy + yyx + yyz + zzz + zzy + zzx + xyz}

Then when the resulting model looked a bit wrong I thought, ah maybe the x and y values don't mean anything when multiplied, so took out the variables where x and y were together. Now it still isn't right and I'm worried I'm just going about it in entirely the wrong way. Maybe there's an exponential in there somewhere?

Is there some mathematical method to finding exactly which variables I should be using?

Cheers!

---

The data I already have is like this. It's to do with calculating final scores in a cricket game based on which batsmen are in and how far through the game we are, the final score is the dependent variable:

X axis ranges from 0 to 10 inclusive (the order of the first batsman in). Y axis ranges from 0 to 10 inclusive (the order of the second batsman in). Z axis ranges from 0 to 19 inclusive (the over we are in, basically means how far through the game we are).

The lower x and y are, the higher the final score will be, as the team have better batsmen still in. The higher the over (when x and y are the same), the higher the final score will be, because the batsmen have lasted longer and so should have their eye in.

I guess it's "what I expect the dependent variable to be" which is the question. How does each parameter effect the final score. I can post some sample data if you want.

I have calculated a data point for each (X, Y, Z) combination, so can't get more. I have data from all the cricket games, and each data point is the average final score of games where this situation has occurred. Some situations ((x, y, z) triples) are far more likely to occur (in more average games) and have been weighted in the regression function accordingly.

Answer 1

There is no exact science behind including or omitting explanatory variables, as adding additional variables changes the meaning of your model. By adding additional variables, what you basically say is: How does x affect y, keeping all other explanatory variables constant?

Apart from paying attention to the p-values, you should also have a look at the $ R^2 $, which measures which portion of the variance in the population is explained by your model.

Unless you are interested in coding the regression algorithm yourself, you might want to have a look at the excellent Weka Java Library or R, an open source statistics software.

Answer 2

It is very hard to determine the relationship between variables without having an underlying, prior assumption on how their relationship works. I am not sure about the problem at hand, but seeing that you assume a linear relationship of data points (be it variables or their products) I take it we are looking at a linear relationship.

Indeed there are several problems which may make your regression and forecast downright invalid. Another issue is your small sample size. The result of this is that you have to be even more careful in selecting the right technique.

First, your regression model must be correct for your regression to be correct. This seems tautological, but it entails that no variable should be missing in the equation, if it also has an impact (more correctly, is correlated) with the other variables.
You can immediately see the issue here: It is almost certain that there are more things to winning a cricket game than just the order of batters. For example if the breakfast of the second batter influences the game outcome AND his standing on the batting list, it technically must be in your equation. Otherwise your estimate of the influence of the batting order on the game outcome will be biased, ie wrong.
On the other hand if you have a variable, like weather (maybe), which does not influence the batting order but does influence the game outcome, then your estimate of the influence of the batting order will still be correct. However your forecast of the game outcome BASED on these estimates will be off and should be considered more of a indication than a concrete fact.

You will probably say, well damn, how we ever gonna get a good model then? Good point, almost never will we be exact in this. However there are ways to mitigate the problem or at least diagnose it. We somewhat get around missing variable bias if the missing part is somehow more or less random. In fact we concede anyway that we can not correctly model reality, but as long as the "wrongness" has certain properties, we are still somewhat "right".

The key is that we add the error term in this, a random factor which is normally assumed to be, well, "normal" distributed and centered around zero, though this is already achieved by including a constant variable.
But if we have such an error term in our model, ie. our understanding of reality, then the residuals, which is the difference between the real data and our estimation, should be only created by these error terms and therefore also appear random.
On the other hand if we have a biased estimate because we are missing variables, we can still be confident about our results if the residuals appear to be random around zero.

So now you have a pretty good understanding of the problems with the model. You should now go ahead and look at your residuals. Calculate those and look at the plot. Does it look normal? Does it have a systematic error? How are the residuals and the independent variables correlated?
You want to make sure that there is no relation between an independent variable and a residual (at the same datapoint, that is). So plot the correlation of this!
Next you want to make sure your residuals look normal, ie literally normal (distributed). If not, you are probably missing some influence which may make your estimate problematic.
At this point it might be that you have to give up because you just can not make the model work.
However in general you start to add and remove variables, keeping in mind how they correlate with each other. You look at p-values and R-Squared to find a model which explains the influence best. But keep in mind that the p-values only make sense if your residuals are correctly distributed.

I wrote this as a general understanding of some problems. You will have additional issues because of your small sample size, though these are more about accuracy of your forecast (depends). I didn't explain the math or how to technically do this because honestly, that's gonna take quite a bit of research and more than I can or should write here. But now you know what dangers lie along the way!

Answer 3

Well, you may run simple OLS regression with your independent variables and then look at p-values and drop out these which are statistically not significant. But if you don't have too many data points, I would suggest playing with the variable specification.