Background: coding in Python, utilizing
scikitlearn (the former supports Poisson, the latter can easily split sample for training and testing). Very little experience in statistics.
Let's assume I have a hypothesis: for every soccer game, several figures (predictors) can be used to estimate the total goals scored (response) in that game. by total goals, I mean the number of goals scored by both teams.
The predictor values are calculated up to the point of the game. So for example if
x1is the home team's average goals, only past game results would affect it. (obvious, I know)
Some of the predictor values are not whole numbers. As I understand, this shouldn't affect the model.
So I've split my sample to
Here's the output I get after I fit the model:
GEE Regression Results =================================================================================== Dep. Variable: total_goals No. Observations: 2458 Model: GEE No. clusters: 2458 Method: Generalized Min. cluster size: 1 Estimating Equations Max. cluster size: 1 Family: Poisson Mean cluster size: 1.0 Dependence structure: Independence Num. iterations: 7 Date: Mon, 16 Jan 2017 Scale: 0.982 Covariance type: robust Time: 14:39:12 ==================================================================================== coef std err z P>|z| [95.0% Conf. Int.] ------------------------------------------------------------------------------------ Intercept 0.2474 0.097 2.552 0.011 0.057 0.437 var1 0.1471 0.027 5.453 0.000 0.094 0.200 var2 0.1115 0.027 4.068 0.000 0.058 0.165 var3 0.2351 0.114 2.069 0.039 0.012 0.458 ============================================================================== Skew: 0.5144 Kurtosis: 0.0916 Centered skew: 0.0000 Centered kurtosis: -3.0000 ==============================================================================
I'm not sure how to interpret all of the column values, but I know that (in this case) if P>|z| is smaller than
0.05 then a variable is considered statistically significant.
What I want to inquire about, is whether I have a better way (compared to the following) to test whether my model is good for predicting whether more than two goals are scored in a match. Here's what I'm currently doing:
- for each game in the
- if the prediction for
total_goalsis above certain threshold (
cutoff), I count it as a counted prediction.
- for each counted prediction, compare against the actual result of the game. If the actual result is above 2 goals, mark it as
correct, if not:
- Output the
hit rate %.
The issue that I don't really know how to address is this:
scikit-learn (rightfully) randomizes games in the sample when it splits it to
train subsets. Thus, I obtain different results (different hit rate %) each time. Some are satisfactory, some aren't.
How can I estimate confidence in my model, then? I thought about running the whole procedure 1,000 times, and obtain the average hit rate %, but that doesn't sound like the right way to do it.
Any help or guidance is much appreciated!
clf = svm.SVC(kernel='linear', C=1) scores = cross_val_score(clf, train, test, cv=5)
train is my training set (0.7 of sample) and
test is my testing set (0.3 of sample), it throws an error and says that the data is of different lengths. I am assuming this because in the scikit-learn example that I'm trying to copy to my code:
scores = cross_val_score(clf, iris.data, iris.target, cv=5)
iris.target are of the same length: the former includes the predictors while the latter contains the correct classification. But I'm not making a classifier. How can I adjust this to fit my code, or should I?