First, let me set the scene.
For each AFL match played, the umpires of that match decide who they deem to be "best and fairest" players on the ground. They allocate 3 votes to the most deserving player, 2 votes to the next deserving and 1 vote to the 3rd most deserving player. Towards the end of the season these votes are tallied and the player with the highest wins the Brownlow Medal.
The problem I am trying to solve is to try and predict the allocation of these votes based on the match statistics which are publicly available. For example, see this game: http://afltables.com/afl/stats/games/2015/031420150402.html
I am taking the statistics of each match (marks, kicks, handballs, goals etc.) for each player and using them to try and predict the BR (Brownlow vote) value assigned to each player for the match.
Unfortunately, I'm fairly new to machine learning and not sure of the best way to structure my data and what algorithms to try etc. I am using this project as a means of learning more about machine learning. I'm currently working with a Python stack, using NumPy, Scikit Learn and also Keras.
Currently, each vector in my input data is the statistics table for each player for a given match, flattened. Since there are usually 44 players in each match, and I am only interested in 21 of the supplied stats, each vector is of length 44*21 = 924. I do perform some column-wise normalization on the input data, before flattening the table to reduce the impact of a low-activity vs high-activity match.
The target data is a vector which contains the brownlow vote assigned to the i-th player (e.g. [0, 0,....,3, 0, 2, 0, 0, 1]), and is of length 44.
I'm processing data from matches spanning years 1998-2015 inclusive, around 3200 matches.
My questions are:
- Is my current approach for formulating the input data sound? Does it make sense to normalize and flatten the input data in this way? As far as I could tell, most machine learning classifiers require data to be flattened (i.e. they do not take a 2-d matrix input)
- What would be some good candidate algorithms for me to use to solve this problem?
To me this seems to be a Multi-output classification problem, as the output is a vector with possible values 0, 1, 2 or 3. Thus I have tried using a MultiOutputClassifier with a RandomForestClassifier with terrible results - I get an accuracy of 0.0?
I've also tried using a Keras basic Sequential model, again with terrible results - my accuracy is around 0.15?
Here is the code I'm using to process Pipe-delimited files with stats in them (each file is one match):
import sys
import os
import numpy as np
import fnmatch
from sklearn import preprocessing
from sklearn.model_selection import train_test_split
from keras.layers import Input, Dense
from keras.models import Model
AFL_STAT_HEADER = ["#", "S_ON", "S_OFF", "T_NM", "P_NM", "KI", "MK", "HB", "DI", "GL", "BH",
"HO", "TK", "RB", "IF", "CL", "CG", "FF", "FA", "BR", "CP", "UP", "CM",
"MI", "1%", "BO", "GA", "%P"]
AGG_STATS = set(["KI", "MK", "HB", "DI", "GL", "BH", "HO", "TK", "RB", "IF",
"CL", "CG", "FF", "FA", "CP", "UP", "CM",
"MI", "1%", "BO", "GA"])
NON_NUMERIC_STATS = set(["T_NM", "P_NM"])
EXTRACT_STATS = ["KI", "MK", "HB", "GL", "BH",
"HO", "TK", "RB", "IF", "CL", "CG", "FF", "FA", "CP", "UP", "CM",
"MI", "1%", "BO", "GA", "%P"]
POS_STATS = ["MK", "DI", "GL", "BH",
"TK", "RB", "IF", "CL", "FF", "CP", "UP", "CM",
"MI", "1%", "BO", "GA"]
NEG_STATS = ["CG", "FA"]
def parse_match_stats(psv_path):
stats = []
with open(psv_path, 'r') as f:
lines = f.readlines()
for line_num, line in enumerate(lines):
if (line_num == 0):
continue
sp = line.split("|")
player_stats = {}
for i, elem in enumerate(sp):
stat = AFL_STAT_HEADER[i]
if stat in NON_NUMERIC_STATS:
player_stats[stat] = sp[i]
else:
player_stats[stat] = float(sp[i])
stats.append(player_stats)
return stats
def preprocess_stats(stats):
feature_matrix = []
targets = []
labels = []
for p_stat in stats:
row = map(lambda x: p_stat[x], EXTRACT_STATS)
feature_matrix.append(row)
targets.append(p_stat["BR"])
labels.append(p_stat["T_NM"] + ": " + p_stat["P_NM"])
f_norm = preprocessing.normalize(feature_matrix, axis=0)
return (labels, f_norm, targets)
labels = []
features = []
targets = []
for root, dirnames, filenames in os.walk(path):
for filename in fnmatch.filter(filenames, '*.psv'):
psvpath = os.path.join(root, filename)
stats = parse_match_stats(psvpath)
preprocessed = preprocess_stats(stats)
labels.append(preprocessed[0])
features.append(np.array(preprocessed[1]).flatten())
targets.append(preprocessed[2])
X = np.array(features)
y = np.array(targets)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=0)
forest = RandomForestClassifier(n_estimators=924, random_state=1)
clf = MultiOutputClassifier(forest, n_jobs=-1)
clf.fit(X_train, y_train)
clf.score(X_test, y_test)
model = Sequential([
Dense(64, input_dim=924),
Activation('relu'),
Dense(44, init="normal"),
Activation('softmax'),
])
model.compile(optimizer='adam',
loss='mean_squared_error',
metrics=['accuracy'])
model.fit(X_train, y_train, nb_epoch=50, batch_size=10)
score = model.evaluate(X_test, y_test, batch_size=10)
