2
$\begingroup$

First, let me set the scene.

For each AFL match played, the umpires of that match decide who they deem to be "best and fairest" players on the ground. They allocate 3 votes to the most deserving player, 2 votes to the next deserving and 1 vote to the 3rd most deserving player. Towards the end of the season these votes are tallied and the player with the highest wins the Brownlow Medal.

The problem I am trying to solve is to try and predict the allocation of these votes based on the match statistics which are publicly available. For example, see this game: http://afltables.com/afl/stats/games/2015/031420150402.html

I am taking the statistics of each match (marks, kicks, handballs, goals etc.) for each player and using them to try and predict the BR (Brownlow vote) value assigned to each player for the match.

Unfortunately, I'm fairly new to machine learning and not sure of the best way to structure my data and what algorithms to try etc. I am using this project as a means of learning more about machine learning. I'm currently working with a Python stack, using NumPy, Scikit Learn and also Keras.

Currently, each vector in my input data is the statistics table for each player for a given match, flattened. Since there are usually 44 players in each match, and I am only interested in 21 of the supplied stats, each vector is of length 44*21 = 924. I do perform some column-wise normalization on the input data, before flattening the table to reduce the impact of a low-activity vs high-activity match.

The target data is a vector which contains the brownlow vote assigned to the i-th player (e.g. [0, 0,....,3, 0, 2, 0, 0, 1]), and is of length 44.

I'm processing data from matches spanning years 1998-2015 inclusive, around 3200 matches.

My questions are:

  • Is my current approach for formulating the input data sound? Does it make sense to normalize and flatten the input data in this way? As far as I could tell, most machine learning classifiers require data to be flattened (i.e. they do not take a 2-d matrix input)
  • What would be some good candidate algorithms for me to use to solve this problem?

To me this seems to be a Multi-output classification problem, as the output is a vector with possible values 0, 1, 2 or 3. Thus I have tried using a MultiOutputClassifier with a RandomForestClassifier with terrible results - I get an accuracy of 0.0?

I've also tried using a Keras basic Sequential model, again with terrible results - my accuracy is around 0.15?

Here is the code I'm using to process Pipe-delimited files with stats in them (each file is one match):

import sys
import os
import numpy as np
import fnmatch
from sklearn import preprocessing
from sklearn.model_selection import train_test_split
from keras.layers import Input, Dense
from keras.models import Model

AFL_STAT_HEADER = ["#", "S_ON", "S_OFF", "T_NM", "P_NM", "KI", "MK", "HB", "DI", "GL", "BH",
              "HO", "TK", "RB", "IF", "CL", "CG", "FF", "FA", "BR", "CP", "UP", "CM",
              "MI", "1%", "BO", "GA", "%P"]

AGG_STATS = set(["KI", "MK", "HB", "DI", "GL", "BH", "HO", "TK", "RB", "IF", 
             "CL", "CG", "FF", "FA", "CP", "UP", "CM",
             "MI", "1%", "BO", "GA"])

NON_NUMERIC_STATS = set(["T_NM", "P_NM"])

EXTRACT_STATS = ["KI", "MK", "HB", "GL", "BH",
              "HO", "TK", "RB", "IF", "CL", "CG", "FF", "FA", "CP", "UP", "CM",
              "MI", "1%", "BO", "GA", "%P"]

POS_STATS = ["MK", "DI", "GL", "BH",
              "TK", "RB", "IF", "CL", "FF", "CP", "UP", "CM",
              "MI", "1%", "BO", "GA"]

NEG_STATS = ["CG", "FA"]

def parse_match_stats(psv_path):
    stats = []    
    with open(psv_path, 'r') as f:
        lines = f.readlines()
        for line_num, line in enumerate(lines):
            if (line_num == 0):
                continue

            sp = line.split("|")
            player_stats = {}
            for i, elem in enumerate(sp):
                stat = AFL_STAT_HEADER[i]
                if stat in NON_NUMERIC_STATS:
                    player_stats[stat] = sp[i]
                else:
                    player_stats[stat] = float(sp[i])

            stats.append(player_stats)
    return stats

def preprocess_stats(stats):
    feature_matrix = []
    targets = []
    labels = []
    for p_stat in stats:
        row = map(lambda x: p_stat[x], EXTRACT_STATS)
        feature_matrix.append(row)
        targets.append(p_stat["BR"])
        labels.append(p_stat["T_NM"] + ": " + p_stat["P_NM"])

    f_norm = preprocessing.normalize(feature_matrix, axis=0)
    return (labels, f_norm, targets)    


labels = []    
features = []
targets = []
for root, dirnames, filenames in os.walk(path):
    for filename in fnmatch.filter(filenames, '*.psv'):
        psvpath = os.path.join(root, filename)            
        stats = parse_match_stats(psvpath)
        preprocessed = preprocess_stats(stats)
        labels.append(preprocessed[0])
        features.append(np.array(preprocessed[1]).flatten())
        targets.append(preprocessed[2])
X = np.array(features)
y = np.array(targets)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=0)

forest = RandomForestClassifier(n_estimators=924, random_state=1)
clf = MultiOutputClassifier(forest, n_jobs=-1)
clf.fit(X_train, y_train)
clf.score(X_test, y_test)

model = Sequential([
    Dense(64, input_dim=924),
    Activation('relu'),
    Dense(44, init="normal"),
    Activation('softmax'),
])
model.compile(optimizer='adam',
              loss='mean_squared_error',
              metrics=['accuracy'])
model.fit(X_train, y_train, nb_epoch=50, batch_size=10)
score = model.evaluate(X_test, y_test, batch_size=10)
$\endgroup$
2
$\begingroup$

I find problematic 2nd and 3rd most deserving players, because those are ordinal; not nominal variables. You should use ordinal methods.

For example I would try to achieve a metric that maps statistics of each match. The metric should be capable to map all the players and rank them. The ranking should match with the votes.

I would first play around the data. I would drop 2nd and 3rd off or merge with 1st and use binary classifier for a start and see what happens.

$\endgroup$
  • $\begingroup$ Yes that sounds like a good idea. I had considered just a simple metric in the beginning - summing normalized statistics for each player. I think another option is to train separate binary classifiers to find the players deserving of 3 votes, 2 votes and 1 vote. To train the classifier for 2 votes, simply remove the stats of the player who got 3 votes. Repeat for the next. Will have to try it out and report back :) $\endgroup$ – racc Oct 3 '16 at 10:13
  • $\begingroup$ No, that is not what I meant. Training a binary classifier for each does not make sense. You would be evaluating 2nd bestness and 3rd bestness. If your research would be whether the 2nd and 3rd are evaluated based on different criterion to 1st, it would make sense. You need a metric that tells how good a player performed. That you can rank, and the ranking should match as well as possible to those top three. Binary classifier tells you what makes the top players distinct from the rest. $\endgroup$ – user3644640 Oct 5 '16 at 6:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.