0
$\begingroup$

I have a data set with 89.5k observations, 12 attributes (related to a sport) and 1 class variable. What I want to do is classify each observation into one of the two available classes, in this particular case: H or A.

Since I prefer to put all columns in the same range of values, I've decided to scale the data from 0 to 1. The issue I'm facing is that every machine learning algorithm I've tried (SVM, ANN, RF, Naive Bayes) only get 55% of the predictions correct. What can I do to make it more accurate?

Summary:

       V1               V2               V3                V4                V5               V6        
 Min.   :0.0000   Min.   :0.0000   Min.   :0.00000   Min.   :0.00000   Min.   :0.0000   Min.   :0.0000  
 1st Qu.:0.1184   1st Qu.:0.1170   1st Qu.:0.02724   1st Qu.:0.02864   1st Qu.:0.1618   1st Qu.:0.1626  
 Median :0.2237   Median :0.2197   Median :0.09576   Median :0.09853   Median :0.2972   Median :0.2992  
 Mean   :0.2416   Mean   :0.2373   Mean   :0.13585   Mean   :0.13897   Mean   :0.3113   Mean   :0.3140  
 3rd Qu.:0.3403   3rd Qu.:0.3343   3rd Qu.:0.20500   3rd Qu.:0.20983   3rd Qu.:0.4420   3rd Qu.:0.4455  
 Max.   :1.0000   Max.   :1.0000   Max.   :1.00000   Max.   :1.00000   Max.   :1.0000   Max.   :1.0000  
       V7               V8               V9               V10               V11              V12        
 Min.   :0.0000   Min.   :0.0000   Min.   :0.00000   Min.   :0.00000   Min.   :0.0000   Min.   :0.0000  
 1st Qu.:0.2476   1st Qu.:0.2520   1st Qu.:0.04586   1st Qu.:0.04523   1st Qu.:0.2878   1st Qu.:0.2863  
 Median :0.3361   Median :0.3414   Median :0.18077   Median :0.17998   Median :0.3960   Median :0.3961  
 Mean   :0.3439   Mean   :0.3497   Mean   :0.22190   Mean   :0.22105   Mean   :0.4005   Mean   :0.4010  
 3rd Qu.:0.4292   3rd Qu.:0.4365   3rd Qu.:0.35574   3rd Qu.:0.35460   3rd Qu.:0.5084   3rd Qu.:0.5098  
 Max.   :1.0000   Max.   :1.0000   Max.   :1.00000   Max.   :1.00000   Max.   :1.0000   Max.   :1.0000  

Variance:

V1   0.02416548
V2   0.0233082933 
V3   0.0185514924  
V4   0.0191324587  
V5   0.034927239  
V6   0.0356891459 
V7   0.0187147471
V8   0.0192708124
V9   0.03920032  
V10  0.0389565089
V11  0.0247022839 
V12  0.0252293020
$\endgroup$
2
  • 4
    $\begingroup$ Who's to say that any more accuracy is possible? $\endgroup$ – gung - Reinstate Monica May 10 '16 at 15:43
  • $\begingroup$ Thanks @gung, please sustain your comments. Btw, Thanks a bunch for editing my question. $\endgroup$ – Andres Alvarado May 10 '16 at 15:45
1
$\begingroup$

Do you obtain similar training and test error in your evaluation? If yes, you are probably really stuck with a high bias (your data might just not reflecting the relation you want to model sufficiently in its current form, as gung indicated). In this case you could try some more feature engineering/feature derivation. In case you are not utilizing train/test/cross validation you should start by adding those.

For feature derivation: you could try to obtain more features for your problem, or derive additional features from the 12 you have so far. This could be done using simple transformations (e.g. Box-Cox), but using domain specific knowledge about the problem and features to transform them into something "more useful" in the respective domain will likely be more helpful.

BTW: you could also normalize to SD=1 instead of using a hard range [0,1]. This might help in case of outliers, but you seem to have less of them anyway, hence a resulting similar SD over all features.

$\endgroup$
3
  • $\begingroup$ Thanks for your reply @geekoverdose. I get same error % in test and training evaluations. I have a'lot of NA data which I have transformed into 0s, do you think this can be causing some issue? Regarding Features, Im sure I can get more. One more thing, how can I reduce the bias of my dataset? $\endgroup$ – Andres Alvarado May 11 '16 at 15:26
  • $\begingroup$ Yes, NAs affecting many samples might well be a problem. The transformation to 0 can be problematic too, e.g. if the value 0 has a certain meaning (like if it is a weight, it would indicate being very light, which will cause trouble with certain classifiers). Bias: you can record more features or you can add more using transformations as mentioned above. Look at e.g. the Box-Cox family (onlinestatbook.com/2/transformations/box-cox.html), but many others might be useful as well. As said, using your domain specific knowledge about the problem to derive new features will likely be the key. $\endgroup$ – geekoverdose May 11 '16 at 15:59
  • $\begingroup$ Tanks a bunch @geekoverdose. Let me try this and Ill let you know my findings. $\endgroup$ – Andres Alvarado May 11 '16 at 16:21
1
$\begingroup$

I am making a couple assumptions here:

  • Since you are using data related to sports and you have a class that is H or A I am assuming this is Home or Away.
  • You are getting 55% accuracy on a classification to H or A. If my first assumption is correct then I assume you are trying to predict the winner of a sports game and not which team is the home team or away team.

Stemming from those assumptions I can tell you that you likely don't have enough features to predict the winner with much more success than a coin flip or a slightly better measure in sport, picking the home team. In sports like hockey and baseball the home team wins roughly 53% of the time (not sure about other major sports as I have not looked into them but I would assume Football and Basketball are a decent bit higher). So getting 55% accuracy is only slightly better than home team picking.

Couple notes about win predicting:

  • Its a tough problem and requires a lot of data and possibly some domain knowledge to help in the choosing/acquiring of the data.
  • Most of the work likely will need to be of your own thought because anyone who can predict wins with a better accuracy than the relative odds will likely have their algorithm hiden away.
  • As alluded to by Gung, maybe you can't get anymore accurate with what you have. There are so many little things that happen in a sports game that win predicting is not going to have a really high accuracy regardless of the method used.
  • Following up on the previous point, if you have exhausted all possible classifiers that you are comfortable using and are not satisfied with the results think about getting adding some features.

If this is not about win predicting then I guess ignore this

$\endgroup$
0
$\begingroup$

Have you thought about using Logistic Regression? It is very useful when dealing with binary class variables. It has also been known to work well on large datasets and can be easily implemented in R.

Aside from that, how are you handling your data? Do you split it into training and testing datasets? Have you performed cross-validation? If you are able to determine if your models perform well on training data, but not so well on testing data (or the other way around), then you may have a case of overfitting/underfitting in your models.

$\endgroup$
1
  • $\begingroup$ Thanks for your reply @Eoin. I havent tried with Logistic Regression. Model performs the same for test data and train data. I split my data in 75% observations for training and 25% for testing. $\endgroup$ – Andres Alvarado May 11 '16 at 15:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.