0
$\begingroup$

I'm performing a binary classification with both logistic regression and SVM, where I've 80%-20% train-test split of the 10000 samples, each with 11 features. In my problem, the features are the data related to a car application, e.g. the time a ride request was made, the time the corresponding booking request(s) was/were made, the latitudes and lognitudes of the rider and the driver etc. The response variables are driver's response 1 (ride request accepted) or 0(ride request not accepted). After classification, I'm getting around 68% accuracies for all of training, cross validation and classifcation, and out of the response {0,1}, about 64% are 1 and 36% is 0. When I perform a grid search to find the best parameters for both linear and kernel SVM, I still get around 69% accuracies.I also tried logistic regression and Naïve Bayes, but the accuracy range in all cases were between 65%-69%.

From a high level, my question is: what should I do next to find a better model? For example, 1) should I worry about class imbalance, since the classes are 64%-36% distributed? 2) a. since the number of features (11) is much less than that of samples (10000), do I still need to do PCA to improve accuracy? b. Should I do PCA in any case to get better understanding of the 2D projected data? 3) I'm not sure if there's lot of noise, as the data features are the car data as described; but if there's noise, how can I identify them?

Any help will be highly appreciated?

$\endgroup$
14
  • 1
    $\begingroup$ 1) ~65%/35% should make you worry at least a little bit. For me that would be enough to at least think about an adapted cost function or over/undersampling. In any case, its just computer time: try it with and without correction and then compare... 2) I would definitely not go for PCA in general: it tends to "destroy" many natural features and sometimes makes life harder for the model 3) I would certainly think about crafting more features from the existing ones: does the customer 'usually' accept? Did he/she accept the last time/in the last week/what is the overall ratio, ... $\endgroup$ – Fabian Werner Nov 28 '18 at 22:49
  • 1
    $\begingroup$ Also you could try even more models: check out gradient boosting and maybe play around with some simple NN architectures (<= 10 layers, <= 500 hidden nodes or so) just to get a grip of whether or not your problem seems to love a particular algo or not $\endgroup$ – Fabian Werner Nov 28 '18 at 22:51
  • 1
    $\begingroup$ Before dumping everything into the model, try and work out how you would use those inputs to accept or reject? Eg. How far is the customer, traffic congestion? Etc $\endgroup$ – seanv507 Nov 29 '18 at 0:24
  • 1
    $\begingroup$ @Mathmath: Oopsie, yes I meant 'driver' instead of 'customer' :-) yes I meant vanilla NN. Complex architectures will be difficult to tweak and I do not (yet) see a clear motivation for using fancy-shmancy-stuff like RNNs (there is no time-series-ish component of the problem so far) or CNNs (because there is no obvious locality) $\endgroup$ – Fabian Werner Nov 29 '18 at 7:49
  • 1
    $\begingroup$ @Mathmath and definitely listen to seanv507: Do NOT (almost never) use lat/lon or timestamps directly as input features for any model except for when you really know what you are doing. Mostly we are interested in differences as sean said: how far is X from Y? How much time passed between these two timestamps, etc... that is something a model can deal with $\endgroup$ – Fabian Werner Nov 29 '18 at 7:51
0
$\begingroup$

Feature selection is one heck of a job and it is a huge domain. Well, in case of binary classification using SVM (or any other method as a matter of fact), the approach I choose is the use of Probability Density Functions (or PDFs). Let me put out a picture:


Why do I use PDFs?

Ok. So, the goal of SVM in your case is to classify your data into two classes. How does SVM work? It aims to find a plane to divide your data into two classes (a line in 2D, a plane in 3D and a hyper-plane in n-D).

When you trace a normalised PDF of a known data set for the two classes, the two curves tend to either be completely separated, maybe overlap a little or completely overlap. Here are three examples of a data set that I use. These are some examples from ECG signals. I have traced the mean of the data (Plot.1), the mean absolute deviation (Plot.2) and a slope Q-R (Plot.3) from ECG signals.

Plot.1

Plot.2

Plot.3

As you can see in Plot.1, the red and blue curves on the right side almost completely overlap. In Plot.2, they are almost separated while in Plot.3, they are totally separated.


How is this useful?

The dashed line in the three plots above, that is very similar to the SVM margin for classification for data both in feature space and the hyper-space (if you're using kernel functions). So if the separating hyper-plane is not able to separate the data into two classes, no matter what kernel or any feature set you use, the learning step tends to fail resulting in low accuracies (That will in turn fail your testing phase). If you have a known data set, you could trace such PDFs and see how the features appear. And if you're able to find say 5-6 features out of 11 that you have get good separation (as in Plots 2 and 3 above), I guarantee you that the learning step will be a 100% accurate and your testing step too will give you a very high accuracy. This is known as physical analysis of signal features (well, that's what they call it here at least :P).


Advantages of using this approach

As most work suggests, learning data should be at least 66% of the total data set. By using this approach, I was able to obtain very similar results even with learning data set size as low as 45% of the total data set.

PS: The PDF plots were traced using the moving average curve smoothing technique; you could use other techniques too.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.