0
$\begingroup$

Some context: I'm trying to train a model on some data. The input per data point is 12 time-signals, and there are 104 classes.

Using only the mean and variance (mean, var) of each of these signals, and feeding it to LogisticRegression from sklearn, my model predicted 80% correct on the training data itself.

I tried polynomial expansion, so that I feed (mean, mean**2, mean**3, var, var**2, var**3) to LogisticRegression. Now the correct rate on the training data is only 25%! I also set C = 10**3, 10**6 and 10**9, but only 10**3 was slightly better with 27%.

From what I have learned, I really thought that using a more complex model to train the data could improve the predictions on the training data itself, and eventually overfit on the training data (up to 100% correctness) while getting bad results on the test data. But somehow I must have gotten the wrong idea... I anticipated that adding features could not hurt other than causing overfitting. Any idea why adding more features makes it perform so much worse on the training data?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Overfitting assumes the model has sufficient complexity to perfectly fit the training data, and that one can discover those model parameters via the learning algorithm used. We know the model complexity is sufficient, since you got a good accuracy with a subset of your blown-up problem dimensions. Therefore, the most likely explanation is that the algorithm is failing due to the data being too high-dimensional (and thereby too noisy) for it to work.

To be more specific, it would be good to get the size of the data, the exact parameters you gave to LogisticRegression, and any output from the command if you can turn 'verbose' on somehow.

Improve this answer
$\endgroup$
6
  • $\begingroup$ By too high-dimensional, do you mean too many features? The size of the training data set is 8895 points (each with 12 time signals), linear_model.LogisticRegression(max_iter=2000, verbose=1). This is part of the output. The total output is 135k lines, but it seems to mostly repeat itself: cg reaches trust region boundary iter 2000 act 4.242e-02 pre 4.105e-02 delta 1.395e-07 f 3.829e+02 |g| 2.175e+06 CG 3 So do you mean that the model could be not powerful enough when the data becomes more complex (more features)? $\endgroup$
    – Ploppz
    Commented Nov 1, 2017 at 15:09
  • $\begingroup$ Assuming in the previous comment that 'model' refers to e.g. LogisticRegression. But in my main post, by 'model' I also meant the feature transformation/extraction. $\endgroup$
    – Ploppz
    Commented Nov 1, 2017 at 15:18
  • $\begingroup$ And there's your problem -- you're hitting your max iteration number before you find your minimum. $\endgroup$
    – bibliolytic
    Commented Nov 1, 2017 at 16:04
  • $\begingroup$ Ah, didn't consider it a big problem since the iteration count is already very high I think. What if I made an ensemble of binary classifiers (one-vs-rest), one for each pair of (class, feature). Then to predict the class of a data point, each classifier votes on class c based on the probability? To illustrate: bpaste.net/show/122034c20a75 ... I already did this and got only 26% correct on features (mean,var) although it seems that individual binary classifiers are pretty good. Asking because I'm wondering if the idea is bad or if my implementation is incorrect. $\endgroup$
    – Ploppz
    Commented Nov 1, 2017 at 16:50
  • $\begingroup$ one for each feature and each pair of classes? I thought the goal was to figure out why things went weird; if not, why not go with your initial approach that got you 80% (which, for 100+ classes, seems pretty darn good to me)? An easier idea that is similar is to just look here: scikit-learn.org/stable/modules/generated/… $\endgroup$
    – bibliolytic
    Commented Nov 2, 2017 at 17:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.