0
$\begingroup$

I have a dataset with 12k samples where each sample is a feature vector of length 4096.
The task it to classify the samples into 12 categories.
I fitted a network with 2 fully connected layers on the entire dataset and the model was able to separate it almost perfectly (it has 60 errors on it) which means the data is almost perfectly separable.

After that I split the data into 10 folds. For each fold I evaluated the model on it after fitting it to the 9 remaining folds.
The model is always able to fit perfectly to the 9 folds, while having very poor performance on the evaluation fold.

This means that the model cannot explain the evaluation fold by the rest of the folds, even though I've shown that all of the data can be explained by a single model of the same architecture,

What could be causing this issue? And how can I solve it?

$\endgroup$
0
$\begingroup$

These types of questions are difficult to answer but overfitting seems likely.

  1. The first error you measure seems to be on the same data you fit the model to, these errors are nearly always too optimistic to say anything about out-of-sample error.
  2. You use a highly flexible model with thousands and thousands of parameters. With such a model you can fit almost any training set perfectly.
  3. You have thousands of predictors, some set of which is very likely to perfectly separate training data by pure chance.

Regularization of some sort might help. There are dozens of ways the neural network people regularize models.

There might also not be anything in to learn in your data. It is still possible, even easy, to get excellent training error.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I'm not sure why you say that I have thousands of predictors. could you elaborate please? and In addition, I further simplified my model to be a single layer (which is a linear model basically) and I still recive the same issue. Does that help provide more information on the issue? $\endgroup$ – Gal Avineri Sep 11 at 12:47
  • $\begingroup$ You say that each each data point is a vector of length 4096, which I took to be your predictors/input features. A linear model may also need regularization: if you pass your 12k data points into a 4k-parameter linear regression with no regularization, you have three observations per parameter you're trying to fit. This is very little. $\endgroup$ – einar Sep 11 at 13:50
  • $\begingroup$ yes threre are 4096 input features indeed. following the observation of low ration between observations and features i'm looking into regularizations and dimensionality reduction methods. $\endgroup$ – Gal Avineri Sep 12 at 9:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.