0
$\begingroup$

I'm not a statistician, so please excuse a possibly wrong use of terminology here.

My dataset has about 400 - 800 samples and about 800 features. The samples are ordered by time, although it is not a time series. There are three class labels (exclusive), which are not highly imbalanced, but slightly imbalanced.

I have a custom metric to check how well a model performs, which is not usable for training, but can be used to evaluate it afterwards (very computationally expensive).

I'm using a training set, a validation set, and a test set. The test set is fixed (last x recent samples), but depending on how i choose the training and validation sets, e.g., for a MLPClassifier, the outcome on my custom metric varies greatly.

To deal with this, i build a few hundred models, each with a differently shuffled training/validation set. Then i compare them on my custom metric with the fixed test set. Among the models, i can then select the best performing one.

The disadvantage is that i have to train a few hundred models every time a new number of samples are available and then select the best one for the next prediction period.

Is the overall approach sound? Is there a better approach?

I read things like stochastic weight averaging might help, but as far as i understand this only helps to reduce the randomness introduced by the different starting weights of a neural network, not with variance introduced by having different training/validation sets.

$\endgroup$
0
$\begingroup$

Are you building different models or are you just shuffling the train/validation sets while retaining the parameters of the model? I think cross-validation is what you are searching for in general if you want to build a robust model.

$\endgroup$
5
  • $\begingroup$ It's the same model, just different training / validation sets. I am using CV, but i cannot use it with my custom metric really because it takes minutes to calculate. $\endgroup$
    – Traveller
    Sep 10 '19 at 14:35
  • $\begingroup$ Can you maybe show your error metric? $\endgroup$
    – Chowkah
    Sep 10 '19 at 14:49
  • $\begingroup$ not really, sorry. it's another algorithm that selects among options based on the predictions of the model. maybe i could simplify it to a degree and then use it directly as the training metric. $\endgroup$
    – Traveller
    Sep 10 '19 at 14:51
  • $\begingroup$ I think it might be helpful to use a simplified version of your error metric for cross-validation, especially if it is very different from common metrics. Otherwise your algorithm might optimize for a different target which could result in the test error discrepancies. $\endgroup$
    – Chowkah
    Sep 11 '19 at 6:24
  • $\begingroup$ Thanks! Actually i could solve it by building my own scorer function and hide relevant data for evaluation in the features. it's a bit of a workaround but seems to do the job. $\endgroup$
    – Traveller
    Sep 11 '19 at 16:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.