2
$\begingroup$

I have an idea for a training strategy (for an ML model), can you please tell me whether it has a name, and whether it makes sense.

I need a model for binary classification with a massive class imbalance problem (~1:100000). There are a lot of features. In the process of sanity testing I started feeding the label as feature into the model - unsurprisingly the model performed well (that was the sanity test).

Now I want to slowly withdraw the 'bad feature' by making it less and less correlated with the label, whilst looking at how model performs and what hyper-parameters give good results.

So my question is this. Can one use it as a training strategy? Start with contaminated features. Train model, then spoil the contamination by adding noise (making features less contaminated). Train again, and keep going until the 'bad feature' is no longer bad (i.e. completely uncorrelated with label).

Thanks

EDIT

It would seem I expressed myself poorly. What I meant is that the model is trained successively but without re-initializing the weights. You start with one feature (feature_A) essentially being the label - model preforms well. Next time, with the same weights, model sees data where feature_A is no longer as good of a predictor, so the model will adapt to pay more attention to other features. Do it again and again, until feature_A is no longer predicts the label at all, but the model is now (hopefully) adapted to make good predictions nevertheless.

I guess my simplistic thinking is that if there exists a uniquely correct composition of features that allows to make good predictions, I would be able to push the model to pay attention to the

Improve this question
$\endgroup$
4
  • $\begingroup$ How would that be ever useful? You cannot use labels on prediction stage.You want to predict something that you don't know, otherwise why to predict it? If that is the case, then the "feature" would not be useful for you. $\endgroup$
    – Tim
    Commented Aug 31, 2020 at 18:44
  • $\begingroup$ @Tim I thought it could be useful as a sort of initialization strategy. What I meant was that in successive trainings you do not re-initialize the weights of the model, you start with previous ones. $\endgroup$
    – Cryo
    Commented Aug 31, 2020 at 20:17
  • 1
    $\begingroup$ but if model had access to the labels at training time, it can simply copy them while ignoring other variables. It does not need to learn anything. So it gives you no guarantees that what it learned is useful. $\endgroup$
    – Tim
    Commented Sep 1, 2020 at 5:18
  • $\begingroup$ @Tim. Makes sense. Thanks $\endgroup$
    – Cryo
    Commented Sep 2, 2020 at 4:58

1 Answer 1

Reset to default
3
$\begingroup$

I assume that by 'label' you mean the correct classification/value of the dependent variable. It appears you are asking if you add the correct classification as an independent variable, will this cause the model to make an association with your rare event and other independent variables, that will persist after you remove the label. Definitely not, using any type of machine learning algorithm I have heard of, or can think of.

A common way of dealing with rare outcomes is with over-sampling and cost adjustments. See https://blogs.sas.com/content/subconsciousmusings/2017/07/19/machine-learning-best-practices-detecting-rare-events/

Improve this answer
$\endgroup$
4
  • $\begingroup$ I thought it could be useful as a sort of initialization strategy. What I meant was that in successive trainings you do not re-initialize the weights of the model, you start with previous ones. $\endgroup$
    – Cryo
    Commented Aug 31, 2020 at 20:18
  • 1
    $\begingroup$ That isn't going to work. The model will be based on the significant relationship, label, and when it's not there, there is no relationship. $\endgroup$
    – Davis70
    Commented Aug 31, 2020 at 20:46
  • $\begingroup$ I updated my answer with "A common way of dealing with rare outcomes is with over-sampling and cost adjustments. See blogs.sas.com/content/subconsciousmusings/2017/07/19/…" $\endgroup$
    – Davis70
    Commented Aug 31, 2020 at 20:50
  • $\begingroup$ Thanks_________ $\endgroup$
    – Cryo
    Commented Aug 31, 2020 at 21:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.