1
$\begingroup$

Is it bad practice to run a Machine learning algorithm on an experimental dataset, check the MAE, and remove the instances that have a value of MAE above a certain limit? If we run the algorithm without those instances the accuracy increases significantly of course.

I know that something similar can be used when we want the ML to learn from its mistakes. Would it be better to make my own loss function? At the moment I am using a regression NN from keras and tensorflow.

I have reached the point where no matter the model, my accuracy doesn't increase anymore. I know that there might be outliers in my dataset but detecting them is not straightforward at all.

I am relatively new in the Machine Learning field so I will be very grateful for any advice, suggestions, or corrections.

$\endgroup$
5
  • $\begingroup$ What will you do when you put your model in production and it makes major mistakes? In other words, when I say, "Siri, play 'Welcome to the Jungle' by Guns N Roses," and Siri tells me, "Sorry, Dave, you are not near any psychological counselors," do you ignore that error? $\endgroup$
    – Dave
    Oct 22, 2021 at 14:53
  • $\begingroup$ So you suggest that removing instances like that will improve the accuracy for the specific dataset, but when applied in different data it could very well fail, right? If this method of removing instances though was included in the machine learning algorithm within the loss function, would that work when in production? $\endgroup$
    – RandML000
    Oct 22, 2021 at 14:58
  • 2
    $\begingroup$ I think what Dave means is that if you're systematically removing the hardest examples from your data set (the ones that your model gets wrong, i.e. the largest values), you're not getting a good estimate of how the model will perform in reality. You're correct that removing the largest numbers from a list will have a smaller mean, but that smaller mean doesn't have any bearing on how your model does. $\endgroup$
    – Sycorax
    Oct 22, 2021 at 15:01
  • $\begingroup$ Yes, you are both right! Thank you for your comments! I suppose that I need to either find a ''real'' way to detect outliers or improve my NN algorithm. $\endgroup$
    – RandML000
    Oct 22, 2021 at 15:04
  • 2
    $\begingroup$ On a positive note: OP came here to ask instead of just going ahead and removing the problematic observations $\endgroup$
    – Aksakal
    Oct 22, 2021 at 16:11

1 Answer 1

1
$\begingroup$

If you're systematically removing the hardest examples from your data set (the ones that your model gets wrong, i.e. the largest values), you're not getting a good estimate of how the model will perform in reality. You're correct that removing the largest numbers from a list will have a smaller mean, but that smaller mean doesn't have any bearing on how your model does.

Some problems are simply challenging, perhaps because there's a large degree of randomness in the process, or because the features don't strongly bear on what you want to predict, or another reason.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.