0
$\begingroup$

I have a dataset having 252 observations. There are two variables having a defined and known linear relationship. By computation, I found 4 observations do not follow the linear equation, and I conclude the difference is due to data entry errors.

Although those two variables are not used in my regression model, I am still concerned the reliability of those 4 observations.

My question is should I remove those 4 observations from the dataset, before doing a linear regression?


EDIT:

There are two variables "body fat" and "density". The value of "body fat" is computed by "density" using a linear formula, stated by the creators of this dataset. I found those 4 entries are far off from the given formula, whereas all others are exactly follow the formula.

$\endgroup$
  • $\begingroup$ How did you decide that those 4 entries do not follow the linear equation? They may have occurred by chance due to random fluctuation. If you did a statistical test to show that they were unlikely to occur by chance then you have justification for removing them. $\endgroup$ – Hugh Feb 17 '17 at 22:42
  • $\begingroup$ @Hugh Sorry it's my bad to make the confusion. There are two variables "body fat" and "density". The value of "body fat" is computed by "density" using a linear formula, stated by the creators of this dataset. I found those 4 entries are far off from the formula, whereas all others are exactly follow the formula. $\endgroup$ – Jay Wong Feb 17 '17 at 22:46
  • $\begingroup$ Well if all the other entries exactly follow the formula then there is 0 error in your linear regression so a statistical test for those other 4 entries will find that they cannot occur by chance. In this case you are justified in removing those 4 entries. $\endgroup$ – Hugh Feb 18 '17 at 7:27
  • $\begingroup$ If there is a little error in your linear regression and the entries do not exactly follow the linear relationship then maybe it is obvious from a graph that those 4 entries are wrong. Looking at the graph isn't a formally correct way to make a decision but it can be good enough for many purposes. $\endgroup$ – Hugh Feb 18 '17 at 7:30
1
$\begingroup$

To summarize from your question and comments:

  • You have a data set containing many "features" for each subject.
  • Some of these features were supposedly calculated from others. Specifically, body fat percentage was calculated from the subjects' density or vice versa.
  • However, you cannot reproduce those calculations for a few subjects (i.e., the body fat percentage is not what you would calculate for a given density).
  • The relationship between body fat percentage and density is not relevant to your research question.

If these are all true, then it's certainly fine to consider these data points "suspect." Similarly, you should question data from subjects who are -2 years old, or weigh 3,019 kg: these are obviously errors. If you have access to the original subjects or data collectors, it might be worth trying to verify these values (and spot-check a few more, just in case). If not, I think you're fairly safe excluding them, though you should mention your quality control checks when describing the results.

Obviously, the answer changes if you are interested in the relationship between body fat and density. In this case, you cannot just throw out observations that fail to capture the trend you expected to see. Likewise, you should be a bit more circumspect if the relationship is heuristic (i.e., most people have a percentage body fat that is a linear function of their density, but some don't).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.