0
$\begingroup$

I have a dataset, equal to 135,997 rows and 9 columns, the head of the table looks like this

fixed_id time_period default_flag score_1 score_2 score_3 score_4 score_5 score_6
1        1           1             3         4       5       2      1         1
1        1           1             3         4       5       2      1         1
1        1           1                                                   
1        1           1             

My question is, when is it ok to remove rows 3 and 4 in this dataset?

Plan: I plan to estimate a series of multivariate logistic regression models to identify the best combination of factors containing any mixture of score_1 to score_6.

My thought process: If I have a dependent variable and no explanatory variables - I see no logical reason to keep these observations in. Is there any statistical reasoning to keep them in e.g. a present dependent variable but no explanatory variables still adds to degrees of freedom or other?

Other comments: My reasoning for wanting to remove these rows is that I will have a substantially smaller dataset and this will greatly improve the time to estimate a series of multivariate logistic regressions.

Edit: to be clear, I only want to remove rows where all scores 1-6 are missing.

$\endgroup$
5
  • 1
    $\begingroup$ Why are they missing though? $\endgroup$
    – AdamO
    Jan 23, 2019 at 20:49
  • $\begingroup$ All scores are missing when there was no information collected that month $\endgroup$
    – user235111
    Jan 23, 2019 at 21:00
  • 1
    $\begingroup$ What was the reason for not collecting them? The concern is whether a failure to collect data could yield information or be related to what you're trying to model, in which case just deleting the affected rows could bias all results. $\endgroup$
    – whuber
    Jan 23, 2019 at 21:09
  • 1
    $\begingroup$ Often times data is not missing at random. If there is any chance that the rows with missing data are somehow different than the rows with data, you are going to run into problems when if you decide to use your model on another dataset. Is it possible to impute your data? $\endgroup$ Jan 23, 2019 at 21:12
  • $\begingroup$ Data collection only happens once a year and the observations are monthly - Unfortunately, the higher authority won't allow me to aggregate onto yearly data before this stage $\endgroup$
    – user235111
    Jan 23, 2019 at 21:37

1 Answer 1

2
$\begingroup$

You can only safely remove these observations, when these are a random sample of your dataset, which is hardly ever the case.

Say, the missing values happen for observations where score_2 is very high and score_1 is low. Then, a regression of score_1 on score_2 will overestimate the regression parameter that would hold in the full population.

If missing values are not random any conclusion that you draw only holds for a population that is defined by the population the dataset was sampled from and the selection mechanism leading to missing values. This might be useful or not, depending on the use case.

Say, the missing values are due to patients not showing up to a follow-up exam, results would still hold for the population of patients returning for the follow-up, but not for the entire population of treated patients. This might still be useful for predicting the future health of these patients.

$\endgroup$
1
  • 1
    $\begingroup$ I'm not sure it's accurate to say that observations are hardly ever missing at random. This is highly dependent on the type of data collected. For example, if this is laboratory specimen data, often times, nearly all data is missing completely at random. This is often the case if test tubes are accidentally dropped or and a specimen testing machine randomly malfunctioned. $\endgroup$ Jan 23, 2019 at 21:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.