Panel data analysis with a lot of missing values and a lot of variables

Question

I want to perform an analysis on panel data I got from an online database. I encounter some problems however. One of them is that the database has a lot of features (110). And the other is that it also contains a lot of missing values. For some of the features the dataset even has more missing values than present values. So my first question is, is it good practice to delete the variables with lots of missing values? And if so, what percentage threshold of missing values is acceptable to still achieve reliable results? Extreme examples: I can imagine that deleting a feature that only has one value for 10,000 observations seems reasonable. After all what does the feature really add to the analysis? On the other hand, I can imagine that deleting a variable with only one missing value for 10,000 observations is very bad practice, since this feature could have still added a lot of information to the dataset (even if you delete the row instead of filling the NA). But where's the turning point? Is there a general rule of thumb?

My next question is, how do we fill in the NAs if we have no information about why they are missing? Is nearest neighbor a reasonable method in this case (and also given the data is panel)? I've read a lot about filling in NAs with the mean, but it seems to me that this ignores the fact that the data is panel, as there might be different trend for each individual over time.

The final question I have is: What are the appropriate order of steps including feature selection? Is it better to first drop the variables with lots of NAs, then feature selection, then filling in NAs in order to save computation, or the other way around because it (perhaps) keeps/adds the most information from the data?

Answer 1

I think first you would have to diagnose the missing data mechanism (i.e., is it missing completely at random - MCAR, missing at random - MAR, or missing not at random - MNAR).

To illustrate the difference between these three missing data mechanisms, consider the following dataset with the variables disease status, level of exposure, and age. Assume that for some participants, the exposure is missing (e.g., they do have an exposure value, but we do not know what it is).

1. MCAR: Any two individuals, regardless of their values of disease status, level of exposure, and age, have the same probability of having a missing value for exposure.
2. MAR: Any two individuals with the same disease status and age have the same chance of having a missing exposure value, regardless of how large or small their actual level of exposure is.
3. MNAR: Even among individuals with the same disease status and age, the chance of having a missing exposure value depends on their level of exposure.

Generally speaking, MCAR is usually unrealistic, MAR is somewhat plausible, while MNAR is often plausible. As such, MCAR and MAR are typically considered "ignorable" as information about the missing data itself is not included when dealing with the missing data. In contrast, MNAR is typically "non-ignorable" because the missing data mechanism must be modelled while you deal with the missing data.

Finally, you need to know what mechanism you have as this will determine your approach. For example, using multiple imputation (MI) and maximum likelihood (ML), you assume your data is at least MAR. Listwise deletion requires data to be MCAR to reduce/prevent bias in results. Diagnosing the mechanism: (Source: https://www.theanalysisfactor.com/missing-data-mechanism and https://psyarxiv.com/uaezh)

In the future, it may be worthwhile to consider using methodological approaches to address/plan for missing data in advance of running your research study using planned missingness (see Graham et al. (2006), Rhemtulla & Hancock (2016), Rhemtulla & Little (2012), Wu & Jia (2021)).