# What to do if samples aren't comparable?

I have an unbalanced panel data set, which covers a set of about 10 variables I am very interested in as controls in a regression. However, only a few of these variables are available in all waves: most of them are only available for a few waves down to two which are only available for 5 waves. Additionally, these variables have high rates of missingness.

If I just create a sample where all variables are available I lose a massive share of observations and years. So I split the sample in several subsamples each depending on the availability of these variables, i.e.

Sample 1 - very small sample but all variables are available for all obs

Sample 2 - somewhere in between

Sample 3 - all waves and observations, but for many individuals observations on the interesting variables not available.

So now a t-test was used to check whether the mean of the dependent variable and some other unaffected key variables differ among these samples. Unfortunately, they do in most cases. I did the t-test in the first instance to prove that for example the dependent variable does not significantly differ between samples.

I am not sure what to do with the information, that they do differ. Is there anything I can do?

• why the data is missing? if the fact that the data is missing tells something about this observation, then you're in trouble – Aksakal Mar 3 '14 at 4:56

The problem of missingness can be handled by a range of different methods (See here for a review). In general, in order to handle missingness you need to be able to make the assumption that conditional on observed covariates, the probability of missing an observation is independent of the true value of the observation. This is called the missing at random assumption. Multiple Imputation (MI) seems to be favored for many applications, but it does require the correct specification of the models used for imputation.

You also have structural missingness in that some variables just aren't collected in some waves. I don't think MI is necessarily appropriate for structural missingness. I've been told to use last value carried forward in that context, but I haven't seen any formal justification for that suggestion, and would be interested to hear other's opinions.

There are three issues (at least) in this question and they are interrelated:

1) What to do about (nonstructural) missing data? 2) What to do about structural missing data? 3) How to analyze the data

Regarding 1) the most common method is multiple imputation. Technically, it requires the missing at random assumption (see @Jeremy's answer) but I have heard Joe Schafer of Penn State say that MI works well as long as the data aren't "Really missing at random. That was a remark at a talk and it was quite some time ago, so there may be more formal confirmation (or disconfirmation) of that claim.

Regarding 2) Depending on 3) there may be no need to do anything.

3) Depends on what you are trying to do. However, you should be using a method that deals with the dependence of the data. Multilevel models and generalized estimating equations are two common approaches.