Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have a dataset of 380 samples of 6 variables. These variables are counts of different types of events in each of the 380 defined regions. These counts are per month, which means that I have several of these datasets (for now, I only have four months).

When looking at the data, I can clearly see that there is some missing (or incomplete) data. For instance, the counts for one given region are about the same for all months, except one (where it's close to zero). However, I does not seem likely that there were actually that few events during this period.

What I would like is to be able, given data for a few months, to detect missing or incomplete values, and possibly to correct/complete them. Detecting missing values is not as obvious as looking for zeros, because it may happen that no event occurred in some regions.

I read a few things about matrix factorization, but I'm not sure it would apply to my case. It seems suited for the cases where you know what data is missing.

I assume this kind of problem is be quite common, for instance in biology for population estimation.

share|improve this question
add comment

2 Answers 2

I think you are really looking for outliers in your data (small values where most 'similar' values, i.e. values where a set of covariates hold the same values, are 'bigger').

You could look at outliers in MV data for this for starters, although you may be able to utilize some particularities of your situation (you are only interested in outliers in one variable, and only in one direction).

If your sample size is big enough for that, you can simply use the outlier definition as used in most boxplots (1.5 IQR away from the outer quartiles). If not, you should apply some truly parametric way of detecting outliers (e.g. residuals in a regression).

share|improve this answer
add comment

Dealing with missing observation is a never ending problem. There are a few approaches i will mention, but I am sure others will add more.

  1. Make sure there is no any problems in recording the data, such as systematic omitting certain values. Given you have no control for it, then other options need to be considered.

  2. Based on the proportion of missing values, you have different options.

If there are less than 5% missing observations:

2.1 I would not care, and use either average or median based on the shape of the sample's distribution. Some use this approach even when there are more than 5% missing observations. It is a very quick solution when time pressed.

If there are more than 5% missing observations, some advanced techniques can used:

2.2 Data imputation methods is a wide class of techniques to impute missing values based on the observed values of the variable/or other variables.

For example, in time series framework, some values could be forecast based on the observed time series data.

In regression framework, knowledge of other variables as well can be used to impute missing variables, i.e., build a regression model for the variable of interest (the one with missing values) as a function of other variables available (probably excluding the original dependent variable Y) and then predict the missing values using the model.

A more advanced technique is Expectation Maximization method.

Finally, a dummy variable can be used just to control for the missing values.

M

share|improve this answer
    
I cannot recommend using "dummy variable adjustment" (aka "a dummy variable can be used just to control for the missing values."). It is known that "this method generally produces biased estimates of the coefficients" (Allison, Missing Data, p. 10). –  Bernd Weiss May 22 '11 at 19:32
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.