2
$\begingroup$

I am working with regression on the following dataset:

https://archive.ics.uci.edu/ml/datasets/Communities+and+Crime

Which relates the crime rate per capita (last column) with 120 different metrics, such as education, wealth, police presence, etc in different neighbourhoods across the United States.

I have little to none experience with the mainstream approaches in problems like the one that appeared here. By looking at the dataset, one can verify a pattern-like missing of certain data in some of the rows, like the following: enter image description here From what I've searched in internet, the main approach in a case of missing data (apparently it's even done by some languages like R) is to remove the rows where missing data appears.

However, if I try to that in this data-set, I end up with only 139 rows out of the 1994 that I have. I then tried to remove the columns, but then some very crucial information (all of the more 'problematic rows' refer to police presence in some areas).

I thus wanted to know what is the standard approach in this case. Clearly there is a pattern on the way the data is missing, but I am not sure on how to proceed to run around (or maybe even solve) this problem.

Improve this question
$\endgroup$
4
$\begingroup$

I guess there are a couple different problems you could try to solve here.

  1. How do I answer a question with this data given that some columns are missing?
  2. How do I fill in the data that's missing?
  3. Why are some data missing? Can I estimate the pattern?

Each question has different approaches. For question 1) you could try something along the lines of imputation, i.e. estimate what's missing as nuisances, or leave out those variables directly as you mentioned. If the data quality of those missing variables is as unbalanced as you mentioned, it would probably be best to leave this column out altogether because you have fairly limited data (10% remaining or so) where the values are actually in the dataset.

If the question is 2) there are a lot of different approaches that you could take, but one of the more interesting ones is using the other columns to impute the values of the one column. For example, you could use a linear model of $ x_{i}= \sum_{j \neq i}{beta_j x_j} $ where i is the missing column. I'm guessing this model wouldn't be the most accurate based on the amount of missingness, but you still might be able to come up with something reasonable.

Finally 3) is answered a lot like 2) probably want to make a model (for example a logistic model) of the probability a row has missing values given the other covariates. Otherwise, you might be able to guess at a convincing mechanism depending on what the variables are in question.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ +1 This answer has good advice, though I'd highly recommend that if you impute, you should also add a dummy indicator to the missing rows as I suggested., $\endgroup$
    – doubled
    Jun 13 '20 at 17:05
  • $\begingroup$ Apparently the first approach doesn't work. I obtained with a recursion decision tree an error that corresponds for about 55% of the average value for the crime. $\endgroup$
    – embedded_dev
    Jun 13 '20 at 20:03
  • $\begingroup$ I then trying using imputation and, interestingly, the improvement is minimal (it goes from around 55% to 50% of the average. I am wondering whether there is an actual solution for this. $\endgroup$
    – embedded_dev
    Jun 14 '20 at 10:01
  • $\begingroup$ If you're using decision trees/random forests, you can try to replace the missing values with large negative values. Because trees make splits on the space, it can more automatically handle missing values. This isn't a great method with things like linear regression though because coefficients can be biased. $\endgroup$
    – JoeTheShmoe
    Jun 14 '20 at 17:29
2
$\begingroup$

It seems like just 4 or 5 variables are almost always missing data, but the rest is pretty complete.. what are those variables? If they are entirely useless variables, then do you need to include them? By your numbers, those columns are missing for 95% of the values...

As to possible approach, this comes down to what is your goal. If you just wanted to understand the relationship between crime and variables, then you could omit those variables. A model is supposed to be a simplification of reality, we almost never have all the information we really want. If instead you wanted to build a causal model, then issues such as omitted variable bias could really matter because an inflation of one the other variables due to the omission of the dropped one could attribute incorrect causal effect. So your approach depends on your goals and the importance of those variables.

As to possible approaches, imputation is a common approach, and one popular strategy is the missing-indicator approach, where you add dummy variables for each missing variable that is one is that variable is missing for a given observation, and you replace the missing values all with the same value (a common one is to take the mean). Then you can think of the dummy as capturing the effect of missing that variable. You can also get fancier with some Bayesian models, but I know less about that. All of these will require certain assumptions about the underlying reasons the data is random, so be careful with any of these.

Improve this answer
$\endgroup$
1
  • $\begingroup$ There are also many other great imputation methods such as "regression imputation", hot-deck imputation, matching, bootstrapping and many more. It's a really fascinating field but you have to be careful of the assumptions the methods make. For example do you think the data are missing at random? Is there an ommited variable like doubled mentioned that is causing the missingness? $\endgroup$
    – JoeTheShmoe
    Jun 13 '20 at 17:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.