2
$\begingroup$

I have 70,000 observations for my dependent variable. I have 12 independent variables. After removing zero value and error and missing value form my data set, my data reduced to 4000. Can I still do the multiple linear regression with this data set? I think 4000 data is more than enough for 12 independent variables, but I am not sure whether removing almost 90% of observations will harm my regression or not?

$\endgroup$
1
$\begingroup$

We'd probably need to know more about the nature of missing and the design of the study.

Generally, if the missing pattern is random, then your regression of n=4000 would still be representative. However, if the missing is associated with both outcome and exposure, then they will become confounders that are unaccounted for. In that case, even you have 4000 and only 12 independent variables, your regression results will very likely be off, over even plain misleading.

Having said that, you really need to explain why such a drastic cut. Some research designs invite a lot of missing. For instance, online questionnaires with price draw usually have this magnitude of missing. Most online respondents may just enter the survey, click through all questions without answering, and leave their e-mail to enter to lucky draw. Some other, like face-to-face interview, should never have missing this prominent.

If it's secondary data, then I'd recommend you to consult their study design documentation. Some study would only take a subset for further investigation, and may create an illusion that the others are all missing. For instance, a health study may collect all height and weight of the participants, but only random selects 10% of them for a blood test due to cost.

Studying the original questionnaire may also help. Some data may record N/A as missing. If you have accidentally chosen a question after a certain skip pattern, you may lose a lot of sample. For instance, there could be a question asking if the respondent had tried crack cocaine, and if yes, then there are a few more follow-up questions. If you have picked one of those follow up questions, big time missing can happen.

Based on what the nature is, you can address them differently in your report. But how and what to say about this problematic missing rate would depend on your study and the questions.

$\endgroup$
  • $\begingroup$ Thanks for your reply. Actually the data that I am working with is not from the questioneares, it's the data collected form GPS to record the place of the vehicle. there are several occassions when the GPS does not record the place of the vehicle, especially in CBD area. My independant varibles come from the GPS data, for example speed. Do you think this is the random missing pattern or not? $\endgroup$ – rose Sep 28 '13 at 23:49
  • $\begingroup$ @rose If you know that the signals tend to be missed in CBD areas, then chance is it's probably not random. There are two points here: 1) if it's time series data tracking many travellers, then there are techniques and assumptions you can apply to estimate the missing values; and 2) because it involves spatial sampling, I think you should also revise this question with more emphasis on the spatial aspect, and consult the GIS board. But do not copy and paste the same question, that'd be counted as cross-posting and is generally not recommended. $\endgroup$ – Penguin_Knight Sep 29 '13 at 1:22
1
$\begingroup$

It's not a simple yes/no answer, but most people will be skeptical and will be hesitant to trust your study.

Non-random missing data are what would make it an issue. You can identify non-random missing data by making a new binary variable that identifies whether the case has any missing data across all your variables in the model. For example, if subject #1 has at least 1 missing data, identify subject#1 as "N". If subject#1 has the full set of data, identify as "Y". Now, you can compare the Y and N groups on all the variables you have in the model (use graphs). Look at the central tendency, as well as the spread. If there is any difference, it's a problem.

There may be one or two variables that are reducing your sample size by a lot. It's your choice, but you can identify those variables and remove them from the model to limit loss of data, especially if those variables are not as important as the others.

$\endgroup$
  • $\begingroup$ Thanks for your reply. As i am working with GPS data, whenever the GPS does not detect the location of vehicle, my depandant and all independant variables become zero, because the independant and dependant variables calculated based on GPS information. $\endgroup$ – rose Sep 29 '13 at 0:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.