1
$\begingroup$

I have blanks in my continuous data as the title says. I'm trying to do a pretty complex regression. I don't want to really mention exactly what type as I'd rather leave that detail out for the sake of trying to get a more general strategy or approach (unless it is absolutely necessary).

The variables that have blanks that are irritating me the most are "confidence percentages" given to me by another database that returns information on a specific IP address when you give it one.

These confidence intervals are obviously very important in the type of regressional analysis I'm doing. I know a blank just doesn't mean "nothing," a blank data point is also a sign that points to some form of variance. It would be easy to recode if it was in a categorical variable column of observations I had but it's not....it's in the list of continuous ones.

I tried doing further background research on these "blanks" by contacting the company the database is governed by and actually got ahold of their engineers (actually did this to two different companies that provide these types of databases)....their responses to my question?

"Well, we just don't have the data."

Me: "Oh, well is there any sort of context you could provide for me surrounding these 'blank cells' of data?"

"Nope, there's nothing...we just don't have the data."

Me: "Yes, as you said before, but there must be some reason you don't have the data - for instance, what caused you to decide to return blank data for that IP address? I mean, blank data cells don't mean -nothing- they are also a sign that points to some degree of variance - actually, they account for a huge degree of variance in your data and I can't recode them as 0s because that skews your continuous 'confidence percentage metric' by a GREAT deal. (They already know what I'm attempting to do regression wise btw - and one that wouldn't be competing with their services but leverage them and spend alot of money with them.) "

"Well, It's just because we don't have any data for it."

Me: "I know but I just need some clues as to why there is no data...to help me formulate some sort of strategy."

"We don't have any clues, we just have nothing because there isn't any data for that."

Me: "Ok, thanks for the customer service."

I was even on the phone with the second company on a conference call with about 5 other people who work there and it was as if what I was trying to explain bewildered them.

Most likely I suppose sharing the information I was trying to get at could of been proprietary...but who knows.

Anyways I'm left with these blanks - they tend to cause huge variances in the outcome when they appear - but they are listed as continuous variables so I can't make them a category...and can't make them a "0" as that would just be incorrect.

I'm thinking of adding some sort of regressional tree matrix as a first step (or somewhere step) within my regression to try to account for these, but I'm not exactly sure this is the right strategy.

What would you all suggest I do?

$\endgroup$
4
$\begingroup$

What you should do exactly really depends on the fine details of your project. Missing data (blanks) is a very common problem in most statistical analyses. A general strategy is hard to define other then the rather trivial one of reading what solutions have been proposed.

A short introductory text, which is good for getting a quick overview is:

Paul D. Allison (2002) Missing Data. Thousand Oaks: Sage.

Longer more detailed texts are:

Roderick J.A. Little and Donald B. Rubin (2002) Statistical Analysis with Missing Data, second edition. Hoboken: Wiley.

J.L. Schafer (1997) Analysis of Incomplete Multivariate Data. Boca Raton: Chapman & Hall/CRC.

Don't expect any miracles from these techniques; they cannot change the basic fact that there are only two types of information entering a statistical model: the stuff we have seen (the data but also the kind of information you tried but failed to obtain from customer support) and the stuff we have imagined (the assumptions).

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

@MaartenBuis gave several good references and I certainly agree that what you should do depends on the details of your project. However, I'd say that, more often than not, the right thing to do is multiple imputation.

You didn't say what software you are using, but the talk of categorical and continuous makes me think it might be SAS. In that case, see PROC MI and PROC MIANALYZE documentation and references therein (which I am pretty sure includes both of the "longer more detailed" texts.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ It's actually XLStat - I've grown a liking to them...and I can't get away from Excel - ha. The type of regression I'm attempting to do is something in their software called "correlated component regressional analysis". More info about it is here: xlstat.com/en/products-solutions/ccr.html. I'm attempting to eventually regress (after possibly a few decision trees) with a binary logistic regression (it's a part of what CCR can do). I'll look up multiple imputation, which I'm assuming is forming a model to fill out the data that is missing? Regardless, I'll look it up. Thanks! $\endgroup$ – Taal Sep 27 '13 at 15:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.