6
$\begingroup$

I had few data analysis and modelling roles and was always "unfortunate" to work with, let's call it bad data. Think of default probabilities 1 of 10 000, or sales time series that has 3 sold products per day which occasionally jumps to 10 or drops to 0 with certain seasonal component of course.

What do you think, can one even to proper analysis on these kinds of sparse data? If yes, what techniques would you recommend?

$\endgroup$
1
  • 2
    $\begingroup$ This question is too broad for a complete answer. $\endgroup$ Commented Mar 25, 2017 at 13:01

1 Answer 1

9
$\begingroup$

Pretty sketchy question but answerable nonetheless. The answer is "yes," there is lots one can do with sparse data. This response is far from "complete" but will review a few options in a kind of "DIY" shotgun listing. In other words, it is up to the analyst to decide which option may be appropriate to pursue.

The first consideration is to identify where the sparsity is occurring, e.g., is it in the features in terms of a large, complex, combinatorial set of possibilities or is it wrt the target or dependent variable which may have few observed responses, or both?

Wrt sparse or rare events in the target variable, e.g., when response to a stimulus is recorded as 0,1 or "yes/no," and the response rate is very small, one common error is to model this using logistic regression. The mistake is this: it is well known that the logistic curve does not provide a good fit to the tails of its distribution. This means that with sparse or rare event data, logistic regression will produce biased results. Commonly recommended "solutions" for this problem are to go out and get a larger sample of data or, alternatively, to specifically subsample those segments that are both important to the analysis and sparsely populated. This is a bad idea for at least two reasons: one, it's not always possible to simply "get more data" and two, even when possible, it can be prohibitively costly in terms of time and money. Better solutions are possible:

Concerning features, one needs to distinguish between structural zeros which occur for logically impossible combinations and sparsity where a combination of features is possible, there just isn't enough information to populate that particular cell in the table. Consider healthcare or hospital data where a combination such as male patients given a diagnostic code of "pregnancy" is possible from a purely computational point of view in terms of cross-classifying a set of features, but males actually giving birth is impossible, i.e., it is to be considered a structural zero. But sex and gender are different constructs. So, until transgender patients (e.g., female to male gender) have children, this will remain a structural zero.

As noted, sparsely populated features are different, requiring special tools to facilitate analysis from those employed for target variables. The following is a "laundry list" or shotgun set of options for dealing with sparse features. Much of it was gathered by simply browsing for the keywords "inference from sparse data." Choose carefully from the list:

And so on.

Good luck.

$\endgroup$
5
  • $\begingroup$ This appears to be a nice long answer but not complete as I am sure the author would agree. $\endgroup$ Commented Mar 25, 2017 at 14:46
  • 2
    $\begingroup$ @MichaelChernick Agreed, as already noted. Frankly, I don't know what a "complete" answer would be to such a vaguely worded question. Moreover, I'm probably guilty of giving the OP enough rope to hang himself. $\endgroup$
    – user78229
    Commented Mar 25, 2017 at 15:04
  • $\begingroup$ @DJohnson. Thanks for the answer and references. My bad I wasn't precise enough. What I had in mind is following. I want to model relationship between probability of default and credit rating (AAA to CCC) in some country where there are 1 000 publicly rated companies, out of which only hand full defaulted. Even worse, some BBB rated company might have defaulted, while none CCC company defaulted. Even if I fit some model, I guess the result will be counterintuitive and companies with better credit rating will have higher default probability. Bayesian with strong priors might help though.. $\endgroup$
    – vukivu
    Commented Mar 26, 2017 at 15:39
  • $\begingroup$ To me, what you are asking for is a kind of risk management shadow credit rating. One approach would be to get known credit default probabilities from published sources such as provided by S&P or Fitch. It would be interesting if they were available by year as well as for different countries. Then, by pooling the information across time and country, a more robust estimate of default could be derived that would also adjust for differing levels (slopes) by country. $\endgroup$
    – user78229
    Commented Mar 26, 2017 at 15:48
  • $\begingroup$ This could be done in a Bayesian framework based on iterative MCMC sampling evaluating the posterior or with random forest resampling using an OLS (not CART) panel data structure as the core engine. $\endgroup$
    – user78229
    Commented Mar 26, 2017 at 15:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.