6
$\begingroup$

I have data that is non-IID, and I want to estimate if the dependence is bad enough that it will have a noticeable effect on a fitted classifier. I don't think the exact model type will matter in this case, but for argument's sake let's say I'm using elastic-net logistic regression. In this case the dependence takes the form of clustering among observations, in that if $Y_k = 1$ has a high probability then the probability that $Y_j = 1$ is very low for all $j\neq k$ within the data cluster.

Ideally, I would like to be able to compare the fitted model from the non-IID data set to a "comparable" or "similar" IID data set. So I'm thinking I could just simulate such a data set, fit the model on the fake data and the real data,

  1. Is there a formal or rigorous definition of "similarity" that makes sense in this case? I certainly know a dissimilar data set when I see one, but it's hard to quantify exactly how I know.
  2. Is there a straightforward way to generate an IID dataset from a non-IID data set that otherwise preserves some structure from the joint distribution of features?
  3. Is this an X-Y problem? Is there a better way to evaluate the effect of data dependence on my estimates?

edit:

  1. For a purely predictive task, does non-IID data even make a difference as long as the cross-validation procedure is constructed correctly? This answer suggests the answer is "no"
$\endgroup$
  • 1
    $\begingroup$ Elaborate more on why your data isn't IID? and what IID data might look like? $\endgroup$ – Matthew Gunn Oct 25 '16 at 23:46
  • $\begingroup$ So you have negative correlations within clusters? $\endgroup$ – kjetil b halvorsen Sep 8 '17 at 13:18
  • $\begingroup$ @kjetilbhalvorsen yes $\endgroup$ – shadowtalker Sep 8 '17 at 14:51
  • $\begingroup$ You could try keeping the same data set, but re-assign the class values $(Y_k)$ independently at random (having the same overall proportion of $Y_k = 1$). You could run this analysis many times and see what changes. I've used a similar approach in a different problem, but it may not work here. $\endgroup$ – knrumsey May 25 '18 at 0:40
  • $\begingroup$ General: What is the classification? Question 1: What do you mean by similarity, e.g. having the same pdf? Q2: I don't understand what you mean. Q3: it might be, but I again I don't understand the example. In bullet point 4 it becomes clear, that two things are mixed: correlation (read as: non-independence) and generalization power are not really related. In other words: if there is correlation in your input/training data, then this does not mean that you're result/prediction quality is worse. $\endgroup$ – cherub May 25 '18 at 14:17
3
+700
$\begingroup$

I have data that is non-IID, and I want to estimate if the dependence is bad enough that it will have a noticeable effect on a fitted classifier. I don't think the exact model type will matter in this case, but for argument's sake let's say I'm using elastic-net logistic regression.

On the importance of the i.i.d. assumption in statistical learning

Regularized Logistic Regression: Lasso vs. Ridge vs. Elastic Net

Predictive Modeling - Should we care about mixed modeling?

Ideally, I would like to be able to compare the fitted model from the non-IID data set to a "comparable" or "similar" IID data set. So I'm thinking I could just simulate such a data set, fit the model on the fake data and the real data,

  1. Is there a formal or rigorous definition of "similarity" that makes sense in this case? I certainly know a dissimilar data set when I see one, but it's hard to quantify exactly how I know.

An R software library can help, consider: "Statistical Patterns in Genomic Sequences" by Hart and Martínez which could be extended to work with arbitrary symbolic sequences, not only DNA sequences.

It features easy to use functions:

  • diid.disturbance - Construct feasible Random Noise Generating a Bernoulli Process

    Produces a sequence of random noise which would generate an observed sequence of finite symbols provided that the sequence of symbols results from a Bernoulli process.

  • diid.test - A Test for a Bernoulli Scheme (IID Sequence)

    Tests whether or not a data series constitutes a Bernoulli scheme, that is, an independent and identically distributed (IID) sequence of symbols, by inferring the sequence of IID U(0,1) random noise that might have generated it.

    Category       Function           Test
    Uniformity     ks.unif.test       Kolmogorov-Smirnov test for uniform$(0,1)$ data
                   chisq.unif.test    Pearson’s chi-squared test for discrete uniform data,
    Independence   lb.test            Ljung-Box $Q$ test for uncorrelated data
                   diffsign.test      signed difference test of independence
                   turningpoint.test  turning point test of independence
                   rank.test          rank test of independence
    

Search for a library that exactly meets your requirements or modify one that nearly does.

Also check out the NIST SP 800-22 Statistical Tests which can be used to test your data.

  1. Is there a straightforward way to generate an IID dataset from a non-IID data set that otherwise preserves some structure from the joint distribution of features?

Create something like a k-distribution by compounding some existing distributions to meet your requirements. Use something like DistributionFitTest to check the generated IID.

  1. Is this an X-Y problem? Is there a better way to evaluate the effect of data dependence on my estimates?

A tiny bit XY, maybe; you've waited a year and a half for an answer, even Tweeted it. You made a minor edit. If I don't get an answer in less than a week I make certain to check if I missed something. You've been asked a few questions and answered one. I have no complaints or questions ...

Test with both (non-IID, IID) generated random models (easy with the software suggested above, though probably more power in a different package), your own data, and both. Create a set of "Ensemble Algorithms" that don't blow up.

  1. For a purely predictive task, does non-IID data even make a difference as long as the cross-validation procedure is constructed correctly? This answer suggests the answer is "no": Predictive Modeling - Should we care about mixed modeling?.

The answer requested further information and links, further it was not an absolute "no".


See also:

"Choosing the Correct Statistical Test" (Derived from Dr. Leeper's work) - General guidelines for choosing a statistical analysis and links showing how-to using SAS, Stata, SPSS and R.

Choosing a statistical test, Data Analysis Resource Center, and GraphPad Analysis Checklist.

I might have time to return for another edit and improve this answer.

$\endgroup$
  • 2
    $\begingroup$ This is exactly what I was hoping to get. I appreciate you taking the time to read between the lines a bit; I struggled to express what I wanted clearly at the time, and even now reading over the question i can see what's wrong with it but I'm not certain how to fix it. I'm going to let this bounty expire so I can award you a bigger one. $\endgroup$ – shadowtalker May 29 '18 at 14:01
  • 1
    $\begingroup$ Yep, that's what I did. You already have one, the next one is coming once the 24 hour window passses. $\endgroup$ – shadowtalker May 29 '18 at 21:05
  • $\begingroup$ @shadowtalker - 👍 $\endgroup$ – Rob Jun 1 '18 at 15:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.