0
$\begingroup$

Say i am training a neural network and have 10 samples with 4 variables each and 1 label assigned to each observation. What does it mean to say that the samples are independent and identically distributed?

I know what IID variables are. And i have read statements such as "Because we usually assume that our samples are independent and identically distributed, the likelihood over all of our examples decomposes into a product over the likelihoods of individual examples: text omitted". What exactly does it mean for the samples (here 4 variables and 1 class label) to be iid?

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
0
$\begingroup$

That samples are identically independent distributed (IID) implies two properties of their generation:

  • Identically - means that all data are coming from the same data generating process. So there are no difference in expectation on the samples you see during and the samples you see in the real world. So if you are predicting out of time or have a bias sample. This is usally not true.
  • Independent - data are generated independently from each other given the individual X. So the datageneration of one sample has no effect on another sample. For example probability to get cancer is proporly close to independent across a population since it does contaminate. While flue is dependent if your friend is getting a flue you are more likely to also get it and therefore the observation of you and your friend is correlated and not independent if both you and your friend is in the data. If how many of your friends having flue fully describe the inter observation dependence then you can make your data independent by including this variable. Since observations are now independent given friend data.
Improve this answer
$\endgroup$
2
  • $\begingroup$ IID - Independent and identically distributed. So when we are talking about distribution, What are the random variables? In this case where i have 10 samples would it mean the random variable is the label (= class) of each sample? so in all im talking about 10 iids? $\endgroup$
    – figs_and_nuts
    Jun 29, 2017 at 7:46
  • $\begingroup$ You have 10 iid draws. In general an observation both label and variables are one draw. Variables and label can be dependent in any way shape or form. If the label is not dependent on the variables it is pretty bad. Typically unbiased models are obtained if $P(y_i|X_i)=P(y_i|X,y_{not-i})$ since this is the relationship you use to derive the likehood function. $\endgroup$
    – Peter Mølgaard Pallesen
    Jun 29, 2017 at 8:04

Not the answer you're looking for? Browse other questions tagged or ask your own question.