8
$\begingroup$

What does it mean when people within the field of Machine Learning talk about biased datasets? I thought it was only estimators that could be biased.

In documenting work I have done, I am being asked:

"All datasets are biased in some way. How is your dataset biased?"

I have no clue what they mean. Also my dataset is population data so I have a hard time interpreting it as sample selection problem ...

The objective of the analysis is primarily descriptive. I will construct three indeces and combining them. The three indeces are 1) Housing price index, 2) Transportation time index and 3) Wage index.

I know, that even with population data there is a sample selection problem for the housing prices - because not all houses are traded every year and the subsample that are traded are most likely not a random sample of the total stock of housing.

I am not generalizing beyond the time period for which I have data which is a 30-year period.

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ Imbalanced is the correct word. I think that bias has become politicized, so it might lose its technical meaning. About 2% of people are Myers-Briggs ENTJ. This means that in 10,000 personality profiles about 200 would be ENTJ. If you went to an ENTJ convention as part of your data collection, and half of your data (5000 rows) was ENTJ, then you would have to find a way to discard about 4900 samples, or intelligently clone the rest to rebalance the data. $\endgroup$
    – EngrStudent
    Commented Oct 16, 2020 at 16:44
  • 2
    $\begingroup$ Bias can come in many flavors, so it would be helpful to clarify what you hope to achieve with your analysis. What is the question you hope to answer? Is it predictive, descriptive, causal? $\endgroup$
    – dimitriy
    Commented Oct 16, 2020 at 17:42
  • 2
    $\begingroup$ For example, having the whole population often does not answer causal questions since you are still missing the other potential outcomes for everyone in your population. Even for predictive questions, the population may not be sufficient if you hope to generalize your results to other periods. For example, a conversion algo you build to predict purchase from custom prices does not take into account your competitor response to you pricing policy. $\endgroup$
    – dimitriy
    Commented Oct 16, 2020 at 17:42
  • $\begingroup$ @DimitriyV.Masterov I have edited the question providing further detail of the analysis and data. $\endgroup$
    – Jesper for President
    Commented Oct 16, 2020 at 17:51
  • 1
    $\begingroup$ en.wikipedia.org/wiki/Selection_bias $\endgroup$
    – Glen_b
    Commented Oct 17, 2020 at 5:19

3 Answers 3

Reset to default
7
$\begingroup$

The term “biased” simply means, that your sample is not chosen randomly.

This is similar to a biased dice, which produces number 6 more often than the other numbers.

It is always difficult how to obtain an unbiased sample, but some notoriously known errors are:

Here and here are listed and explained some other types of biases.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ the first line of this answer is wrong. randomness and representativeness are not the same thing $\endgroup$
    – carlo
    Commented Oct 19, 2020 at 15:10
  • 1
    $\begingroup$ @carlo, no serious sampling book gives a definition of representativeness. It's only an intuitive concept. And representativeness is about randomization, too, so we are back in the realm of randomness. $\endgroup$
    – MarianD
    Commented Oct 19, 2020 at 15:51
  • $\begingroup$ still, they are no synonym, you can make an incredibly biased random sampling if you want $\endgroup$
    – carlo
    Commented Oct 19, 2020 at 16:25
  • 1
    $\begingroup$ @carlo, “you can make an incredibly biased random sampling if you want” — randomness and want? I may want to obtain 6, but a fair dice will give me what it “wants”, not me. I don't understand what you mean. $\endgroup$
    – MarianD
    Commented Oct 19, 2020 at 17:59
  • $\begingroup$ @carlo, “still, they are no synonym”, you're right. I didn't assert the opposite, so I don't understand, why you wrote this sentence. $\endgroup$
    – MarianD
    Commented Oct 19, 2020 at 18:06
7
$\begingroup$

From working as a statistician where my main role is a consultant to the subject matter experts that also work for us, I have noticed that people with less of an understanding of statistics throw the word bias out when they just want to say something is wrong.

They really have no idea what they are saying when they are saying something has bias and will say it anytime they are concerned as a kind of catch all even if the context has nothing to do with bias. Many times when I am explaining something to someone they respond "what about bias" even though it has nothing to do with conversation at hand.

I suspect this may be the case with your scenario specifically when you see them saying something like:

"All datasets are biased in some way. How is your dataset biased?"

Which is certainly not true.

Just a note that this gets multiplied when we start talking about buzz words like machine learning. I've had people give me a dataset and ask me "can you machine learn this....".

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ As a pessimist on human nature I always assume the worst and are therefore inclined to say you are probably spot on :) $\endgroup$
    – Jesper for President
    Commented Oct 16, 2020 at 17:53
2
$\begingroup$

Perhaps you know that when iPhone users text each other, there is a blue “send” arrow instead of the green that you get when you text someone who uses another type of phone. To collect data, you randomly text numbers, but only if the arrow is blue. Your sample is biased, since you’ve excluded people who, for whatever reason, do not use iPhones. Perhaps political viewpoints influence phone purchase decisions. If you were texting about something political, you’ve excluded certain viewpoints.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Yes, ok so I interpret both your answers and the comment from @EngrStudent as saying that I should most likely interpret it as Sample Selection Bias. However I have population data as in micro osbervations of everyone in the country ... so I guess im good. $\endgroup$
    – Jesper for President
    Commented Oct 16, 2020 at 17:33
  • $\begingroup$ To what population are you trying to draw inferences? $\endgroup$
    – Dave
    Commented Oct 16, 2020 at 17:36
  • $\begingroup$ For the population of the country under consideration. The theoretical field is international but whether the mechanisms hypothesized by theory and "shown" to hold in one country will offcourse always just be suggestive. $\endgroup$
    – Jesper for President
    Commented Oct 16, 2020 at 17:42
  • $\begingroup$ So you’re going to poll everyone in your country and then determine that the whole world feels that same way? $\endgroup$
    – Dave
    Commented Oct 16, 2020 at 17:44
  • $\begingroup$ No I am not going to do it, someone else did it for me, for the last 30 years. And it is not so much about what people feel more about how long they have to commute to get to work, how much they pay to live where they choose to live and how much they earn. Standard economics stuff. $\endgroup$
    – Jesper for President
    Commented Oct 16, 2020 at 17:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.