3
$\begingroup$

In every example I see(spam, negative vs positive tweet , weather study...) there is always the assumption that the input features (or variables) are independent.

In order for me to be able to understand whether naive Bayes can be applied to my problem, I need to truly understand what dependent variables means. I know what dependent variables mean in mathematics but I can't seem to be able to grasp the notion in this sense because I've never come across a case study where the person says that they have dependent features so they can't use a particular data mining algorithm.

My main confusion is due to the fact that in my mind, all features are connected somehow(and I picture them as being dependent) for example in the weather case, if we choose humidity, temperature and wind as inputs aren't they related to a certain degree? like if it's really windy the temperature usually drops. I feel like maybe I'm misunderstanding dependency when it comes to data mining algorithms.

Can someone provide me with an example in which the independent features assumption can't apply?

Improve this question
$\endgroup$
4
  • $\begingroup$ Yes, we often treat features that are dependent as if they were independent if this is not a too big assumption and it helps us to simplify things (e.g. use Naive Bayes algorithm mentioned by you that can work pretty well in many cases even for not truthfully independent features.). $\endgroup$
    – Tim
    Commented Apr 7, 2017 at 21:10
  • $\begingroup$ In statistics we usually mean that the variables are correlation. This is different from a functional dependence in mathematics. $\endgroup$
    – Michael R. Chernick
    Commented Apr 7, 2017 at 21:25
  • 1
    $\begingroup$ Here's an example. Take a sentiment analysis problem. You want to see how people feel about coke. A model trained on Naive Bayes, would probably predict a negative sentiment on the sentence "I hate going without coke", probably because $p(negative \mid hate)$ is pretty high. However, the sentiment is clearly positive. Here's a simpler example. Take a DC circuit, knowing $R$ and $V$. Ohm's law tells us that $I = V/R$. If I did a linear regression on the current $I$ with inputs $V$ and $R$, (i.e. modelling $I = w_0 + w_1 V + w_2 R$) I'd get a very bad result. $\endgroup$
    – Bridgeburners
    Commented Apr 7, 2017 at 21:48
  • $\begingroup$ @Bridgeburners thank you for the examples. This clears up my confusion $\endgroup$
    – engineering student
    Commented Apr 7, 2017 at 21:54

1 Answer 1

Reset to default
5
$\begingroup$

In Naive Bayes, they are talking about conditional dependence. First let's start with conditional independence. $X_1, \ldots, X_n$ are conditionally independent given $C=c$ if for all $x_1, \ldots, x_n,c$ $$ F_{X_1,\ldots,X_n|C}(x_1,\ldots,x_n|c) = F_{X_1|C}(x_1|c)F_{X_2|C}(x_2|c) \times \cdots \times F_{X_n|C}(x_n|c), $$ aka the joint cumulative (conditional) distribution function factors into (conditional) marginal cumulative distribution functions.

So they are dependent if the above is not true. There only has to be one sequence $x_1', \ldots, x_n',c'$ such that $$ F_{X_1,\ldots,X_n|C}(x_1,\ldots,x_n|c) \neq F_{X_1|C}(x_1|c)F_{X_2|C}(x_2|c) \times \cdots \times F_{X_n|C}(x_n|c). $$ Intuitively this means that knowing something about one random variable will tell you about the probabilities of the others.

One example where this is true, say $C$ denotes the category of someone who either has a disease or does not. Let's say $1$ means they do and $0$ means they do not. Then let $X_1, \ldots, X_n$ be random measurements of this person. Depending on the measurements, $F_{X_1,\ldots,X_n|C}(x_1,\ldots,x_n|c)$ is not likely to factor, so they would be conditionally (on the disease status) dependent measurements. Intuitively this means that if you're looking at a person with the disease, the doctor is likely to know that the measurements he/she is about to take are likely to be related somehow.

Improve this answer
$\endgroup$
1
  • $\begingroup$ is this assumption true about Naive bayes that the test data is normally distributed or is it that the test data is linearly seperable ? $\endgroup$
    – MUK
    Commented May 31, 2021 at 10:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.