I have three data sets, let's call them A, B and C. These data sets contain the following variables:

A = { x, A1, A2, ..., An }
B = { B1, B2, ..., Bm }
C = { A1, A2, B1, B2 }

x is the dependent variable which I'd like to predict. As you can see, only the data set A contains x. A and B are disjoint. On the other hand, the variables of C are the union of a subset of A and a subset of B.

The goal is to predict x from the B variables.

To achieve this, I thought of creating a Naive Bayes classificator for A. As soon as the classificator has learned to classify items based on the A variables, I'd include the data from C. At this point, the classificator doesn't have any knowledge of the additional variables coming from C (B1, B2), so they don't have any influence on the probability distribution of x. This way, the classificator 'extends its knowledge' onto the variables (B1, B2), and thus will be able to predict x from the B variables, without the presence of the original A variables.


Suppose the classificator is an e-mail spam filter: The x variable is the categorical variable (Spam | Not spam), the A variables are the words of the e-mail, and the B variables are other e-mail properties like it's origin (sender IP address).

  • The A dataset is a collection of e-mails where we know which one is spam, but we don't know the origin IP adresses.
  • The C dataset is another collection, where we don't know which e-mail is spam, but we know the e-mail text and the origins.

We want to be able to classify the e-mails based on their origin.

First, we'd create a Bayes classificator based on the A dataset. Now we're able to classify the mails based on their text. Then, we'd extend the variables by adding the C dataset, which would 'teach' the Bayes classificator what IP addresses are the sources of spam e-mail. After this step, the classificator could detect spam just by looking at the e-mail origins.


Is that possible, feasible? Are there better ways?

I suppose it kind of violates the requirements of the Bayes classificator, since the variables are not independent (the e-mail origins correlate with the words in the e-mails). But still, it seems like it could work.

  • 1
    $\begingroup$ How about building a joint dataset using your sets A,B,C by joining them on the variables in C? $\endgroup$ – ziggystar Oct 5 '13 at 17:30
  • $\begingroup$ That would be possible if for each item in dataset A there's an item in dataset C with the same variable values (A1, A2). But I don't have that. There are many combinations of variable values that are not present in all datasets. The classificator must be able to generalize. $\endgroup$ – cheesus Oct 5 '13 at 17:40

I'm not sure exactly what you mean by "extend the variables", but one way to solve this problem is to train two separate classifiers.

First, use the dataset A to train a classifier that predicts $X$ based on $A_1$ and $A_2$. Then, for each datapoint $(a_1,a_2,b_1,b_2)$ in the set C, use the classifier to compute a prediction $f(a_1,a_2)$ of $X$. Now, train another classifier using datapoints of the form $\{(b_1,b_2,f(a_1,a_2))\}$ to predict $X$ directly from $B_1$ and $B_2$. Essentially, you imputing missing values of $X$ from the dataset C by using the classifier that you trained on the dataset A.

In terms of your example with emails, the first step would be use the data in A to build a classifier that tells you, based on the content of an email, whether an email is likely to be spam. Then, for the set of emails in C, where you have both content and metadata, you build a second classifier that tells you, based on metadata, whether the content of an email is such that your first classifier would judge it likely to be spam. This second classifier can then be used to attempt to predict spam from metadata alone.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.