# Theory

I have three data sets, let's call them A, B and C. These data sets contain the following variables:

A = { x, A1, A2, ..., An }
B = { B1, B2, ..., Bm }
C = { A1, A2, B1, B2 }


x is the dependent variable which I'd like to predict. As you can see, only the data set A contains x. A and B are disjoint. On the other hand, the variables of C are the union of a subset of A and a subset of B.

The goal is to predict x from the B variables.

To achieve this, I thought of creating a Naive Bayes classificator for A. As soon as the classificator has learned to classify items based on the A variables, I'd include the data from C. At this point, the classificator doesn't have any knowledge of the additional variables coming from C (B1, B2), so they don't have any influence on the probability distribution of x. This way, the classificator 'extends its knowledge' onto the variables (B1, B2), and thus will be able to predict x from the B variables, without the presence of the original A variables.

# Example

Suppose the classificator is an e-mail spam filter: The x variable is the categorical variable (Spam | Not spam), the A variables are the words of the e-mail, and the B variables are other e-mail properties like it's origin (sender IP address).

• The A dataset is a collection of e-mails where we know which one is spam, but we don't know the origin IP adresses.
• The C dataset is another collection, where we don't know which e-mail is spam, but we know the e-mail text and the origins.

We want to be able to classify the e-mails based on their origin.

First, we'd create a Bayes classificator based on the A dataset. Now we're able to classify the mails based on their text. Then, we'd extend the variables by adding the C dataset, which would 'teach' the Bayes classificator what IP addresses are the sources of spam e-mail. After this step, the classificator could detect spam just by looking at the e-mail origins.

# Question

Is that possible, feasible? Are there better ways?

I suppose it kind of violates the requirements of the Bayes classificator, since the variables are not independent (the e-mail origins correlate with the words in the e-mails). But still, it seems like it could work.

• How about building a joint dataset using your sets A,B,C by joining them on the variables in C? – ziggystar Oct 5 '13 at 17:30
• That would be possible if for each item in dataset A there's an item in dataset C with the same variable values (A1, A2). But I don't have that. There are many combinations of variable values that are not present in all datasets. The classificator must be able to generalize. – cheesus Oct 5 '13 at 17:40

First, use the dataset A to train a classifier that predicts $X$ based on $A_1$ and $A_2$. Then, for each datapoint $(a_1,a_2,b_1,b_2)$ in the set C, use the classifier to compute a prediction $f(a_1,a_2)$ of $X$. Now, train another classifier using datapoints of the form $\{(b_1,b_2,f(a_1,a_2))\}$ to predict $X$ directly from $B_1$ and $B_2$. Essentially, you imputing missing values of $X$ from the dataset C by using the classifier that you trained on the dataset A.