Bayesian approach for comparing the predictability of different datasets for another

Question

Suppose I have three datasets A, B and C with not necessarily the same amount of data. Now, I want to know whether dataset A or dataset B is better in predicting C. I want to use a Bayesian approach here.

To give more details about what I mean with datasets. Let us assume the following three datasets (variables).

A   B   C
----------
2  0.6  10
1  0.7  5
10 0.2  50
3  0.3  1

Let us assume that C refers to the amount of burgers kids in a school eat over the period of a year. Now, we want to know whether (A) the number of siblings of the kids or (B) the social class of the family rated from 0 (poor) to 1 (rich) can better predict C.

I have some ideas which I quickly want to elaborate next. However, the main goal of this post is to get ideas of how I would tackle the task of determining whether A or B are better in predicting C within the Bayesian framework.

My current idea is to build two models (with different priors) and calculate the Bayes factor with data C to check their relative plausibility. The priors should resemble data A and B. Thus, the priors could be posteriors from A and B with some kind-of flat proto prior. In simple terms, we observe data A and B, determine posteriors for both of them, and use them as priors for C. Now, we want to compare them by using Bayes factors.

Another, more simple idea of mine is to just calculate correlation between A or B and C in a Bayesian way. Or, we could also focus on regression or similar methods.

Answer 1

I found the wording of this question a bit confusing, and moreover, I'm not quite sure why a Bayesian technique was suggested as a possible solution.

Here's the question that I think was being asked:

I think the original poster has a table that looks something like this:

 A      B      C
-----------------
0.08  0.17  -0.72
0.13  0.29  -0.45
0.26   --    0.38
0.41  0.56   0.87
 --   0.77   1.42
0.63  0.91   1.69

In other words, variable C is always present, but in some rows, variable A is sometimes missing (e.g., because it wasn't measured, or whatever), and in other rows, B is sometimes missing.

The poster posits that there might be various different types of relationships possible between A and C, or between B and C, which can be illustrated, at least at a conceptual level, with a couple of figures:

If the relationship between A and C (or alternatively, between B and C), looks more like the figure on the left, then we say that the two are distributed randomly with respect to one another, and having knowledge about variable A offers no predictive power whatsoever about the value of C.

On the other hand, if the relationship between A and C looks more similar to the figure on the right (namely, that the two variables sort of track with one another, albeit not necessarily in a linear way) then having knowledge about variable A obviously does offer some predictive power over C, even if the prediction may not be perfect.

I think that the original poster is probably asking:

• For either variable A or B, compared against C, how can he distinguish whether the relationships hidden in his data look more like the plot on the left or the one on the right?
• If it turns out that both A and B are fairly predictive (i.e., both look a lot like the plot on the right), how can he measure which of those two variables will be objectively better at making a prediction about C?

Assuming that I've understood the question correctly, I would recommend using mutual information as a formal criterion for measuring how well one variable predicts another.

Please note that mutual information does not, by itself, actually make any prediction; i.e., given some fresh new observation of A, it will not actually estimate for me what value of C I should expect. (That sort of question, which would be a natural follow-on to what I believe the original poster was asking, may be addressed with other statistical techniques such as statistical regression.) But, given a large sequence of values A and C, mutual information will produce a metric which summarizes, in a generic, model-independent way, whether the relationship between the two quantities looks more like the plot on the right, or the one on the left; i.e., whether A is predictive of C, or completely random, or somewhere in between.

ADDENDUM:

In the time since I posted my initial response, the phrasing of the original question has been substantially revised. The questioner now makes clear that he is thinking about comparing the relative predictive value A and B by developing two different models, to relate each variable separately against C. Then he proposes computing a Bayes factor in order to determine which of these model is better. This idea was not exactly clear to me in the original wording of the question.

The potential problem that I can see with this approach is that the answer to the main question (which variable is intrinsically more predictive, A or B?) will then depend heavily on the quality and appropriateness of the models that the poster has selected. If variable A is intrinsically more predictive than B, but a poor model is chosen for relating A to C, while a good model is chosen for relating B to C, then one can potentially get the wrong result: i.e., that B is a better predictor than A.

Bayes factors are really most useful for comparing the performance of two different models, both based upon the same underlying data. So for example, let's say the poster has three alternative hypotheses:

• The number of hamburgers consumed in a year (C) is equal to five times the number of siblings (A)
• The number of hamburgers consumed (C) is equal to ten times the number of siblings (A)
• The number of hamburgers consumed (C) is a quadratic function, equal to $3 A^{2}$

For that kind of problem, where the data underlying the comparison (A vs. C) are always the same, I think that Bayes factors would be an appropriate-not to mention completely Bayesian!-way of selecting the best model.

But that's not the kind of problem the original poster actually faces.

The issue with the Bayes factor approach is that when we try to use that method to compare between different data sets, it's not possible (or at least, I can't think of a way that would be possible) to ascertain whether the real reason that a particular variable (A or B) seems to be a poor predictor of C is merely the result of a poor choice of model, or whether it's actually because the two variables don't share any underlying correlative relationship.

Rather than Bayes factors, I believe that what the original poster would really want to use here is a model-free way of ascertaining predictive power. The mutual information criterion that I mentioned previously does precisely that, and even though it's not necessarily "Bayesian" in derivation, it also isn't clearly identifiable as frequentist either, since mutual information only measures the relatedness of variables, rather than dealing with probabilities or inference, the domain in which Bayesian and frequentist approaches typically diverge from one another.

In summary, I think that mutual information provides a valid technique for measuring the thing that the original poster actually wants to measure, and to me it seems kind of meaningless to try to characterize that approach as either Bayesian or frequentist because those concepts only apply (metaphorically, at least) to almost a kind of orthogonal domain or dimension.