# Bayesian approach for comparing the predictability of different datasets for another

Suppose I have three datasets A, B and C with not necessarily the same amount of data. Now, I want to know whether dataset A or dataset B is better in predicting C. I want to use a Bayesian approach here.

To give more details about what I mean with datasets. Let us assume the following three datasets (variables).

A   B   C
----------
2  0.6  10
1  0.7  5
10 0.2  50
3  0.3  1


Let us assume that C refers to the amount of burgers kids in a school eat over the period of a year. Now, we want to know whether (A) the number of siblings of the kids or (B) the social class of the family rated from 0 (poor) to 1 (rich) can better predict C.

I have some ideas which I quickly want to elaborate next. However, the main goal of this post is to get ideas of how I would tackle the task of determining whether A or B are better in predicting C within the Bayesian framework.

My current idea is to build two models (with different priors) and calculate the Bayes factor with data C to check their relative plausibility. The priors should resemble data A and B. Thus, the priors could be posteriors from A and B with some kind-of flat proto prior. In simple terms, we observe data A and B, determine posteriors for both of them, and use them as priors for C. Now, we want to compare them by using Bayes factors.

Another, more simple idea of mine is to just calculate correlation between A or B and C in a Bayesian way. Or, we could also focus on regression or similar methods.

• It is difficult to tell what you are asking, because you appear to use "dataset" synonymously with "variable," which is unusual and confusing. Could you provide a more explicit description of what kinds of data you actually have? The questions at the end about comparing predictive accuracy and testing goodness of fit are too broad to be addressed (even in separate questions) because the setting is too general and abstract. – whuber Aug 24 '15 at 15:33
• I have edited my original post to be more precise for a single example. Also, I have removed my final pointers and have expressed my current idea of using Bayes factors. – fsociety Aug 24 '15 at 16:03
• Unfortunately, your question still doesn't make much sense, even after a rewrite, because a Bayesian prior is a type of probability distribution. Your variables A and B, on the other hand, are not probability distributions, so therefore you can't just "have either A or B as a prior". You are speaking nonsense. – stachyra Aug 24 '15 at 16:58
• @stachyra It is important to follow the site guidelines for being nice, for otherwise you put yourself at risk of having someone point out just as publicly that maybe this post does make sense, which you might find embarrassing. For instance, I wouldn't be surprised to see someone point out that the idea of replacing a dataset by its empirical distribution is fundamental to modern techniques such as the bootstrap. Thus, when you don't understand a question, just ask (nicely) for a clarification, lest your comments look rude and ignorant. – whuber Aug 24 '15 at 17:50
• Barsch, nevertheless I still don't follow what you are doing! I can't understand what these vectors of numbers mean. How exactly do they represent probability distributions? – whuber Aug 24 '15 at 17:51

I found the wording of this question a bit confusing, and moreover, I'm not quite sure why a Bayesian technique was suggested as a possible solution.

Here's the question that I think was being asked:

I think the original poster has a table that looks something like this:

 A      B      C
-----------------
0.08  0.17  -0.72
0.13  0.29  -0.45
0.26   --    0.38
0.41  0.56   0.87
--   0.77   1.42
0.63  0.91   1.69


In other words, variable C is always present, but in some rows, variable A is sometimes missing (e.g., because it wasn't measured, or whatever), and in other rows, B is sometimes missing.

The poster posits that there might be various different types of relationships possible between A and C, or between B and C, which can be illustrated, at least at a conceptual level, with a couple of figures:

If the relationship between A and C (or alternatively, between B and C), looks more like the figure on the left, then we say that the two are distributed randomly with respect to one another, and having knowledge about variable A offers no predictive power whatsoever about the value of C.

On the other hand, if the relationship between A and C looks more similar to the figure on the right (namely, that the two variables sort of track with one another, albeit not necessarily in a linear way) then having knowledge about variable A obviously does offer some predictive power over C, even if the prediction may not be perfect.

I think that the original poster is probably asking:

• For either variable A or B, compared against C, how can he distinguish whether the relationships hidden in his data look more like the plot on the left or the one on the right?
• If it turns out that both A and B are fairly predictive (i.e., both look a lot like the plot on the right), how can he measure which of those two variables will be objectively better at making a prediction about C?

Assuming that I've understood the question correctly, I would recommend using mutual information as a formal criterion for measuring how well one variable predicts another.

Please note that mutual information does not, by itself, actually make any prediction; i.e., given some fresh new observation of A, it will not actually estimate for me what value of C I should expect. (That sort of question, which would be a natural follow-on to what I believe the original poster was asking, may be addressed with other statistical techniques such as statistical regression.) But, given a large sequence of values A and C, mutual information will produce a metric which summarizes, in a generic, model-independent way, whether the relationship between the two quantities looks more like the plot on the right, or the one on the left; i.e., whether A is predictive of C, or completely random, or somewhere in between.

In the time since I posted my initial response, the phrasing of the original question has been substantially revised. The questioner now makes clear that he is thinking about comparing the relative predictive value A and B by developing two different models, to relate each variable separately against C. Then he proposes computing a Bayes factor in order to determine which of these model is better. This idea was not exactly clear to me in the original wording of the question.

The potential problem that I can see with this approach is that the answer to the main question (which variable is intrinsically more predictive, A or B?) will then depend heavily on the quality and appropriateness of the models that the poster has selected. If variable A is intrinsically more predictive than B, but a poor model is chosen for relating A to C, while a good model is chosen for relating B to C, then one can potentially get the wrong result: i.e., that B is a better predictor than A.

Bayes factors are really most useful for comparing the performance of two different models, both based upon the same underlying data. So for example, let's say the poster has three alternative hypotheses:

• The number of hamburgers consumed in a year (C) is equal to five times the number of siblings (A)
• The number of hamburgers consumed (C) is equal to ten times the number of siblings (A)
• The number of hamburgers consumed (C) is a quadratic function, equal to $3 A^{2}$

For that kind of problem, where the data underlying the comparison (A vs. C) are always the same, I think that Bayes factors would be an appropriate-not to mention completely Bayesian!-way of selecting the best model.

But that's not the kind of problem the original poster actually faces.

The issue with the Bayes factor approach is that when we try to use that method to compare between different data sets, it's not possible (or at least, I can't think of a way that would be possible) to ascertain whether the real reason that a particular variable (A or B) seems to be a poor predictor of C is merely the result of a poor choice of model, or whether it's actually because the two variables don't share any underlying correlative relationship.

Rather than Bayes factors, I believe that what the original poster would really want to use here is a model-free way of ascertaining predictive power. The mutual information criterion that I mentioned previously does precisely that, and even though it's not necessarily "Bayesian" in derivation, it also isn't clearly identifiable as frequentist either, since mutual information only measures the relatedness of variables, rather than dealing with probabilities or inference, the domain in which Bayesian and frequentist approaches typically diverge from one another.

In summary, I think that mutual information provides a valid technique for measuring the thing that the original poster actually wants to measure, and to me it seems kind of meaningless to try to characterize that approach as either Bayesian or frequentist because those concepts only apply (metaphorically, at least) to almost a kind of orthogonal domain or dimension.

• I agree that my question was not question was a bit confusing. You got it right though. However, I was specifically asking for a Bayesian approach which is not part of your answer. – fsociety Aug 24 '15 at 9:11
• I don't think your question (i.e., which is more predictive, A or B?) is one that could even have a Bayesian approach; it falls outside the domain which Bayesian thinking is intended to address. I think you're trying to treat Bayes as some kind of golden hammer, universally applicable to every possible question that could ever be asked. But it's not. – stachyra Aug 24 '15 at 13:14
• Given that the original question had to be almost completely rewritten due to the that fact that people couldn't understand what was being asked, I'm a bit dismayed to see that my genuine attempt at being helpful was downvoted. Lesson learned: next time a question contains flawed assumptions, is ambiguous, or doesn't really make much sense to begin with, I'll just keep my mouth shut and downvote the original question so that others are not tempted to waste their time on it! – stachyra Aug 24 '15 at 17:12
• That's a valid option. A more constructive option--especially when you do have something useful to share and you think it might be appropriate--is to write comments to the original question asking for clarification about specific points. Once those points have been resolved, you can then have confidence that the effort you put into writing an answer will be appreciated by the OP as well as the many other readers who come by later. – whuber Aug 24 '15 at 18:11