# Correlation between event occurrence and data

I'm trying to figure out what's causing my stomach pain recently and I have logged the following data twice a day:

• A grade on a scale from 1 to 10 on how much my stomach hurts
• Foods (unique names) I've been eating (e.g., banana, yogurt, tomatoes) twice a day also

So I have my data like the following (almost 3 months of data, so ~120 lines):

14/06 AM    5    banana, tomatoe
14/06 PM    3    milk, beans, banana
15/06 AM    2    apple, meat, tomatoe
15/06 PM    3    chicken, banana, coffee
16/06 AM    6    milk, beans, chicken
16/06 PM    7    tomatoe, orange, coffee
...

How can I find the correlation (or combined occurrences) between those two set of data? Like the more I eat tomatoes with bananas, the higher my stomach hurts?

I know how to figure out the correlation between two sets of discrete values, but here I have discrete values (pain) and occurrences (food). Is there a formula for this? Can I use R?

Complementary question: I have a third column where I input how much water I drank (5 oz, 3 oz)... How can I include it in my stats ?

• If the truth is anything beyond a single food causing problems & the rest not, you are going to need a lot of data. Commented Jun 18, 2016 at 0:02
• Please register &/or merge your accounts (you can find information on how to do this in the My Account section of our help center), then you will be able to edit & comment on your own question. Commented Jun 18, 2016 at 0:29

In order to analyse this you'd use an ordinal probit regression with the dependent variable being your pain rating, and the independent variable(s) being the food and the quantity of water.

Such a model would assume that your ratings of pain map onto an underlying continuous latent variable (the amount of pain you're experiencing), and would model how that latent variable varies as a function of the food you're eating and the volume of water you consume.

As gung has pointed out though, if you're interested in looking at the combination of foods then you'll need vast numbers of observations (you'd want to have a reasonable number, maybe >5, of each combination).

Really, though, if you're experiencing pain which you believe is related to your diet then either altering your diet (on the basis of patterns you observe) or seeking professional help is more important than determining the appropriate statistical approach to take

Basically you have a regression problem where a number of binary coded fruit (variable is either 0 or 1) predicts a pain score. If you want the door to all sorts of combination effects, a machine learning technique is probably better suited then a probit model. Consider a random forest or a neural network or ... They will respect all the combinations that you feed them (no pun intended) without forcing you to try all possible combinations. They will try to draw all the information from your 120 observations and find a model that fits these observations.

Be warned though, that 120 orbservations is not that much (depending on the number of possible foods) and it may well be, that many different models might fit your data and that the model found by an algorithm does not reflect your biological reality well.

That being said, I'd try a regression tree as well as a random forest. And yes, you can use R to compute both. There will be a lot of explanations and examples on the web. I'd recommend starting with Youtube searching for "StatQuest" and the topic at hand, but that is a weakly founded personal preference.

If given the sample size, interactions are not that important, you might also want to look into Naive Bayes, which is also explained in a StatQuest video : https://www.youtube.com/watch?v=O2L2Uv9pdDA

• Those methods require 10x larger sample sizes than regression. Commented Nov 2, 2023 at 13:12
• @FrankHarrell Obviously you have far more knowledge and experience so I will not doubt you. But out of curiosity: If there was a clear signal and no noise: one combination leads to reaction, no other does. I could see how many of the trees in a random forest could find that. Can you advise a regression? LASSO with all possible interactions? If there is not such a clear signal to noise ratio, the whole thing is futile, right? Commented Nov 2, 2023 at 21:12
• Machine learning methods such as random forests, thought they are likely to be severely miscalibrated in their predicted values, can work when the signal:noise ratio is very high and there are smoking guns in the data. I’ll bet that regression works pretty well there too but regression needs either (1) potential interactions to be pre-specified or (2) include lots of potential interactions and penalize them in a reasonable way such as using a Bayesian horseshoe prior (more severe penalty function than lasso). Commented Nov 3, 2023 at 11:39
• @FrankHarrell That is going to be an awful lot of potential interactions in this question. Thank you for answering. Commented Nov 3, 2023 at 14:09
• Yes, but penalization helps. Keep in mind that machine learning effectively looks at tons of interactions, with some penalization, but probably not enough. Commented Nov 3, 2023 at 17:38