Can predictive models make predictions for individuals or only for groups? [closed]

Question

I am an MBA Student taking courses in statistics.

In this last week, I have heard two conflicting opinions from my professors on Statistics.

• 1) Predictive Models can NOT make predictions on individuals - only on groups: My professor from my "Research Methods" class introduced us to the concept of the "Ecological Fallacy" (https://en.wikipedia.org/wiki/Ecological_fallacy). He pointed out that very often, the behavior of individuals within a group will not be similar to the average behavior of the group. He mentioned that statistical models can ONLY be used to describe variations within groups - he mentioned the importance of concepts such as "Propensity Score Matching" and correctly designing experiments while accounting for possible inconsistencies.

• He mentioned that if a pharmaceutical drug was found to be effective on a cohort of men all having similar age groups, lifestyles, backgrounds and medical histories - the best we can do is say that any man from the general public who fits this cohort profile will likely experience similar effects from this drug : we can not really make any individual prediction apart from this.

• He mentioned the field of "Survival Analysis" and told us that statistical models are routinely used to estimate the survival odds and hazard of surviving at the group level and not the individual level - he gave us an example: if you have 1 Male Asian Patient and 99 Male Non-Asian patient, you have no choice but to analyze the average survival rates of MALES ... how can you perform "Asian Specific Analysis" and make inferences about the average Asian when you only have 1 observation!?

• He closed by mentioning the new and emerging field of "Precision Medicine" in which "accurate molecular taxonomy of diseases that enhances diagnosis and treatment and tailors disease management to the individual characteristics of each patient " (https://www.mdpi.com/2227-9717/10/6/1200/pdf) - i.e. for the first time, medical treatments are being considered for the individual patient's condition, and not for the average profile of this patient (although he mentioned that this is still in its infancy and should be used with great caution).

• With this, he largely dismissed "Data Science" as a "pseudo science" and said although many "Data Science" models have demonstrated success, the methodology is not mathematically rigorous and can only be considered as an "engineered solution" (e.g. a Random Forest predicting if an individual patient will develop a disease - he said that this should not be done for many reasons, e.g. interpretability, blackbox, lack of odd's ratio and individual prediction).

• He closed with this final example : Imagine two bridges. The first bridge can only support a load of 100 kilograms, but has been rigorously tested in theory (e.g. physics, material science) and empirically - the reputation and behavior of this bridge has been well documented and observed under various conditions. The second bridge has been demonstrated to support loads of up to 500 kilograms, but we have no idea how this bridge was built or whether it will collapse in the future when subject to even 50 kilograms. He asked us - which bridge are you more confident walking across? He said this in a rhetorical tone, and said obviously the first bridge that has demonstrated both theoretical and empirical success should be favored over the second bridge - even if the first bridge is believed to be less stronger than the second bridge. He quipped that the first bridge is Classical Statistics and the second bridge is Data Science. He told us that perhaps this example is an extreme example, but this is precisely why a drug that has initially shown strong potential for curing a disease must be rigorously studied in both theoretical and empirical settings before it can be released - and in the interim, potentially less effective but better understood drugs must be administered of which we have higher confidence in.

• 2) Predictive Models CAN be used to make predictions on individuals: On the other hand, my Data Science professor thinks otherwise. I told him the views of my "Research Methods" professor and he stated that research publications would suggest otherwise. He showed me publications (e.g. https://www.sciencedirect.com/science/article/pii/S2666827021000694, https://pubs.rsna.org/doi/full/10.1148/radiol.2018180547) in which predictive models have indeed demonstrated success in making individualized predictions - for example, successfully predicting the presence of COVID-19 in individual patients. He basically said that he sees no reason as to why Predictive Models can not be used for individual prediction - and if the opposite was true, he would have been long out of a job.

This has left me conflicted - I find myself agreeing with the opinions and views of both professors at the same time.

Can someone here provide some insights to bridge these two views together?

Answer 1

There is legitimacy to each stance.

Your data science professor is right because it is routine to use a model to make a prediction about an individual. For instance, I am writing this answer on an iPhone that keeps trying to guess which word I will type next.

Your research methods professor is right because there is an entire conditional distribution to consider. While we might call $\hat y_i$ the predicted value, that is just the expected value (or median or some other quantity, depending on what exactly we do) of an entire distribution of plausible values for the individual. This is a standard argument in pop-culture statistics: the statistic means nothing to the individual, and it is true that an individual can wind up above or below average, not right smack on the average. Cross Validated has an interesting question on this very topic that I believe has some relevance to this question.

Then again, this leads to a defense of your data science professor. Yes, $\hat y_i$ is just the expected value of an entire conditional distribution, but if we believe our loss function to quantify what we value, then that is exactly the prediction we should make to minimize the pain we experience from having results that are not perfectly accurate.

Answer 2

Of course you can use predictive models on individuals. They wouldn't be very useful if you couldn't.

Your statistics professor's position is so bizarre that I wonder if maybe something got lost in communication. For example, if we were to take your reporting of his comments at face value, we would have to conclude that he had never heard of prediction intervals, which seems unlikely for a professor of statistics.

Formally, a statistical model model predicts a probability distribution for the random variable $Y$ that it is modeling. For example, an ordinary least squares model predicts: $$Y_i \sim \text{Normal}(\alpha + \beta x_i, \sigma),$$ where $\sim$ means "is distributed as". The $\alpha$, $\beta$, and $\sigma$ are all parameters of the model; finding them is what we mean by "fitting" the model. So, it's baked into the definition of the model that we can't predict an individual case with certainty.

When people talk about the model "predicting" $\hat{y}_i$ for a case, they usually mean that $\hat{y}_i$ is the expected value of the predicted distribution. Depending on the circumstances that may or may not be a useful summary of the model's predictions, but it's not the whole story. For example, if you care about individual variation, you might compute a 95% prediction interval as $\hat{y}_i \pm 2\sigma$, which gives an idea of the range of actual events you might commonly encounter. Note, by the way, that this prediction interval will generally be much larger than a "confidence interval" for $\hat{y}_i$. The confidence interval tells you about the uncertainty in the estimate of the expectation value, which is something that is only relevant to populations.

Now, machine learning practitioners often elide all of this and just talk about "the prediction" of the model. Depending on what those are trying to do, this might be a reasonable approach, or it might not. Sometimes you need a system that is going to take its best guess and move on; it all depends on what you are trying to accomplish. I happen to think that ML models should make probabilistic predictions more often than they do, so perhaps some criticism of data science is due there, but a blanket dismissal dramatically overstates the case.

Most of your professor's other criticisms seem like red herrings to me. Obviously a model fit to data from a very limited population is only useful within that population. This comes as a surprise to nobody. And I can't even figure out what he's trying to say with his bridge analogy.

One thing I think we can all agree on is this: it's good to give some thoughts to what your model's predictions actually mean and what their limitations are. It's fair to be skeptical of boiling a prediction down to a single number, as long as you recognize that sometimes that is the right solution for certain problems. The best way to use a statistical model is to think about what question you are trying to model, and consider all of the information returned by the model and how it might be used to answer that question. That last part demands judgement, and developing and exercising good judgement is where data scientists and statisticians alike earn their keep.

Answer 3

You've done an excellent job summarizing the key points, and you are to be lauded for readily catching the apparent discrepancy. It's simpler than that: The purpose of statistical models is to summarize and infer stochastic trends. A stochastic trend is one that's true on average. Men are on average stronger than women. Yes, there exists a woman who is stronger than a particular man. However, I can rigorously defend the notion that a randomly sampled man should be stronger than a woman. I use data to infer those choices, and if I have more data - more information specifically - I can refine my choices in scope and precision. "Big data" data science, therefore, is just a special case of statistics, I don't think it deserves special treatment.

When presenting analyses to the FDA for a drug's approval, they are gravely concerned about generalizability. It's expected that the cohort that enrolls to a study is not representative of the general population, and it's known that undetected interactions and reduced compliance will negatively affect the estimated efficacy of treatment. To that end, many analyses are requested, to the tune of hundreds or even thousands of pages of tables, figures, etc. analyzing data at all levels of collection - even drug concentration in the blood, and clearance in liver and kidneys - so that there's a high degree of confidence that even if the trial results are not achieved per se, a favorable result is expected by adopting the drug in the population. Regarding the analysis of the one patient, if you make assumptions, you can in fact conduct tests or estimations of effect using specially tailored analyses.

Answer 4

Wow, professor 1 seems to suffer from a massive reverse Dunning-Kruger effect. In essence, I know a lot about one thing therefore I am an expert in all things. A little humility on his part is required.

The problem of course is that:

1. He doesn't understand the model building process. Sure, he has examined it. He's been tested on it. He knows a lot about it, but he doesn't understand it. The point he seems to be making having read through your post is that your model, when trained on data that came from a group, will essentially predict something about the group instead of the individual. This is, of course correct. However, he seems to equivocate this with predicting the mean of the group given some covariates. On average, this is correct and what we do when modelling. However, we can predict 95th percentiles, medians, variance, or any other meaningful statistic for the group. We then apply it to the individual. He seems very smug about the fact that we can't know anything about the individual. And the points he makes are valid. Of course, you can train a model on data that comes from a single individual and make inferences based off of that. Economists do it all the time. There is only 1 U.S. economy after all, and you can design statistical methods to make inferences on that one individual. Usually, you are stuck with time-series type models in those cases, but it can be done. However, even if you don't what about conjoint analysis where you give a subject a battery of choices in a survey. With enough choice pairs, you can generate a rigorous enough sample to make inferences on an individuals behavior. Again your professor doesn't seem very familiar with these actual scientific methods that can work on an individual basis. The question isn't can you predict on an individual, the validity of his argument is based on what is the unit of observation, and he just doesn't seem to have much experience with a wide-breadth of different observational units other than individuals. I won't fault him for this lack of experience but I will fault him for spreading disinformation onto unsuspecting students based on his ignorance.

2. Your professor sneers because the data science isn't "rigorous". What he really means by that isn't that there is no mathematical rigor or that there isn't even empirical rigor, or even statistical rigor. What he means is that the models aren't causal. That seems to be his biggest beef. I'll return to statistical rigor in my next point, but for now let's focus on causality. Causality is hard. And a research method class is all about trying to look for causal relationships. You want to design your experiments so that you can say that A causes B to happen. You don't want to have A and B caused simultaneously by something else, or for B to cause A. You are testing A causes B. That is the point of this class. Never lose sight of that one fact. I think this professor has lost the forest for the trees. He isn't looking at it from that perspective any longer. He's looking at research methods as its own thing that must be protected from encroachment. That's kind of sad, because it means that he has actually given up on scientific curiosity. The root of all scientific curiosity is this question: "What if?". He has closed his mind to the possibilities that these new methods could be useful. What if I could use a random forest in a causal way to do "rigorous" scientific research? What would that take? And thus he has closed entire lines of research into new methods for himself. That's sad.

3. Statistical rigor. These data science models do not fit neatly into his paradigm of statistics. Therefore, they do not exhibit any statistical rigor at all. However, a model like random forest does fit into this paradigm pretty easily, if instead of using mean squared error, you used a normally distributed, maximum likelihood loss (which is equivalent to mean squared error if you hang some additional assumptions on your model) then it becomes statistically meaningful. Same with binary cross-entropy loss, add a couple of assumptions and it all works out. The problem is the parameters of your model become much less meaningful. And that gets back to the causality, what the hell does your model even mean? He seems to want to get to a point where he can say something. Again that is a noble pursuit, but is a correlation really all that bad? Sure, you can make mistakes with correlation, but if you've got an underlying theory, you might be willing to take those risks in your prediction. Here's an example: I want to predict whether or not someone will buy my product. If I know their income, can I spit out a probability that they will make that purchase? Of course, with enough data I can. Why? Because income/wealth effects are real according to theory. So, do I need to spend all that time and energy in proving that to be the case to use it? Of course not. Just utilize the correlation, it might be slightly off but the broad strokes are likely just fine.

4. He also seems to equate uncertainty with bad. Surely, this one is a fine point. Except, every time I get in my car there is uncertainty around whether or not something mechanical will break lose and cause me to crash and die. The truth is, you deal with uncertainty with tolerances. This goes back to his bridge analogy. What is your tolerance for failure? If the first bridge spans a 1000 foot drop over spikey rocks, I am with him, I want a bridge that has been rigorously tested. If the bridge is a 2 foot fall over a small stream with rounded pebbles at the bottom of it. I will be annoyed if the bridge falls apart when I cross it, but it didn't cause too much distress. Tolerances are important and drug discovery is a high stakes, low tolerance environment, whereas, say recommending a skirt to a potential customer is a low stakes, high tolerance environment.

Answer 5

There are already many great answers, but I would like to provide my two cents.

He mentioned that if a pharmaceutical drug was found to be effective on a cohort of men all having similar age groups, lifestyles, backgrounds and medical histories - the best we can do is say that any man from the general public who fits this cohort profile will likely experience similar effects from this drug : we can not really make any individual prediction apart from this.

How is this not a prediction for an individual? It is for an individual with that age, lifestyle, background, and medical history. The prediction for the individual is made on the basis of their belonging to the group.

If your professor's requirement is that the prediction be bespoke for the individual, I think that is a fairly high standard to have. This would require encoding all information on the individual in a vector to be used in regression. That's a tall order (and a wide design matrix).

He mentioned the field of "Survival Analysis" and told us that statistical models are routinely used to estimate the survival odds and hazard of surviving at the group level and not the individual level - he gave us an example: if you have 1 Male Asian Patient and 99 Male Non-Asian patient, you have no choice but to analyze the average survival rates of MALES ... how can you perform "Asian Specific Analysis" and make inferences about the average Asian when you only have 1 observation!?

I don't think a reasonable person would argue that this could be done.

He closed by mentioning the new and emerging field of "Precision Medicine" in which "accurate molecular taxonomy of diseases that enhances diagnosis and treatment and tailors disease management to the individual characteristics of each patient " (https://www.mdpi.com/2227-9717/10/6/1200/pdf) - i.e. for the first time, medical treatments are being considered for the individual patient's condition, and not for the average profile of this patient (although he mentioned that this is still in its infancy and should be used with great caution).

The extent to which this is being done is often greatly exaggerated. Morse and Kim 1 identify personalized medicine (sometimes used interchangeably with "precision" medicine" as "[an emerging field] with a goal of applying genomic information as a predictor of disease risk as well as individualization of drug therapy". No one person is completely and uniquely described by their genetics, lifestyle, and demographic factors. These are only means of further conditioning the outcome on additional variables which may be relevant to the outcome. It is merely an extension of the approaches your professor seems to be criticizing.

With this, he largely dismissed "Data Science" as a "pseudo science" and said although many "Data Science" models have demonstrated success, the methodology is not mathematically rigorous and can only be considered as an "engineered solution" (e.g. a Random Forest predicting if an individual patient will develop a disease - he said that this should not be done for many reasons, e.g. interpretability, blackbox, lack of odd's ratio and individual prediction).

This seems mostly handwavy to me. What does lack of odds ratios have anything to do with individualized prediction. Furthermore, to criticize machine learning and data science on their inability to provide an individualized prediction is simply begging the question -- we're trying to determine if individualized prediction is even possible.

---

What I anticipate has happened is that your professor has interpreted the word "personalized" in its literal sense. That this prediction $y_i$ is bespoke for me, like a well tailored suit. In that sense, it seems personalized prediction is not possible, because as I have said no one is completely and uniquely described by their finite set of covariates.

However, that does not preclude us from using those predictions to some success. I do not know if smoking will give me lung cancer or not, but based on data obtained from people who are unlike me in several ways I choose not to smoke. Additionally, as the set of covariates we use to condition on increases, we shrink the group of people for which the prediction applies. Granted, this is not personalization in the way your professor might want, but shit it's the closest we've gotten now isn't it? If you condition the prediction on my behaviour which is so finely measured and so meticulously recored, that is fairly personal. Data science and machine learning -- which I think your professor unduly criticizes -- are really great examples of this. Just open your friends youtube homepage to see just how well predictions can be personalized.

Answer 6

I don't think they would be as accurate for individuals. We can look for similarities between an individual and the group but groups will be effected differently and get different results than just one individual because when you have just one individual there is no control group.

Answer 7

Yes, they can make predictions for individuals, but there is a higher risk of significant errors. For instance we can predict that walking home was safer than driving after consuming alcohol; that didn't work too well for this poor man. But the prediction does work well better than 99% of the time...

He quipped that the first bridge is Classical Statistics and the second bridge is Data Science.

Warning: I'm always suspicious of argument by analogy.

He told us that perhaps this example is an extreme example, but this is precisely why a drug that has initially shown strong potential for curing a disease must be rigorously studied in both theoretical and empirical settings before it can be released - and in the interim, potentially less effective but better understood drugs must be administered of which we have higher confidence in.

During a previous life I developed medical devices and pharmaceuticals. Imagine that you are dealing with hazards that have high severity. One problem is getting enough test coverage, but what patients do you use for clinical trials? How do you know that you have covered all the unknown unknowns? Of then the best you can do is test with a large number of people, and cover as many possibilites as possible: male/female, young folk/elderly, fat people/slim people, as many ethnic groups as possible. Your statistics professor is right to be cautious.

Answer 8

The problem often lies not with the models themselves, but with how they are applied and with how people make decisions.

Too often I see the same pattern in companies starting with ML/Data Science:

1. They face a hard technical problem.
2. Someone high-ish up the management chain figures out it might be solvable by Data Science because everything is these days... Alternatively, they just catch on the hype and do not even have a problem formulation, just "more Data Science more better".
3. They hire someone, very rarely a full team (or, if you are lucky, they actually outsource the problem).
4. The new hire is able to come up with a proof-of-concept which performs better than all the old models on a small subset of test data.
5. They are eager to deploy it.

They want deliverables and start using "common sense" to rationalize how the model works. The added bias tends to downplay failures and emphasize successes - even if the model itself is terrible, it may be hailed as one of the greatest achievements ever by someone on C suite just because it is "AI". And then the issue is that with many statistical models the engineer can reason about the limits of their applicability fairly well. We have a decently developed framework for examining statistical models by now, and while the similar level of rigor is achievable in theory with ML/AI, it is seldom done so in practice. Also, crucially, the cost analysis is often performed poorly or even omitted entirely. That is how you have your tale of two bridges: one could factor in the cost of it failing spectacularly and so on but take a guess what ends up being done in practice? Execs taking over the "undecisive" engineer who is "too dumb" to take advantage of their competence, and the cost-saving measure gets rolled out.

And if we are talking technical execs, many of them are engineers who will try to bridge the explanatory gap using their experience and intuition, neither of which is necessarily applicable to ML. Very basically, their reasoning would go like

1. This model is smarter than all the old and dull models we have used before.
2. Old models were well-regularized and you could extrapolate their behavior well.
3. Ergo, this new and smart model will not freak out when encountering something it has never seen before.

So there you have it - of course, predictive models can predict individual outcomes and even be fairly efficient at that. The problem is that the resulting models are opaque, hard to reason about, and people making decisions often will be incompetent about statistics and ML. A LOT of work data scientists/statisticians/engineers do ends up being communication and ensuring their models are applied properly - and it is just a lot easier to communicate with "classical" models. When something gets done more efficiently, it is not necessarily good, either - if this something was erroneous to begin with (remember Knight Capital?..).

Answer 9

Broadly no, of course not.

Who doubts that the only useful Question is whether predictive models work?

If you dropped all that detail and looked at the broad Question, would it not be obvious that anything like half-way predicting outcomes for individuals would make you richer than Croesus?

Would it not be equally obvious that the idea of predicting a group outcome would in itself be so powerful, anything to do with individuals would matter only by comparison?