# Examples to expose fundamental problems when using predictive models

The way some people learn statistics feels (in some sense to me) overly method oriented.

The problem is not the teaching material itself. The teaching material usually contains good illustrative examples of when to apply a method and what can go wrong (often overlooked by students). I take this from the feedback that I get when discussing statistics with people that have taken a basic stats course or maybe something more advanced.

People that are going through a basics stats course usually go over quite a number of new methods and techniques that may be very foreign to them. The way they learn/consume this new information is often similar to the way I feel mathematics at a lower level is learnt by many people. The students emphasize a specific type of problem and learn how exactly to apply a method to it. When the problem changes slightly, they often have no idea on how to attack it, although the method they are using can generalize to this change trivially.

This way of thinking that some people tend to have is likely not because the way things are taught in the course, but rather the mindset that the students are in with regards to solving mathematical problems.

Maybe the underlying issue is more fundamental than what I thought to begin with.

This all leads to the misconception that I have observed too often:

My model doesn't work well, so I need to try another model.

What the person should often in fact be doing is taking a step back and think more in depth about their data and what is underlying their problem. E.g. ask themselves the following questions/or do the following:

• How do my residuals look? Do they contain a trend?
• Plot the residuals.
• Plot the variables against each other and the response.
• Do I need all these variables? Are some of them redundant?
• etc.

So my point is that often I feel that students are too focused on learning how to solve specific problems and they do not learn well to generalize, or question what is fundamentally wrong when they do their analysis.

I want to find examples to motivate people further to question what is fundamentally the problem, rather than trying another model/method. I am especially thinking about predictive models of continuous variables, especially linear models.

One example that I feel students should be exposed to when learning about linear regression is Anscombe's quartet. Which clearly shows the importance of plotting your data.

Another typical example is to generate points from a parabola with noise and fit a linear model to it. Then ask the students to observe the trend in the residuals.

What more examples could create common pitfalls that students should fall into and learn to deal with? These should be questions that most students get wrong and they should learn by failure.

• I have many bookcases of stats books dating back a half century. The number that are purely method-oriented is precisely one: Gonick & Smith's Cartoon Guide to Statistics. Even those that focus on methods emphasize understanding and suitable diagnostics. Stat ed sites focus on understanding, not methods. That suggests you and I may have different experiences of "the way statistics, statistical learning and machine learning are taught today." Could you briefly characterize this "way" and provide references to document the correctness of your characterization? – whuber Sep 15 '15 at 14:20
• I will edit my question, it may have been a bit hastily written. – Gumeo Sep 15 '15 at 14:37
• I have completely changed the question, I guess it could be a lot shorter and only contain the last quarter of it. But I wanted to express what I have observed in regard to this problem. – Gumeo Sep 15 '15 at 15:22
• It's a great set of questions, but it is an awfully broad subject. I don't see how it could effectively be addressed here. Is there any way you could narrow it? – whuber Sep 15 '15 at 15:34
• I think I could restrict this completely to multiple linear regression to begin with and then maybe create other questions later to expand on this for other models later. From a pedagogical point of view I feel that there should be a collection of these pitfalls somewhere. This should be to help people that are self-studying and also for teachers to find examples to use in their class. – Gumeo Sep 15 '15 at 15:43