Formal statistics vs naive Statistics

Question

In the last year I started to study Data Science by some Udemy and Coursera courses. As a pure mathematician, my curiosity makes me study statistics more formally and deeper.

Yesterday, I rewatched one of these Udemy courses (btw this course is one of the tops in terms of the ratings and the number of people watching) and I was impressed how sloppy the instructor was in an univariate linear regression problem. He basically just verified if the data is more or less linear and applied a cross-validation analysis with $R^2=0.8$, He didn't check more carefully the linear assumption with the residuals graph, the independence and constant variation of the residuous neither, etc.

I suppose many of the 6-months-DS-without-stat/math-background make predictions in this way. (In some kaggle challenges most of people just apply a bunch of algorithms blindly (maybe some algorithms may need opposite assumptions) and choose the one that gives them the best prediction(type I/II error or a mixed of the two).

So my question is if I'm doing a linear regression project during the night and I'm happy with $R^2=0.8$ can I just close the lid of my laptop and sleep deeply?

Answer 1

He basically just verified if the data is more or less linear

Granted I know nothing about the course in question and its purpose, in real life assumptions are rarely if ever fully satisfied so realizing that the data is more or less linear may be enough to proceed provided you are aware of the limitations.

I suppose many of the 6-months-DS-without-stat/math-background make predictions in this way.

I'd say the opposite. Often it's inexperienced practitioners that get overly concerned about assumptions not being fully satisfied. In the end, statistics is an art and a science (a reference to this The Art of Statistics) and you need to strike a balance between rigor and practical utility.

if I'm doing a linear regression project during the night and I'm happy with R2=0.8 can I just close the lid of my laptop and sleep deeply?

That's one of the perils of data analysis: you torture the data until you get what you want regardless of whether it is sensible results.

---

I'll try to give an illustration where rigor and practical utility clash. Here, x and y are unbelievably well correlated. However, a formal test rejects the assumption that the residuals from a linear model are normally distributed. The rejection is correct since I simulated noise from a t-distribution. However, unless you want to estimate the slope with extreme precision, you can safely ignore the non-normality.

set.seed(1234)
x <- 1:200
y <- x + rt(n= 200, df= 5)

plot(x, y, pch= 19, col= 'grey30', cex= 0.2)

fit <- lm(y ~ x)
shapiro.test(fit$residuals)

    Shapiro-Wilk normality test

data:  fit$residuals
W = 0.98533, p-value = 1.795e-08 <<<<<

Answer 2

Keep in mind that the data you got in any context is just one possible realization of infinitely many. Hence, it is not really desirable to hit R^2=1 for a given dataset (keyword overfitting). Rather, you want to estimate parameters that provide for different samples on average reasonable results. From my point of view the instructor should have validated his results by using a different sample and show that his R^2 remains relatively stable. Further, with regard to the linear assumption, hardly any data/context/relationship is exactly linear. Therefore, an R^2=1 would be really unrealistic and further, an R^2=1 would make me really sceptical if there is not a problem with multicolinearity etc. Take it like that: There is a monotonic relationship between independent and dependent variables (more or less linear) and you try to approximate that monotonic relationship with a linear model. In this case a R^2=0.8 is quite good. Of course you don't have grasped the relationship perfectly, but, understanding parts of it is better than understanding nothing. cheers