Model has a Great Fit, Significant Variables but Residuals are Not Normally Distributed. How should we proceed?

Question

I have a data on some overall conversion rates (i.e. out of x users visiting, y buy something hence y/x is my conversion rate, essentially proportions) over a time period, now this overall proportion can be broken by if they came from channel 1, channel 2 or channel 3 and for each channel there would be again similar proportions. My objective is to see how these proportions from different channels impact the overall proportion

I have run a simple linear regression in R and below is the result.

Call:
lm(formula = target_variable ~ . - date, data = data_lcr)

Residuals:
  Min        1Q    Median        3Q       Max 
-0.034173 -0.003217 -0.000704  0.002331  0.073845 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)    
(Intercept) -0.0049876  0.0006139  -8.124  7.4e-15 ***
exp1         0.0785438  0.0086230   9.109  < 2e-16 ***
exp2         0.0290531  0.0175517   1.655   0.0987 .  
exp3        -0.1026385  0.0080550 -12.742  < 2e-16 ***
exp4         1.0760312  0.0669632  16.069  < 2e-16 ***
exp5         0.2466149  0.0195844  12.592  < 2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.007503 on 358 degrees of freedom
Multiple R-squared:  0.9843,    Adjusted R-squared:  0.9841 
F-statistic:  4503 on 5 and 358 DF,  p-value: < 2.2e-16

The Model has great R-squared which is significant, all variables turn out to be significant. Next I am checking if my residuals are normally distributed

>  skewness(fitlm$residuals)
[1] 2.863341
> kurtosis(fitlm$residuals)
[1] 33.83711

Shapiro-Wilk normality test

data:  fitlm$residuals
W = 0.72781, p-value < 2.2e-16

Anderson-Darling normality test

data:  fitlm$residuals
A = 17.485, p-value < 2.2e-16

These tests suggest that my residuals are not normally distributed. Should I still consider the model based on R-squared and F-Value or make some corrections? Please suggest

Here is the residual plot:

EDIT After removing outliers:

Answer 1

It is always good to look at plotted data. In case of regression, it is good to look at residuals plots. In your case residuals seem to come from distribution with longer tails than normal. Distribution of your data is closer to $t$-distribution and actually I was able to produce a similar data example for $t_2$ distribution.

It even produces quite close Shapiro-Wilk estimates:

> shapiro.test(x)

    Shapiro-Wilk normality test

data:  x
W = 0.78051, p-value < 0.00000000000000022

But this is of smaller importance. What the residual plot really shows you is that your distribution has longer tails, i.e. it has some outliers. Now you should ask yourself what are the outlying values? Identify and check the values. Why are they outlying? Is it the measured phenomenon having long tail distribution that produced them, or maybe there was some issues with measurement (are they erroneous)? The outlying values can influence your final estimate, so you have to make a number of decisions on what do do with the outlying values. Check also How should outliers be dealt with in linear regression analysis? and Interpreting the residuals vs. fitted values plot for verifying the assumptions of a linear model threads.

Answer 2

When data are not normally distributed the inferential results on p-values (and on F-statistics) do not hold and so it is not correct to look at them.

However the leasts squares fit does not rely on the normality assumption and so if the fit is good and R squared is high there is no reason to discard the model.