How to deal with an influential case I'm creating a linear model for a data set with a fairly small amount of observations (roughly 40). I've found that one observation has a significantly larger value than the others for the response variable in my model. The data point is not an mistake, it is a correct value but it just happens to be an extreme value. When I include the value, it heavily influences the inference, in a way which supports the linear relationship. However, when I take it away the linear relationship in my model is much weaker, to the point where most of the inference from the dataset including the extreme point is invalid.
Is there a good way of dealing with this? I don't want to fully exclude it as it's still a valid observation, but this single observation seems to have a very substantial effect on the model.
 A: You can look into methods that deal well with outliers. Two of these are quantile regression and robust regression. (Robust regression is actually a bunch of methods). 
A: It may also mean that your error model is wrong - maybe the assumption of a normally distributed error is not warranted? (hard to guess without knowing more about the data)
You can also try to fit a hierarchical/mixed effects model with a per-subject random effect (intercept) this would look something like:
$Y_i \sim N(\mu_i,\sigma)$
$\mu_i = \beta X + \alpha_i$
$\alpha_i \sim N(\gamma, \tau)$
plus some priors on all parameters to have good regularization. Here $\alpha_i$ is the subject-specific intercept. In this context, the single outlier will more influence the corresponding $\alpha_i$ and the between-subject variability $\tau$ and will have smaller effect on  both the average intercept $\gamma$ and the linear coefficients $\beta$. Note that $\gamma$ and $\tau$ bind all the $\alpha_i$ together and thus the model will not overfit even though we have more parameters than data points.
Such models are best fit with INLA or rstanarm (both provide fully Bayesian treatment and will give you correct uncertainty estimates for all parameters). I would expect hierarchical model with those packages to give similar results regardless of inclusion of the outlier. 
If you want to stay within the frequentist framework, you may try the package lme4, but I've heard its estimates can be unstable with few observations (I have no personal experience with lme4, but have good experience with both INLA and rstanarm).
