Influential case - to remove or not to remove? I'm doing a multiple regression analysis with 3 predictor variables (RELAT, SSS and FAITH), and the criterion variable is SSE. I have found that SSS is a mediator between RELAT and SSE, and FAITH is a separate predictor which adds to the model:
 
However, I have identified that there is one influential case. The Cook's distance is .35, and the covariance ratio is just under 1 - [3(k + 1)/n],
which, as I understand it, means the case should be removed. After removing it, I ran correlations again, and RELAT no longer correlates with SSE. So I've run the regression without RELAT.
In the original model, R square = .43, RMSE = 9.63. After removing the influential case and excluding RELAT from the regression, R square = .37, RMSE = 8.96. So the original model explains more variance, but the model without the influential case and without RELAT in the regression has lower RMSE, meaning its predictions are more accurate. Is that correct? Which model should I go with here? Any help would be greatly appreciated!
Thanks,
Simon 
 A: The danger wrt to an influential observation is that the outcome is totally unrelated to the observed variables. For example, if you had done a survey of Jamaican high school track athletes in 2002 relating their training methods to their track times, and happened to include Usain Bolt as a subject, it is likely that whatever methods he used would have appeared unduly effective due to Bolt's degree of natural aptitude. In short, where removing one influential case changes the outcome, you need to be very careful, and it is very unlikely if your goal is prediction, that results depending on the influential case will generalise well. For example, adopting Bolt's choice of shoe or breakfast as a schoolboy is unlikely to assist any other athlete significantly. If it is possible, though, it may be instructive to attempt to find out more about the influential case, and how it differs from the other cases (this obviously presumes you were involved in collecting the data or can communicate with the people how were).
