I'm currently working on building Random Forest Models in python. My topic is to investigate the Imoportance of specific variables for the accuracy of machine learning to explain the market capitalisation. My Problem now is, that my R squared are quite reasonable but my the MSE are exorbitant large every time. I think its because I am handling with values that are really large (billions). Do I have to normalize or scale the data? I dont know what to do, because i have read that normalisation is not quite neccessary for random forest. I've tried everything else like feature importance, feature engeneering, cross validation , removing outliers etc etc... but the MSE stays in ranges like 2.72883556377383e+20 for example. Can anyone help me with that?
-
1$\begingroup$ Welcome to Cross Validated! Most implementations of an $R^2$ calculation, particularly for a random forest, will calculate by transforming the MSE. Thus, MSE and $R^2$ are two sides of the same coin: either both are good or both are bad. Thus, how do you come to the conclusion that $R^2$ is good yet MSE is bad? $//$ Remember that MSE is a squared value. Simply taking the square root will lower those values and put them in the natural units (say dollars or euros) instead of being in squared units (dollars-squared or euros-squared, whatever those mean). $\endgroup$– DaveCommented Sep 9 at 16:52
-
$\begingroup$ Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. $\endgroup$– Community BotCommented Sep 9 at 16:52
1 Answer
I see two possibilities for how you calculate $R^2$, and the answer depends on which one you use.
SQUARED PEARSON CORRELATION
This would be something like (np.corrcoef(y_true, y_pred))**2, to you some Python syntax.
$$ R^2 = \left(\text{cor}\left(y, \hat y\right)\right)^2 $$
You calculate by squaring the Pearson correlation between the true and predicted values, probably out-of-sample.
When you do this on in-sample (training) data for an OLS linear regression with an intercept, this is equivalent to other ways people express $R^2$, discussed here, for instance. However, outside of that scenario, such a calculation can miss major problems with the predictions. Here, I give some examples of when this happens.
Therefore, if your squared correlation is assessed to be good yet your mean squared error is not good, then you might consider plotting to make sure the true and predicted values align with each other. They should roughly follow the line $y = x$ or $y = \hat y$, with more variation about the line as the model fit gets worse.
TRANSFORMATION OF MEAN SQUARED ERROR
More likely, you are using an $R^2$ calculation equivalent to sklearn.metrics.r2_score in Python, perhaps even that function itself.
$$ R^2=1-\left(\dfrac{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\hat y_i \right)^2 }{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2 }\right) = 1-\left(\dfrac{ N\times MSE }{ \overset{N}{\underset{i=1}{\sum}}\left( y_i-\bar y \right)^2 }\right) $$
(This is what I meant in the comment that a common implementation of a random forest $R^2$ is a transformation of the MSE. The denominator is a property of the data, not of the model.)
If you are calculating this way, then your $R^2$ is a deterministic function of the mean squared error. I do recommend a tweak to the sklearn.metrics.r2_score calculation, but their calculation is a lower bound on mine; if their $R^2$ score is good, my recommended calculation is no worse, probably better. (Proving this could be a useful exercise, one I do not think should be very hard.)
With this calculation being a deterministic transformation of the MSE, either both are good or both are bad. If you assess one to be good and the other to be bad, you need to figure out which of those is not correct. There is no magic formula for model evaluation that gives you an objective letter grade like a $90\%$ is an $\text{A}$-grade in school that makes us happy while a $40\%$ is an $\text{F}$-grade that makes us sad. You have to give consideration to the context, in your case, your knowledge of the finance.
FINALLY, do note that mean squared error is a squared quantity. You are working with numbers in the billions of dollars. If you have a market capitalization of $200$ billion and predict a market capitalization of $190$ billion, that (probably) sounds pretty good. However, you still missed by ten-billion dollars! When you put that error through the mean squared error calculation, it contributes $1\times 10^{20}$, on the order of what you are getting.
You may feel more comfortable taking the square root of mean squared error, often called root mean squared error. This puts the units back in the natural units, say dollars instead of dollars squared, and you will wind up with a performance metric more like 16-billion dollars instead of 272-quintillion dollars-squared.
