I want to make sure I'm correct in my assumptions. I'm predicting financial returns by using different ML models. There are 4500 values in each dataset. The density plots of all models are shown like this:Density plot comparison

I clipped the x-axis at [-0.025, 0.025]. Now I wanted to see if the predicted values of the models have the same distribution as my observed values from the "sp500". I used z-normalization on the datasets to make them comparable.

For my Non-NN models, these QQ plots were created:

QQ plot Non-NN

Imho, it seems that all Non-NN models have the same distribution as the observed dataset.

For my NN-models, these QQ plot were created:

QQ plot NN

Here it seems that both, especially the LSTM model, have distributions that differ from the observed dataset.

Is it correct to assume that the NN models have less predictive value than the Non-NN models as their distributions differ from the observed dataset?

  • 3
    $\begingroup$ If you want to check that variables have the same distribution, don't "z-normalise" them first. If mean and SD are incidental, then do. Since I am not clear which applies here, I can't advise on correctness. Regardless of that, plotting log density is better for comparing tail behaviour and typically does no harm in comparing the middles of distributions. $\endgroup$
    – Nick Cox
    Sep 18 '19 at 12:29

I don't think a QQ plot is a good way to answer your last question. QQ plots compare quantiles -- which isn't what you want -- and you've normalized your data sets -- which isn't what you want either.

I applaud looking at accuracy graphically. But i would compare predicted to actual values (unnormalized) by

a) Using a scatter plot of one vs. the other for each model

b) Using a Tukey mean difference plot (aka Bland Altman plot - it's such a great idea that several brilliant people invented it)

c) A density plot and maybe a box plot of the differences.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.