1
$\begingroup$

I thought I'd first give a brief description of what my data is so that it's easier to understand what my problem is. I have a dataset which goes as follows:- binned mass differences of compounds vs binned pearson correlation coefficients vs frequency

I've calculated the differences of the masses of compounds and then calculated the pearson correlation coefficient values of those mass differences to see whether some of those mass differences derive from compounds that are of the same parent compound (i.e. fragmented compounds etc). I did this by using some other data that I won't talk about here but it's sound. Now, I want to look at the frequency of the mass differences across the pearson correlation coefficient bins to see at which point of the correlation coefficient do we begin to see an increase in mass differences that derive from the same compound and to compare datasets obtained from different instruments and biological matrices etc to see whether this makes a difference. From plotting the maximum value of frequency against the pearson correlation coefficient bins, I get this graph here:- enter image description here

Going up is the frequency and going across is the pearson correlation coefficient bins (there's 40 bins that stretch from -1 to +1, meaning they go up in 0.05, like -1 to -0.95, -0.95 to -0.9 etc). As you can see, the data is normally distributed for pretty much half the data (up until 20, which represents the bin -0.05 to 0 etc), but it reaches a point where it begins to go back up and it no longer follows a normal distribution (around 30). Is there a way of somehow using the distribution of data to define a cut off point as to when the data no longer follows normal distribution? So like plotting a Gaussian curve onto the data and looking at the point at which the data deviates from the curve etc. Hope this makes sense.

Edit: I don't know if this would be a good way of doing it, but if we used the first half of the data and mirrored it to generate a mean and std etc, is it possible to somehow then define an equation for that normal distribution curve and somehow use that to see which points from the original data (the points in the second half) deviates from the normal distribution and where this begins?

Improve this question
$\endgroup$
3
  • $\begingroup$ Your correlations are bounded whereas a Gaussian (normal) isn't. Looking bell-shaped isn't sufficient to define a normal distribution. Why not just focus on identifying a change in slope? $\endgroup$
    – Nick Cox
    Apr 13, 2015 at 13:51
  • $\begingroup$ Yeah I was contemplating on doing just that, could you write that as an answer so I can accept it? Cheers $\endgroup$
    – user2062207
    Apr 13, 2015 at 14:23
  • $\begingroup$ I don't think I want to say more! $\endgroup$
    – Nick Cox
    Apr 13, 2015 at 14:25

1 Answer 1

Reset to default
0
$\begingroup$

Yes, probably the best way is to fit a Gaussian on tyou data and then you could truncate them as the differences between the data and the fitted distribution exceeds a certain level (e.g. 20%). This way you should get a value around 30 in your case.

Is a bit tricky to determine the correct parameters for the Gaussian, since you probably will have to do it semi-manually, because the right tail would derail your estimates.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Could I use the first half of my data to create the Gaussian (i.e. the left tail) and then from that fit the Gaussian and truncate points for which the difference between their value and the fitted distribution exceeds a certain level as you explained before? Would there be a significant disadvantage in using just the first half of the data to do this etc? $\endgroup$
    – user2062207
    Apr 13, 2015 at 13:45
  • $\begingroup$ Probably it would: the mode (i.e. $\mu$) may not correspond. Anyway you can give it a try and see if the two distibutions are very shifted. $\endgroup$
    – Stefano R.
    Apr 13, 2015 at 13:59
  • $\begingroup$ @user2062207 discarding a part of your sample because it does not fit to your model is one of the worst things you could do. Imagine that you need to fit a linear regression and remove from the sample every value that does not lay on the straight line for regression - by doing this you could possibly "fit" any regression line to any dataset (in worst case you'd have to remove all but 2 cases for fitting a line). $\endgroup$
    – Tim ♦
    Apr 14, 2015 at 15:21
  • $\begingroup$ That is totally off-topic. user2062207 asked how to truncate his data: what he'll do with them is none of our business. $\endgroup$
    – Stefano R.
    Apr 15, 2015 at 8:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.