1
$\begingroup$

I have been stuck with a couple of problems, so I decided to open an account here to post these questions of mine. Sorry if I am posting this in the wrong section.

I want to analyze a frequency distribution for neuronal currents before and after a treatment. I have two types of neurons; the values for each are depicted in the graph as red and black. I measure the interevent interval (IEI), that is, the time at which events (currents) occur (which is related to frequency; the shorter the interevent interval the higher the frequency). The most common test for frequency distribution people use in this field are Kolmogorov–Smirnov tests (ks-test).

These are my graphs (the number of events are close to 1000, those are obtained from ~10 individuals, so each individual has around 100 events that are pooled together for this test): data

Now my problem: The ks test is so sensitive that I always get a significant result. I can “see” from the graph that in the control condition the frequency distributions are very similar. On the “treatment” condition on the other hand differences in frequency distribution between both neurons are considerably bigger. But even when this is the case I always get a highly significant result for both cases

If I compare the distributions for red vs black neuron for control in GraphPad prism I get:

Kolmogorov-Smirnov test/ P value < 0,0001/ Exact or approximate P value? Approximate/ P value summary ****/ Significantly different? (P < 0.01) Yes/ Kolmogorov-Smirnov D 0,1244

When I compare red vs black neuron for “treatment” I get

Kolmogorov-Smirnov test/
P value < 0,0001/ Exact or approximate P value? Approximate/ P value summary ****/ Significantly different? (P < 0.01) Yes/ Kolmogorov-Smirnov D 0,1861/

If I do the same analysis in R I get:

For “control”: D = 0.12418, p-value = 4.34e-08/ alternative hypothesis: two-sided

For “treatment” D = 0.1779, p-value = 1.71e-13/ alternative hypothesis: two-sided

I guess I can decide that something is significant in this case if p < 1e-09 o p < 1e-10, but I have never seen something like that expressed on a paper.

So; is there any other way to analyze such frequency distribution to better represent what the graph is showing? I was thinking on splitting the IEI in bins (for instance 0-100 msec, 100-200msec, etc) and compare means, but then I would be losing information, as the most represented frequencies would “push” the data to that center. Also the error bars would be quite big, if I take the means for each individual.

Thanks!

A

|cite|improve this question
$\endgroup$
  • 6
    $\begingroup$ As you've discovered, a narrow focus on null hypothesis significance testing is frequently (pun unintended) not the best way to address a scientific question. To oversimplify, a p-value will tell you that they are different, and no more. So perhaps the question (for you) to address is: what are you trying to learn from this analysis exactly? $\endgroup$ – mkt Jun 3 '18 at 17:34
  • $\begingroup$ Thanks for the reply mkt. I agree with your comment, as I have always been of the idea to be careful when making claims from statistical results. To put in layman's terms: what I see from the graph (and when gathering the data) is that there is no difference between neurons in the control condition, but that the frequency of events is lower for the red neurons in the treatment condition. That also makes sense with other results I have. But I suppose it is easier to make an argument for something being “irrelevant” in real life when p<0.5 than with p<1e-08! $\endgroup$ – And Jun 3 '18 at 17:59
  • 2
    $\begingroup$ You point is true, but is not exactly the one I'm trying to make. Basically, I advise moving on from the narrow binary of different/not different to a focus on a more careful evaluation of the scientific hypothesis/question (as opposed to the statistical hypothesis). Typically, the next question a researcher asks after your present stage is of the magnitude of the difference i.e. effect size; you could examine this. And even more interesting would be if there were a priori quantitative predictions of such differences for you to evaluate. You know your field best, though. $\endgroup$ – mkt Jun 3 '18 at 18:18
  • $\begingroup$ You could probably use the D statistic as an effect size measurement to compare how different the distributions are in each plot. It would be up to your interpretation as to how large a difference in D values is meaningful. $\endgroup$ – Sal Mangiafico Jun 3 '18 at 20:32
  • $\begingroup$ You could try looking at uncertainty on the CDFs. This can be done easily with a bootstrap method or even with the DKW inequality. Since your sample size looks large, you might see that the uncertainty bounds heavily overlap in the first case, but not in the second. $\endgroup$ – knrumsey Jun 3 '18 at 21:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.