0
$\begingroup$

I have a volume of brain which contains thousands of connections between hundreds of cells. Suppose I observe that Cell type A connects three times more to Cell type B than Cell type C, and calculate the probability (p-value) of observing this pattern if all of the connections were in fact randomly distributed amongst the cells (the null hypothesis), for instance, by running 10,000 simulations using the same cell positions with randomised connections.

My confusion comes when I think about how I would test for the replicability of the finding across different brain samples. I could obtain two or three more biologically independent samples, repeat the exact same analysis, and see what the p-value is in each case - and if the p-value is very low in each case, then is this adequate to demonstrate replicability? If the p-value in each case were 0.001, would an 'overall p-value' for three independent samples be 0.001*0.001*0.001 = 0.1e-9? It seems too simple, and I don't think I've ever seen P-values combined in this way.

Alternatively, if the test statistic is the ratio (number of cell type A to cell type B connections):(number of cell type A to cell type C connections), then perhaps I would actually need to repeat the same experiment across many independent biological replicates to establish the distribution of this statistic, which would enable me to calculate a 95% confidence interval, and then see whether it overlaps with 1 (which would be the null hypothesis where cell type A connects just as frequently to cell type B as to cell type C).

Improve this question
$\endgroup$

2 Answers 2

Reset to default
0
$\begingroup$

Think of p-value as the probability of an expected event to happen by chance and not because of your expectation or treatment. Indeed, if you have three independent events, the probability that all three to happen is the product of their probabilities. However, you cannot consider the three samples independent, as we try to compute the probability of the phenomenon to happen by chance, and not about the probability of a specific sample to perform in a certain way.

However, there is a certain amount of approximation in computing the p-value. I don't know how you landed to the value of 0.001 in the first place, but it looks like you use a bootstrapping approach. If you did 10000 determinations, of which 10 appeared to be against your expectation, it doesn't mean that the p-value is 0.001. You need to identify the distribution of your variable, and based on how much your variable deviates from this distribution you can compute an accurate p-value.

Improve this answer
$\endgroup$
5
  • $\begingroup$ My null hypothesis would be that the connections between brain cells are in fact random. Since there are no real brain samples where this is known to be the case, the only way I have of generating a distribution of my variable under the null hypothesis is with a series of simulations, keeping certain features the same from my single real sample (e.g. positions of cells, total numbers of connections, number of connections made by each cell), but randomizing the actual connections each time. $\endgroup$
    – Alex
    Commented Dec 1, 2017 at 22:11
  • $\begingroup$ This will give me the distribution of a variable under the null hypothesis (e.g. number of connections from cell type A to cell type B), albeit from a simulated dataset (but with random constraints). With these caveats, one could calculate the probability of observing any variable observed in the single real dataset, under the null hypothesis (which is that all the connections are random). $\endgroup$
    – Alex
    Commented Dec 1, 2017 at 22:12
  • $\begingroup$ In such a situation, where so much depends on a simulated set of connections, it seems ideal to at least repeat the analysis in biologically independent (i.e. from different individuals), and see if the same variable is observed. If one did this procedure for three independent samples, one would generate three different p-values (quantifying the probability of observing the variable in question under the null hypothesis in each sample). $\endgroup$
    – Alex
    Commented Dec 1, 2017 at 22:12
  • $\begingroup$ I suppose I'm not sure whether it is necessary to combine these p-values, do a different analysis when one moves from a single sample to multiple samples, or just report that the same significant difference from the null hypothesis was observed in each of the three samples, quoting each p-value separately. $\endgroup$
    – Alex
    Commented Dec 1, 2017 at 22:12
  • $\begingroup$ You can divide samples as you like, probabilities shall remain the same. In your case, you have P = probability that connection between brain cells are random. Further you have P|A - P given that the phenomenon is observed on the data-set A, P|B - P - given that the probability is observed on the set B. P|A and P|B are not independent, even if A and B are independent. Your objective is to determine P, which is the p-value you look for, but you consider P|A to be your p-value. $\endgroup$
    – user121113
    Commented Dec 9, 2017 at 13:04
0
$\begingroup$

Suppose your p value is 0.01. Now suppose you test 100 brains. Even if the relationship is true, you have a good chance that one of the brains will not reject the null hypothesis. P values are more about the rejection of the null hypothesis than the measured probability of some event.

Given the limited information you gave, I think what you really want to do is much closer to your #2 idea. You want something like that ratio, and in an ideal world that statistic would look adequately like a normal distribution. If, in fact, you believed the numbers you are dividing are themselves normally distributed, then their ratio would be a Cauchy distribution - which has the unfortunate property of an undefined variance/standard deviation. But that is a technicality to attack if and when you get there...

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.