0
$\begingroup$

I am working with protein data at different time-points. After log transformation many proteins show a linear incorporation over time. Therefore I can use a simple linear model to study its trend.

For some proteins I have a 7 data for 7 time-points, but some others I have only data for 3 time-points (it was not detected at some time-points).

Since I am going to do many tests on the significance of the regression I thought I should be correcting my p-values to reduce the amount of false positives. I have been using the Bonferroni, Holm-bonferroni and permuted FDR methods. When I calculate the adjusted p-values I use the data from all the proteins, regardless if they have measurements for 3 or 7 time-points. This causes that many proteins with 3 and 4 time-points are not deemed significant by the Bonferroni and Holm-Bonferroni adjustments. The number of proteins that have 3 time-points is 5 times smaller than the number of proteins that have 7.

My question: should I perform the p-value adjustment within the time-point? Therefore calculating a different threshold for proteins with 3 timepoints, a different one for proteins with 4... etc.

Thanks

$\endgroup$
5
  • $\begingroup$ p-values do not serve you well here, and remember that when the p-value is large you have no conclusion whatsoever (i.e., you can't say "no effect"). Create confidence intervals for the effects you are interested in. These will be wide if you are on shaky ground, i.e., have too small $n$ to draw conclusions. Thinking about thresholds for p-values has gotten us all into trouble, and you certainty don't want to play with different thresholds. $\endgroup$ Jan 4 '18 at 12:59
  • $\begingroup$ But for a linear correlation, isn't it that large p-values indicate that there is no significant correlation and vice versa? I quite do not understand why to make use of confidence intervals since I am trying to asses if the correlation is linear (I do later look into the residuals to make sure). $\endgroup$
    – Marc P
    Jan 4 '18 at 13:55
  • $\begingroup$ Large p-values define, that there is no significant effect. That does by no means imply, that there is no effect. Not being able to reject $H_0$ does not mean, that you can accept $H_0$. p-values have a psychological tendency to make you believe they mean something, that they do not. $\endgroup$
    – Bernhard
    Jan 4 '18 at 14:48
  • $\begingroup$ Also: Bonferroni and friends lessen the risk of $\alpha$-error whilst increasing the risk of $\beta$-error. It's what you observe with 3-time-point-proteins: You increase the risk of not detecting an effect that is truely there. $\endgroup$
    – Bernhard
    Jan 4 '18 at 14:53
  • $\begingroup$ Yes, I understand that Bernhard. What worries me is that applying a correction based on all the proteins is being too strict on proteins not found in all time-points, since there are less of that kind. $\endgroup$
    – Marc P
    Jan 4 '18 at 14:57
0
$\begingroup$

We have to consider what a p-value is telling us. A p-value is the probability that the data will occur if the null hypothesis is assumed to be true. Changing your p-value in order to get a result is just moving your target so that you can discard your null hypothesis. Is there an intuitive/logical reason that one of your data sets should be expected to be closer to the null hypothesis based on what is happening in the real world? Unless there is a defensible reason that you are altering the p-value it is likely that altering the p-value will cause you to be accused of p-hacking.

Unfortunately, in order to be scientifically rigorous, you need to plan your p-value alteration before you collect your data and provide rationale for why you are doing it.

You may be better off performing a data transformation (such as dropping points from your 7-point data set) or with using a method that does not require data sets have matching numbers of points.

Whichever method you choose, it should be clear in your write-up that your analysis methods changed after data collection. This is necessary for understanding the scientific rigor of your analysis.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.