Using a different p-value threshold depending on number datapoints

I am working with protein data at different time-points. After log transformation many proteins show a linear incorporation over time. Therefore I can use a simple linear model to study its trend.

For some proteins I have a 7 data for 7 time-points, but some others I have only data for 3 time-points (it was not detected at some time-points).

Since I am going to do many tests on the significance of the regression I thought I should be correcting my p-values to reduce the amount of false positives. I have been using the Bonferroni, Holm-bonferroni and permuted FDR methods. When I calculate the adjusted p-values I use the data from all the proteins, regardless if they have measurements for 3 or 7 time-points. This causes that many proteins with 3 and 4 time-points are not deemed significant by the Bonferroni and Holm-Bonferroni adjustments. The number of proteins that have 3 time-points is 5 times smaller than the number of proteins that have 7.

My question: should I perform the p-value adjustment within the time-point? Therefore calculating a different threshold for proteins with 3 timepoints, a different one for proteins with 4... etc.

Thanks

• p-values do not serve you well here, and remember that when the p-value is large you have no conclusion whatsoever (i.e., you can't say "no effect"). Create confidence intervals for the effects you are interested in. These will be wide if you are on shaky ground, i.e., have too small $n$ to draw conclusions. Thinking about thresholds for p-values has gotten us all into trouble, and you certainty don't want to play with different thresholds. Jan 4 '18 at 12:59
• But for a linear correlation, isn't it that large p-values indicate that there is no significant correlation and vice versa? I quite do not understand why to make use of confidence intervals since I am trying to asses if the correlation is linear (I do later look into the residuals to make sure). Jan 4 '18 at 13:55
• Large p-values define, that there is no significant effect. That does by no means imply, that there is no effect. Not being able to reject $H_0$ does not mean, that you can accept $H_0$. p-values have a psychological tendency to make you believe they mean something, that they do not. Jan 4 '18 at 14:48
• Also: Bonferroni and friends lessen the risk of $\alpha$-error whilst increasing the risk of $\beta$-error. It's what you observe with 3-time-point-proteins: You increase the risk of not detecting an effect that is truely there. Jan 4 '18 at 14:53
• Yes, I understand that Bernhard. What worries me is that applying a correction based on all the proteins is being too strict on proteins not found in all time-points, since there are less of that kind. Jan 4 '18 at 14:57