Low p-value, but many trials I am not well-versed in statistics, so please bear with me. I will try to explain things to the best of my knowledge.
I have some hardware that produces a signal, this signal has an offset and I want to subtract this baseline from my signal. This is the procedure I use to determine the baseline:
I take many noise samples (with long time between them to avoid any correlation) and from these I obtain the mean (my baseline) and the standard error on the mean (sample standard deviation / sqrt(samples)).
This looks good, but I now want to determine when I need to calibrate my baseline again, so I repeat the same procedure and obtain a new_mean, new_std_error, new_number_of_samples. I perform a two-tailed Welch's unequal variances t-test on these and if the p-value is low enough (say 0.05) I determine that my baseline has changed and I need a new calibration.
What is the problem?
My device has 256 channels, and if I perform this procedure on all channels I will most certainly obtain a low p-value for a couple of channels by pure luck. I don't think that this means these channels need a new baseline.
Other device has 20 thousand channels, even more likely to get a low p-value for some channels. Do you have any suggestion on how to deal with this? How could I determine if there is any actual reason to update the baseline of any given channel?
Thanks
 A: When deciding whether to adjust the baseline you need to consider more than just the long run rate of adjusting it when that channel does not actually need it.
The p-value is an index of the strength of evidence in the data against the null hypothesis according to the statistical model. In the hypothesis testing framework you make a decision on the basis of the relationship between that p-value and a pre-determined fixed threshold. That threshold should be set to a level that is chosen to optimise not only the long run rate of false positives (i.e. channel adjustments that are made but not needed), but also the power to detect false nulls (i.e. detect occasions where a channel needs adjustment) and the necessary sample size. The considerations of threshold and sample size involve thinking about the consequences of false positive and false negative errors. DO NOT use p=0.05 as a threshold and think that you are 'doing' statistics!
What are the consequences of adjusting the baseline of a channel that does not actually need adjustment? If they are trivial then adjust when there is modest evidence against the null hypothesis that the baseline is correct. What are the consequences of failing to adjust an out of baseline channel?
I would suggest that you consider a procedure where you monitor the channels and whenever a channel's p-value (or other index of out of balance condition) is low enough you sample again and adjust if a second p-value is also low. The chances of getting a spuriously low p-value two times in a row is very low, but if the channel needs adjustment then it should regularly give low p-values. That way you will hugely reduce the rate of false positive errors without also greatly reducing the ability to detect the need to adjust. (This assumes that a fresh sample of values has minimal cost.)
On a non-statistical level, why not use an AC-coupling approach (capacitative coupling) for the hardware? That is often used when there can be baseline drift and it deals with the problem very gracefully. It can also be done digitally. See here: https://en.wikipedia.org/wiki/Capacitive_coupling
A: This is closely related to the concept of change detection.
For example the CUSUM algorithm may be suitable. Basic implementation is:

*

*Cumulatively sum (sample - current_baseline) as data arrives.

*For every sample subtract some fraction of the standard deviation from the cumulative sum.

*If sum goes below 0, clamp it to 0. If sum goes above N times standard deviation, consider a change to have occurred.

Compared to taking discrete samples and computing statistics on them, a continuous process will react faster to a consistent change, such as new baseline value. The cumulative structure also means that small changes in baseline will still be eventually be detected, even if they don't immediately cross the threshold.
If you prefer the statistics structure, you can use a similar cumulative approach: expand the new sample count as long as the baseline is in effect. The longer this goes on, the smaller p will become and eventually it will cross even a very low p criteria.
This does not eliminate the fundamental problem of setting a threshold: a very small baseline change is indistinguishable from random variation for a very long time. But a cumulative approach should eventually detect even a tiny baseline change.
A: From a purely statistical perspective, this is exactly what multiple testing correction is for. The first "design" question for you is what exactly you would like to control:

*

*Either you want to keep the probability that there is any false positive (i.e. spuriously significant test) below a certain threshold $\alpha_1$ - this leads to the so-called Bonferroni correction method, where you essentially multiply your raw p-values with the number of tests (in fact, there exists a number of variations on this, but all of them share this goal and principle)

*Alternatively, you could say that of your "hits" of low p-values, you'd like, in expectation, at most a certain fraction $\alpha_2$ to be false positives, i.e. randomly spuriously significant tests, so that on average a proportion of $(1-\alpha_2)$ of your significant tests results from a real signal. This leads you to the False-Discovery-Rate family of methods, which prescribe a somewhat more complex procedure to correct your raw p-values

The former is evidently much more conservative than the latter, but there is no "objectively correct" choice - rather it really depends on the tradeoff between false positive and false negative rate. From your description, it seems that the FDR correction could be more suitable to your problem, but the decision ultimately rests with you.
