Understanding hypothesis testing-compare two series

This is rather simple, but am always confused so perhaps somebody can explain me once for all. I have two sets of observations coming from two sources that I want to compare, say $e_1$ and $e_2$ both having $N$ points. Three things could happen:

i) $e_1$ is consistently larger than $e_2$, ii) $e_1$ is consistently smaller than $e_2$, iii) there is no significant difference between $e_1$ and $e_2$.

I compute the difference in the two series, $e_1 - e_2$, create a test statistic. This could be for example the mean of $e_1 - e_2$ divided by the standard deviation of $e_1 - e_2$ (divided by square root of $N$), or it could be another test statistic. If I wanted to do the Diebold-Mariano test it would be another test statistic. But the point is I get some test statistic. I don't know how to interpret this statistic.

I know I need to compare it against say the extreme value of a $T$ distribution or Normal distribution. But know my test statistic could be >1.96, <-1.96, or between -1.96 and 1.96. Help me understand how to interpret the three cases to answer the question- which is consistently larger, $e_1$ or $e_2$.

• This looks like a very generic question on hypothesis testing and it is likely a duplicate of one or more older threads on the topic. Other than the sketch in my answer, I suggest consulting an introductory statistics textbook. Commented Apr 26, 2023 at 17:28

The test statistic is a summary characteristic of your dataset. It can be treated as a realization from a random variable with a known distribution under a point-valued null hypothesis $$H_0$$ or a known family of distributions under a set-valued $$H_0$$. You can evaluate how atypical this realization looks from the perspective of the null distribution(s). Taking the distributions under the alternative hypothesis $$H_1$$ into consideration, this yields a $$p$$-value that you can compare to a predetermined significance level $$\alpha$$ to help you decide between keeping (i.e. not rejecting) $$H_0$$ or rejecting it in favour of $$H_1$$.
Alternatively, you can refer to the critical values that again are determined by the distribution(s) under $$H_0$$ in light of $$H_1$$. Compare your test statistic to them and see if the statistic is in a critical region (suggesting to reject $$H_0$$ in favour of $$H_1$$) or not (suggesting not to reject $$H_0$$).