Test to prove two means do not have a difference I have a population that I measure a value from once a day. I then made a change, and continued to measure that same value from the same population. 
What test could I use to prove that my change did not affect the mean of that value? I'm not sure if the normal difference between two means tests that I know would apply because I want to prove that the means have no difference, not find the difference. I think this would be different from a standard t-test. 
 A: Normally in a statistical hypothesis test, you are seeking to reject the null hypothesis in favour of the alternative hypothesis, so the precautionary principle suggests that the onus is on us to show that the null hypothesis is likely false, so we require $\alpha$, the false-positive rate (the probability of rejecting the null hypothesis when it is true) to be low.  It is also possible to run the test the other way round, where the experimental hypothesis is the null hypothesis.  In this case the onus is on us to show that $\beta$ the false-negative rate (the probability of accepting the null hypothesis when it is false) is low.  This amounts to showing that the test has sufficient statistical power.
Essentially if we are unable to reject the null hypothesis it is because the null hypothesis actually is true, or it can be because the null hypothesis is false, but we don't have enough data to be sure that it is false.  We can be confident it is the former, rather than the latter, if the test has good statistical power.
Caveat: I find frequentist statistical tests conceptually rather complicated, so I may well have written something that would make a purist wince, but hopefully the point about the need for statistical power of the t-test will be helpful.
A: You cannot prove that there is absolutely no change unless you have infinite data/information.  However you can show that the amount of change is limited.  The best approach is probably the Bland-Altman methods, see the links on this page for more detail.  The traditional Bland-Altman method deals with 2 different measurements on the same subjects, but would work equally well for other paired data cases such as you describe.
A: When you said that you made a change, I assume that you meant that you made a change that may impact the outcome that you are measuring.  In this case you should consider trying to measure a treatment effect, which can be recovered by difference-in-differences.  
You are correct that you cannot simply compare differences in means before and after the change since unobserved variables may influence your outcome variable over time.  To control for these time effects, you would want to keep measuring one group (the control group) without the treatment and then measure a treatment group that receives the change.  If the two groups are statistically similar then we can consider the control groups outcome after the treatment period to be a proxy for the counterfactual outcome.  
However, with all of this being said, you really cannot prove "no effect" but rather simply fail to reject that there is no effect.
A: To show that 2 groups are eqivalent you can run an equivalence test where you compare the confidence limits against a predefined equivalence boundary. This test is similar to a t-test but the hypothesis are reversed. Instead of the null hypothesis being no difference and the alternative hypothesis being that there is a significant difference in a t test, the null hypothesis in an equivalence analysis is that the samples are not equivalent and the alternative hypothesis is that the samples are equivalent. An equivalence test thus places the burden of proof on proving equivalence. Equivalence is proved if both the lower and upper confidence bound is inside an equivalence boundary, if one of the confidence bounds is outside the equivalence limits, then equivalence cannot be concluded.
The equivalence test is available in a lot of statistical software like Minitab and SAS.
