2 added quote from source
source | link

EDIT: Here is a quote from Deming (decide for yourself whether he is a credible source, but he knew both Fisher and J Neyman):

Limitations of statistical inference. All results are conditional on (a) the frame whence came the units for test; (b) the method of investigation (the questionnaire or the test-method and how it was used) ; (c) the people that carry out the interviews or measurements. In addition (d), the results of an analytic study are conditional also on certain environmental states, such as the geographic locations of the comparison, the date and duration of the test, the soil, rainfall, climate, description and medical histories of the patients or subjects that took part in the test, the observers, the hospital or hospitals, duration of test, levels of radiation, range of voltage, speed, range of temperature, range of pressure, thickness (as of plating), number of flexures, number of jolts, maximum thrust, maximum gust, maximum load.

The exact environmental conditions for any experiment will never be seen again. Two treatments that show little difference under one set of environmental circumstances or even within a range of conditions, may differ greatly under other conditions-other soils, other climate, etc. The converse may also be true: two treatments that show a large difference under one set of conditions may be nearly equal under other conditions.

There is no statistical method by which to extrapolate to longer usage of a drug beyond the peritd of test, nor to other patients, soils, climates, higher voltages, nor to other limits of severity outside the range studied. Side effects may develop later on. Problems of maintenance of machinery that show up well in a test that covers three weeks may cause grief and regret after a few months. A competitor may stop in with a new product, or put on a blast of advertising. Economic conditions change, and upset predictions and plans. These are some of the reasons why information on an analytic problem can never be complete, and why computations by use of a loss-function can only be conditional. The gap beyond statistical inference can be filled in only by knowledge of the subject-matter (economics, medicine, chemistry, engineering, psychology, agricultural science, etc.), which may take the formality of a model [12], [14], [15].

Deming, W. Edwards "On probability as a basis for action" The American Statistician, volume 29, 1975

https://www.deming.org/media/pdf/145.pdf

EDIT: Here is a quote from Deming (decide for yourself whether he is a credible source, but he knew both Fisher and J Neyman):

Limitations of statistical inference. All results are conditional on (a) the frame whence came the units for test; (b) the method of investigation (the questionnaire or the test-method and how it was used) ; (c) the people that carry out the interviews or measurements. In addition (d), the results of an analytic study are conditional also on certain environmental states, such as the geographic locations of the comparison, the date and duration of the test, the soil, rainfall, climate, description and medical histories of the patients or subjects that took part in the test, the observers, the hospital or hospitals, duration of test, levels of radiation, range of voltage, speed, range of temperature, range of pressure, thickness (as of plating), number of flexures, number of jolts, maximum thrust, maximum gust, maximum load.

The exact environmental conditions for any experiment will never be seen again. Two treatments that show little difference under one set of environmental circumstances or even within a range of conditions, may differ greatly under other conditions-other soils, other climate, etc. The converse may also be true: two treatments that show a large difference under one set of conditions may be nearly equal under other conditions.

There is no statistical method by which to extrapolate to longer usage of a drug beyond the peritd of test, nor to other patients, soils, climates, higher voltages, nor to other limits of severity outside the range studied. Side effects may develop later on. Problems of maintenance of machinery that show up well in a test that covers three weeks may cause grief and regret after a few months. A competitor may stop in with a new product, or put on a blast of advertising. Economic conditions change, and upset predictions and plans. These are some of the reasons why information on an analytic problem can never be complete, and why computations by use of a loss-function can only be conditional. The gap beyond statistical inference can be filled in only by knowledge of the subject-matter (economics, medicine, chemistry, engineering, psychology, agricultural science, etc.), which may take the formality of a model [12], [14], [15].

Deming, W. Edwards "On probability as a basis for action" The American Statistician, volume 29, 1975

https://www.deming.org/media/pdf/145.pdf

1
source | link

The tests you mention are not appropriate for your situation. They will only tell you the probability of getting a difference difference between years 1-2 and years 3-5 as or more extreme than the difference you observed if there was exactly zero difference from year to year. This null hypothesis is highly unlikely to be true regardless of whether tools were changed. It is also unlikely that classes in the future will be exactly the same type of students as in the past.

What you care about should be (I think) whether tool B will lead to higher participation in the future than tool A. This is an "analytic" problem, while the statistical tests you are attempting to use are meant for "enumerative problems". Yes, this type of use is very common and it has lead to about 80 years of misleading results in many fields.

The only way to make rational decisions is to have understanding of the underlying data generating process. If there is little background knowledge all you can do is plot the data and look for patterns that indicate there may be some lurking/confounding variable that offers an alternative explanation for the increase in participation. You should try to break up the data into as many plausible subgroups as possible (e.g., type of student) and look for patterns.

If you and other experts cannot think of any plausible alternative explanations then it would be rational to decide to continue using tool B. If an alternative explanation is available then further study is necessary to determine which is responsible. A good source on this issue would be William Edwards Deming.

https://en.wikipedia.org/wiki/Analytic_and_enumerative_statistical_studies

https://www.deming.org/media/pdf/081.pdf