Lets say I am looking at a sample of records, say, sales data from several McDonald's restaurants.
Lets say I detect a difference between two groups of McDonalds. For example, McDonald's stores located in Oregon sell fewer McRibs per month than do stores in Tennessee.
Some descriptive stats (let's assume large outliers have been removed and that the sample size of each group is 2000):
Quantity of McRibs sold in the month of July
Oregon stores: 450 mean, 420 median
Tennessee stores: 700 mean, 520 median
I can conduct significance tests to see if the difference between the two groups is real.
However, is there another way to look at this: based on probability? For example, the odds that a given Store in Tennessee sells more McRibs than a given store in Oregon?
For example, if 40% of the Tennessee stores actually sell below the Tennessee median, say, 450 McRibs each, meanwhile, 40% of the Oregon stores sell at or above the Oregon mean, also 450 McRibs.
In this scenario, I would write a report concluding that McRibs are much more popular in Tennessee. This conclusion would be supported by descriptive statistics and tests of significance.
However, in reality, There is a 40% chance that a store in Tennessee sells no more McRibs than the average store in Oregon, weakening the validity of my report.