1
$\begingroup$

I'm hoping for a gut-check from someone with more formal stats education than I to tell me if my proposed approach makes sense or if I'm overlooking anything. I have a survey database where each record is a person who took a public transit trip on a certain day, the fare that they paid for that trip, the fare that they would pay under a new fare proposal (which varies based on how far they traveled), as well as demographic info. The data is a probability sample of all riders and is weighted.

I want to run an analysis to see if people from x demographic would be affected differently under the new proposal than people from y demographic. The results have implications for equity and our policies.

My plan was simply to calculate the difference in current vs proposed fare for each record, then calculate mean fare change for each demographic. The end result would be that x demographic would have a mean fare change of 23 cents and y demographic would have a mean change of -5 cents, for example. I could then test for statistical significance.

Is there a better way to determine if one group is disproportionately affected?

This seems overly simplistic so I just want to be sure I'm not missing a better way to do it because I don't know what I don't know. Thanks in advance for helping a newbie!

$\endgroup$
4
  • 1
    $\begingroup$ I don't know that an equality null makes sense here. We shouldn't expect that for something that's fair. I'd expect that a trivial effect should still be seen as fair. E.g if a new proposal left dime group half a cent per trip worse off relative to another group, would that matter? You may want an equivalence test or perhaps a non inferiority test, perhaps $\endgroup$
    – Glen_b
    Aug 3 at 22:43
  • $\begingroup$ Thanks @Glen_b. $\endgroup$
    – Kelly
    Aug 4 at 14:09
  • $\begingroup$ Sorry, I didn't see the "dime" in place of "some" error in my comment. A typo followed by a poor autocorrect. $\endgroup$
    – Glen_b
    Aug 5 at 2:23
  • $\begingroup$ I figured so, thanks! $\endgroup$
    – Kelly
    Aug 5 at 21:53
1
$\begingroup$

This seems correct to me. I have a few recommendations:

  1. Make sure you state your hypothesis clearly up front and clearly define X and Y. You run the risk of trying multiple slices and definitions of X and Y and "p-hacking" your way to a statistically significant result. It was unclear in your question whether you have a solid definition of X and Y up front, or if you'll be slicing through your data.
  2. Clearly distinguish between a statistically significant result and a meaningful result. While raising the fare for a group of users by $0.01 extra may be statistically significant, it may not be meaningful.
$\endgroup$
2
  • $\begingroup$ Thank you! That all makes sense. X and Y are either white people/people of color or low income/not low income riders based on definitions in the survey so I think it's pretty defined. I will look more into p hacking to understand the risks there. I appreciate your help. I'll keep this open for a day or two in case others have insight but will up vote your response in the meantime. $\endgroup$
    – Kelly
    Aug 3 at 21:51
  • 1
    $\begingroup$ p-hacking is about checking a bunch of slices of your data until you find something statistically significant. The problem with this is that a significance threshold of 95% (p<0.05) means that you have a 5% chance of finding a significant finding when there is no actual difference. The way you get around this is by declaring what you're going to look at in advance, and then run the analysis. $\endgroup$ Aug 5 at 6:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.