How to determine if application acceptance is "fair"

Question

I'm a part of a volunteer organization that organizes a bunch of events. For each event, members need to submit an application in order to attend the event. A lot of people want to go to these events and we only have a limited number of spots available, so the selection process is usually pretty competitive. The application selections (accept/reject) for each event are usually handled by a different group of people, so it can be somewhat challenging to ensure that the selection process is as "fair" as possible across the board. I happen to have access to all of the relevant data (who applied to which events, who was accepted/rejected, some stats on the applicant, etc.), and I'm trying to assess whether the current acceptance policy is "fair" or not.

In a perfect world, each applicant would be accepted the same proportion of the time. Obviously in real life this may not necessarily be the case. For example, if 5 people apply to 10 events each, and are accepted to 5,7,6,6,5 of them, the system appears to be mostly fair, since the difference between the acceptance proportions isn't that big. On the other hand, if the acceptance numbers are 2,3,9,8,3, the system is obviously unfair since a few people are being selected repeatedly at the expense of the others. What's the most meaningful way to quantify this in terms of actual statistics?

Also, I have access to some data which might influence the acceptance rate of a particular individual, for example the number of articles written by them in the past year. It makes sense that someone who has written more articles should be accepted to more events. Also, if someone has, for example, only applied to 1 or 2 events ever, it makes sense for them to have a higher acceptance rate than someone who has applied to dozens.

So in short, I'm trying to come with a way to figure out if the current distribution of acceptance proportions is correlated in terms of some of the variables to which I would expect it to be correlated, as opposed to based solely on luck, and if it does turn out to be based solely on luck, whether the proportions are roughly uniform across the population or some people are repeatedly being favored at the expense of others.

Can someone guide me in the right direction here?

Answer 1

Undergrad Stat Major here. I'm interested in your question, but am no expert. Experts, do please correct me where I'm wrong.

So it sounds like the current criteria for selection are subjective, and possibly nondeterministic (e.g. the same application may be accepted one day and not another for no apparent reason). That sounds difficult, if not impossible, to quantify in any meaningful way. You've got a complicated situation, but I think you need to first analyze what you're trying to achieve.

It sounds like you'd like to do 2 different things:

1. come up with a way to quantify the process that you think can be relied on to make "fair" decisions, and
2. determine whether the current system is "fair" in some respect

The first is "model building". You take what you know and build a model of how you think the selection process works, or how it should work ideally. This depends on what you think is fair, based on what you value in these applications. In other words, maybe perfect equality in acceptance isn't "fair" if it does not favor those who are more valuable to your organization. Maybe you value number of articles a lot more than you do years of experience. I'd recommend coming up an idea of how valuable features are relative to each other, building models based on that, and seeing if you like the results based on how your models would decide acceptance based on sample applications. When you have a result you think is "fair", you've probably got a decent model for your purpose.

The second thing you want to accomplish involves generating hypotheses about what you think is fair (i.e. building ideal models), and testing past results based on those models to see how they match up.

You can also perform regressions against the data from the current selection process to see which features are significant and which are not. Simple correlation data would paint a nice picture to begin with, but performing some formal variable selection and regressions would be the next step in analysis.