How to analyse a quota sample if some combinations are over-sampled?

Question

We did a quota sample going for age group, sex, and education level. Given that we asked strangers on the street, we could only assess education level after speaking to the people. Given the short questionnaire, we ask the questions anyway.

Now we have about 500 filled in questionnaires, but can match the quota only with about 250 of them. If we were to use all participants, academics would be over-sampled (university towns), while non-academics would be under-sampled (both men and women, more severely for women). To analyze the data with 250 cases, we take everyone from the limiting combinations. But what do we do with the other combinations which are over-sampled?

1. Just do a random drawing in these over-sampled combinations (so the number of cases matches the quota for that combination, e.g., we have too many academics in some age groups from which we can draw randomly)?
2. Draw repeatedly from the over-sampled combinations, and average the results?
3. Depends, depending on ...?
4. Some other way?

Not sure how to proceed here.

Edit: I forgot to mention, we aim for a representative sample (using distributions of the age, sex, and education level in the population to determine the quotas).

Answer 1

Raking is a statistical technique for adjusting or balancing observed values of categorical features in empirical data to their theoretical (true) values in the population, e.g., here ... https://en.wikipedia.org/wiki/Iterative_proportional_fitting. This wiki article tells us that the first methods date back to Yule in the early 20th c. Essentially, it's an approach to reweighting sample data to reflect population expectations.

Raking assumes that you have comparable information for both the empirical sample and the population that can be cross-classified into a contingency table. The approach then boils down to fitting a log-linear model. However, the "table" is an outcome or product of the routine. The inputs are usually in the form of counts or percentages of the univariate marginals for the grouping factors. While IPF routines have traditionally used less than 6 or 8 factors, macros have been written that are virtually unlimited wrt the number of factors that can be invoked (e.g., here http://www2.sas.com/proceedings/sugi25/25/st/25p258.pdf).

Here's a simple example (completely fabricated) of two demographic inputs to an IPF:

Based on these marginals and model parameterization, predicted percentages would be iteratively generated until convergence based on a criterion minimizing the change versus the previous iteration. The resulting predicted percentages would be the "weights" used to rebalance the sample to reflect the theoretical marginals in the population.

Raking still sees wide use today, e.g., in opinion polling, to name just one application. Problems with raking have been much in the news since the 2016 presidential election, specifically the failure of many pollsters to correctly predict the outcome of the election. As a consequence and while the final national vote totals were accurately forecast, the outcomes in several key, swing states were not. One of the problems with the approach most pollsters used to reweight their samples was that 'education' wasn't a factor. E.g., here ... https://www.nytimes.com/2017/11/06/upshot/after-a-tough-2016-many-pollsters-havent-changed-anything.html

Answer 2

You may have wanted to sample people according to an education-related quota, but unless there are details of sampling you left out in your description, that's not what you did—you sampled people without checking their education first. Now that you have the data, the best you can do is analyze all the data you have.

If you're concerned that education relates in an important way to your outcomes of interest, either directly or in interaction with another variable, you can still control for it in your analyses, since you measured it.