2
$\begingroup$

From Doruk Cengiz. Seeing beyond the trees: Using machine learning to estimate the impact of minimum wages on affected individuals. 2019.

I noticed a strange way of building the training and test sets which suggests they come from different probability distribution. The author, however, does this on purpose!

In the training sample, we construct the prediction model [...] We drop all observations from the states where there occurred at least one minimum wage increase in the last 4 quarters to allow wages to adjust to the new environment.

[...]

The test sample, on the other hand, is composed of all observations in the states where there is a minimum wage event in the subsequent 4 quarters that are not in the training sample.

Aren't training and test sets coming from different distributions usually a problem to be avoided, not something that is done on purpose?

I e-mailed the author to ask what he meant but he is not answering.

Thanks in advance!

$\endgroup$
  • $\begingroup$ I haven't read the paper, but this is in general to be avoided - manufacturing your training and test set. I can't see a reason for doing this, unless maybe they are really not of interest. $\endgroup$ – user2974951 Sep 20 at 6:28
1
$\begingroup$

Very intersting question, and I've been reading through this paper for a little while now, trying to understand, and here's my best guess at why this was done:

The ML part of this paper is just restricted to just the first part of the study, where the objective is to simply predict (using the hourly wage variable in the dataset) which individuals earn an average hourly wage of less the 125% of the statutory minimum.

The authors are trying to build their prediction model only using demographic features ("age, race, ethnicity, gender, education, marital status, citizenship/nativity status, and rural status of the residency"), so they want to make sure that the wage rate value that is observed for each individual in the dataset is only influenced by these demographic features and not prior minimum wage hikes, which in theory, could effect everyone's wage rate in different ways. For this reason, they exclude all observations from state's where there's been "at least one minimum wage increase in the last 4 quarters to allow wages to adjust to the new environment".

With regards to the test set, if the the model developed in training does a good job using only these demographic features to predict, then it shouldn't matter so much whether or not these events are included, but in doing so, they can potentially increase the model's robustness.

If you hear back from the authors please post their answer here, as I'm also very curious to hear what they have to say. Thanks for sharing the paper.

$\endgroup$
  • $\begingroup$ Thank you! I will make sure to post here if the author answers me. $\endgroup$ – vaporK Sep 20 at 10:14
1
$\begingroup$

Disclaimer: I have not read the 95 pages paper.

But I'll tackle the more general question:

Aren't training and test sets coming from different distributions usually a problem to be avoided, not something that is done on purpose?

No it's not a problem. On the contrary, IMHO it should be done more often. What is problematic is using a model to predict data whose distribution was not tested.
(Usually, testing other distributions is done only after testing the training distribution, so that is available as well.)

Testing other distributions on purpose

The testing usually presented in papers using the same distribution is part of verification (making sure the model meets its specifications when used on the distribution it was intended for). Validation asks a wider question: whether the model gives suitable answers for the application. (The terminology varies by field, and is somewhat confusing as well).
In my field, ruggedness means the ability of a method (e.g. model) to work under less than ideal conditions. So we take the view, that data distributions will change (e.g. sensor aging, instrument drift, taking the method from a nice lab into a factory, ...). So you may want to validate that your method will work also under less than ideal conditions (i.e. is suitable for actual, every-day application use).

Someone else operating the instrument than the one who collected training data, temperatures slightly outside spec, you may also deliberately distort data according to physical effects that you know could affect the instrument (see e.g. Sattlecker et al.: Assessment of robustness and transferability of classification models built for cancer diagnostics using Raman spectroscopy, J Raman Spectrosc, 2010, 897-903. DOI: 10.1002/jrs.2798)
I sometimes work with data where cosmic rays leave artifacts. Such data are often best excluded from training, but it is important to know how the model performs both for clean and affected data (i.e. I check whether I can omit the cleaning step for application use).

Using a control distribution

From what you describe about the paper, I suspect a different use: they identify a "control" population and model it. Then predict rather than test the "affected" population and check whether the prediction is off (= there's an effect of the minimum wage events).

If that is so, the prediction of the states that were not in training is somewhat risky as it is an extrapolation to unknown states. However, the performance for that could be approximated by splitting the training states into training and testing states, and predicting the other states only after establishing that the performance for unknown states is fine.

Once that is established, it is fine to predict different distributions and check whether there are differences (this would be very much opposed to the implicit claim there are no differences which is hidden in using the model for normal prediction). So in a way, this would be a set up where the model is challenged by predicting a population that differs in important potential influencing factors. Iff those predictions are good, then the model was robust or rugged wrt. that influencing factor. If not, they presumably dig into the differences and attribute them to their influencing factor.

$\endgroup$
  • $\begingroup$ Thank you, you were very informative and thorough!! $\endgroup$ – vaporK Sep 22 at 10:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.