0
$\begingroup$

Example: I have am building a dog vs cat classifier and I have collected data from 15 countries.

Europe: 1. UK 2. France 3. Germany 4. Italy 5. Finland

Asia: 1. India 2. China 3. Japan 4. Russia 5. South Korea

South America: 1. Brazil 2. Argentina 3. Chile 4. Uruguay 5. Peru

These are the approaches that I am aware of:

Approach 1: Collect images from all these countries, shuffle them and divide them into train, dev and test data ( 60%, 20%, 20% ).

Approach 2: Randomly select 9 folders ( 60% ) use them as training data, out of the remaining 6, chose 3 to be dev data and 3 to be test data.

Question: So I want to know which of the two approaches is better or is there any other way of dividing the dataset??.

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ Specify the total sample size. Whether it's a good idea to split will depend on this being very large (as opposed to resampling validation). And think about not splitting on country but using country as a categorical feature. $\endgroup$
    – Frank Harrell
    Commented May 19, 2019 at 11:36
  • $\begingroup$ Total Sample size is around 450, 30 images per country, if I want the model to work on other countries as well, using country as a categorical feature would be counter intuitive right?? $\endgroup$
    – Ayush Pandey
    Commented May 20, 2019 at 10:54
  • 1
    $\begingroup$ Your sample size may not be large enough for holding back data. The question of prediction for countries not in the data is always a good one. I think that understanding country differences rather than declaring poor prediction for a currently represented country is valuable. You can also use country as a predictor and predict new countries using the mean over all current countries. If you use country as a random effect in regression you can predict a between-country range. $\endgroup$
    – Frank Harrell
    Commented May 20, 2019 at 12:01

1 Answer 1

Reset to default
0
$\begingroup$

It depends on what do you think is going to be happening to the model after deployed for inference. Is it going to be fed the data only from those 15 countries? Then the first approach is a way to go. Or do you expect the data coming to the model to come from different countries than you already have? Then the second option is the way to go.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.