Why would somebody use a hash function for creating a test/train split instead of random seed?

Question

I'm going through some ML training material from Google (I can't post a link because I'm getting the material through my company).

In the part about how to extract data and split it into train and test sets, they're using a hash function on one of the data fields to provide a deterministic and repeatable test/train split instead of a random one.

But can't the same thing be accomplished with a random.seed function?

Moreover, using a hash function would mean we can no longer use the field on which the hash was generated (which might be potentially useful for a model) or it might be inserting some unknown bias into the model?

What advantage does using a hash function have over using random seed?

Answer 1

But can't the same thing be accomplished with a random.seed function? ... What advantage does using a hash function have over using random seed?

Sampling is less straight forward when you can't fit the entire dataset in memory. In the context of a DBMS, this article suggests that using RAND() with a seed may not be reproducible when writing SQL. This is due to the multithreaded nature of the application, which does not guarantee the order of the returned items (unless you add the ORDER BY clause, which might be expensive). The author of the article proceeds by hashing one of the date fields in each row to get around this problem.

One other plausible use case would be when dealing with files. If I have a huge directory of images that I want to use for training/testing, it might be easier to work with a hash of the filename rather than trying to maintain a reproducible ordering of the files.

Moreover, using a hash function would mean we can no longer use the field on which the hash was generated (which might be potentially useful for a model) or it might be inserting some unknown bias into the model?

Computing the hash of a field is not the same as computing the hash and then overwriting the original value. The hash would just be computed in some other memory block and used to assign the item to the train/test/validation set, the same way generating a random number does not overwrite any data.

With respect to introducing bias, I found this question on the cryptography site which attempts to address the statistical properties of SHA-1 mod n.

Answer 2

Adding to Joel's excellent answer, your MLops people will thank you for using a hash instead of a random seed. In many real world applications, you want to continue monitoring and retraining a model that is in production. Or you might want to try using a different programming language or data pipeline in the future. By using a hash function, you know that every observation (past, present, and future) will be reliably and reproducibly categorized, even as new data comes in and you move to a new system.

Here's a minimum reproducible example. Imagine I want to see if the mean of an important value has changed since we last updated our model. I don't want to use the holdout data in my analysis, which I'll define as an MD5 hash that starts with a, b, c, d, e, or f. I also don't feel like exporting all the data to my workstation just to look at some averages. I can write a quick SQL query to check the means over time without any holdout contamination.

SELECT date, AVG(value) FROM tbl WHERE LEFT(MD5(id),1) IN(0,1,2,3,4,5,6,7,8,9) GROUP BY 1

Now imagine I've downloaded some data into R. I can run the same analysis without having to worry about seeds being consistent between SQL and R.

tbl %>% mutate(group = str_sub(md5(id), 1, 1)) %>% group_by(date) %>% summarise(mean(value))

In addition to SQL and R, you could do the same thing with Python, Julia, Beam, BigQuery, etc.

My workplace is currently moving from one database system to another. Instead of having to worry about maintaining the same random seed across systems, I just have to Google "generate md5 in new system."