Here's some background before my question (Bear with me! :D)
I am working on a data pipeline that deals with a massive input log dataset; instead of saving the full input log, current version of this pipeline captures a reservoir sample of the input log over a certain set of fields; e.g. say we are capturing reservoir samples for columns C1, C2 of the log with sample count of 10K.
The pipeline is very resource-intense (burns a lot of CPU) and having us explore ways of cutting down costs of this and other related computations. We have a badass in-memory compute engine that is super efficient with many things, but with this input dataset, it runs into 2 challenges -
- The approach to building reservoir samples over C1, C2 in this compute engine requires reading the entire input dataset - causing obvious
- In concept, we could break the input log into "segments" based on
C1, C2values, so the in-memory compute engine processes a few segments at a time, and gets around the OOM problem. However, the resulting distribution has terrible skew (coefficient of variation > 20) - this skew hurts the logging application (critical process) badly and is unacceptable
- However, if we split the input log by
C1, C2, C3, C4i.e. add a few more columns to the "key" used to break the input log by - the resulting distribution is a lot more uniform (coefficient of variation drops to around 6) - at which point the performance hit to logging is pretty low
Stats question :)
Is it possible to sample the superset C1, C2, C3, C4 to get a reservoir sample of the subset C1, C2?