Here's some background before my question (Bear with me! :D)

I am working on a data pipeline that deals with a massive input log dataset; instead of saving the full input log, current version of this pipeline captures a reservoir sample of the input log over a certain set of fields; e.g. say we are capturing reservoir samples for columns C1, C2 of the log with sample count of 10K.

The pipeline is very resource-intense (burns a lot of CPU) and having us explore ways of cutting down costs of this and other related computations. We have a badass in-memory compute engine that is super efficient with many things, but with this input dataset, it runs into 2 challenges -

  1. The approach to building reservoir samples over C1, C2 in this compute engine requires reading the entire input dataset - causing obvious **out-of-memory (OOM)** failures
  2. In concept, we could break the input log into "segments" based on C1, C2 values, so the in-memory compute engine processes a few segments at a time, and gets around the OOM problem. However, the resulting distribution has terrible skew (coefficient of variation > 20) - this skew hurts the logging application (critical process) badly and is unacceptable
  3. However, if we split the input log by C1, C2, C3, C4 i.e. add a few more columns to the "key" used to break the input log by - the resulting distribution is a lot more uniform (coefficient of variation drops to around 6) - at which point the performance hit to logging is pretty low

Stats question :)

Is it possible to sample the superset C1, C2, C3, C4 to get a reservoir sample of the subset C1, C2?

  • $\begingroup$ I think you are more likely to obtain meaningful answers if you edit your question and present it as a sampling problem in terms that everyone will understand - not just technical people working with AWS' data pipeline. Can you more generally describe what a "data reservoir" is and why you are trying to sample from it? Many of the sampling statisticians here, have probably never worked with AWS. $\endgroup$ – StatsStudent Jan 8 at 18:36
  • $\begingroup$ Thanks @StatsStudent! This isn’t on or related to AWS. $\endgroup$ – Bi Act Jan 9 at 0:15
  • $\begingroup$ I see, @Bi Act. Even more reason to clarify some of the purely non-statistical and perhaps overly technical terms that appear in your post. $\endgroup$ – StatsStudent Jan 9 at 0:17
  • $\begingroup$ Sorry not sure what happened there with my comment there - reposting my comment $\endgroup$ – Bi Act Jan 9 at 0:37
  • 1
    $\begingroup$ Thanks @StatsStudent! This isn’t on/related to AWS. By ‘reservoir_sampling’, I meant :) en.m.wikipedia.org/wiki/Reservoir_sampling since like I said the input dataset is massive & we wanted a sample representative of input distribution while ensuring rare events are at least rarely seen. I can rephrase specifics in the question - but overall this seems like an exercise in sampling/probability/algorithms - how do you build a reservoir sample for a set of ‘columns’ from samples you collect over a ‘superset of those columns’? ‘Columns’ in this case refers to fields in the input dataset $\endgroup$ – Bi Act Jan 9 at 0:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.