0
$\begingroup$

Per https://stats.stackexchange.com/a/204977/384097 the suggestion was made that, given a known distribution, applying a cutoff as means of dealing with outliers is not data leakage. I feel uncomfortable with this unless there is a perfect way to ensure autocorrelations are not significant.

As an example given [1,2,2,3,3,3,4,4,4,4,5,5,5,6,6,7] we could then generate for each value a count [1 2 3 4 3 2 1] and may choose to throw away the 1 counts yielding [22333444455566].

Suppose the following:

By looking at the values, we may suspect that there is an upward trend with oscillations of decreasing frequency as we move away from 4 in both time directions. So context may say the trend is generally more monotonic the further from 4.

Question:

  1. Suppose we remove low-frequency values from both sides of the distribution [1 2 3 4 3 2 1], ie resulting in frequencies [23432] ie [22333444455566]

  2. Can we say that this removal of {1,7}, if our suppositional structure is not examined, results in a problematic peeking by removal of 1, but not in removal of 7 since rarity is measured w/r/t other values, and for 1 all those other values are known to be at later times? Whereas looking at 7, even if we know of a generally monotonic structure in time, is a look backward?

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.