# Iterative Deletion Algorithm to Test Robustness?

Just had an idea, and was wondering if there were existing strategies to implement it:

Let's say I run some well-specified linear regression and discover that a coefficient of interest (say a treatment variable in an RCT specification) is significant. Is there a strategy for discovering the fewest number of observations that would need to be deleted from your sample in order to negate the effect found for a particular dependent variable?

The brute force approach of leaving an observation out, and then every pair-wise observations out, and then every triplet, and so on, is obviously computationally prohibitive. Is there a way to make the search more efficient, though? Looking for high leverage points, for example?

This is more than just looking for outliers. Some combination of observations may collectively exert some kind of unique influence, but individually may blend in during visual inspections.

There are two separate ways of specifying 'the smallest number of records' to negate the effect. The first is to remove a specific subset of records, and the second is to remove a random subset of size n.

You describe the way to find the first method in your question. This will give you a specific subset of n records where removing them will negate the variable effect and there is no subset of n-1 records that will negate the effect. To find this you need to do an exhaustive search of looking at every record, every pair, every triplet, etc.

If instead you want to know on average, what is the minimum number of randomly chosen records that you can remove to negate the effect, then you can randomly remove subsets of different sizes until you find the smallest generic subset without the effect. Start by randomly removing 1/2 of your records and look for the effect (do this 10 times). If the effect is not there then remove 1/4 of the records from your full dataset and look for the effect. If the effect was there then remove 3/4 of the records from the full dataset and try again. Each iteration should cut the distance between unknown points in half.

EX: Remove 1/2 - effect is there

Remove 3/4 - effect is not there

Remove 5/8 - effect is there

Remove 11/16 etc.

• Thanks Bryan -- I have thought about this strategy. The issue is that this random strategy does not resolve the specific observations that would need to be removed (just, on average, what proportion of the dataset would need to be deleted). If there were 4 specific observations that, if deleted, would undo the effect -- how would I find these? An exhaustive search isn't possible. Are there heuristics to figure out ahead of time which four observations these are likely to be? – Parseltongue Sep 6 '17 at 14:22
• The only way I know that will give you a specific subset of 4 that fits your criteria, while being sure there are no subsets of 1, 2, or 3 that would fit your criteria is to do an exhaustive search of every single, double, and triple in the dataset. I would guess that there might be heuristics that help you be reasonably sure that there are no subsets of 1, 2, or 3 that fit your criteria. I would be surprised if there are any that let you be 100% sure – Bryan Schwimmer Sep 7 '17 at 15:37