Suppose I have a big dataset and I compute some statistical summary of it - e.g., the correlation of one dimension with another.
I think a reasonable question to ask would be "what data points explain this result" - e.g., perhaps, is it because there were two huge outliers that explain the whole correlation? Or are all the points approximately equally important?
Or, say, I'm computing the mean of a response variable, measured at two different values of a parameter. I find that the means are approximately the same. Why is it the same: is it because the distributions of the variable are the same, or is it because the mean in both cases is determined by a big cluster that determines the whole mean and obscures the differences in the rest of the distribution?
I guess, in general, I'm interested in computing sensitivity of a statistical summary (not necessarily a single number) w.r.t. the data points and parameters involved in the computation.
This notion of sensitivity would help me, on one hand, avoid meaningless results (which are entirely explained by abnormal measurements), and on the other hand, it would guide my further exploration.
So: is there a branch of statistics that studies this kind of sensitivity? If yes, what are some useful methods from that branch / what further reading would you recommend? E.g., I imagine, it might be useful for data visualization methods - not just draw the data, but colorize each point according to the sensitivity of some metric w.r.t. that point?
I tried googling for things like "explanatory statistics", "sensitivity of statistical summaries" etc., but did not find a lot. I found Uncertainty and sensitivity analysis but this is not quite what I'm looking for - I'm not interested in sensitivity of one variable of the dataset w.r.t. the other, I'm interested in making deeper answers to questions like "do these two variables correlate".
P.S. Some more googling yielded the keyword "input importance" and some data visualization methods, e.g. http://vis.cs.ucdavis.edu/papers/TVCG_Chan_GSS.pdf . But I crave for advice from experts :)