I have a way of calculating sensitivity of a regression that is very useful for my particular domain, but I don't know what it is called. I would like a name for it so I can look up additional information.

I do a regression over some data to find the optimal parameters of the model. I then pick particular values for the independent variables and determined the predicted value for the dependent variables to synthesize a new data point. I then determine how much the predicted value changes as the position of this data point changes when re-optimizing the model parameters as the data point is perturbed. This is represented as a ratio that ranges from 0 to 1.

This creates a measure of how resistant the model parameters are to new data that is measured at a particular point. If the ratio of predicted value change to data point change is one, then the current data is offering no resistance at that point, and it would clearly be better to have more measurements for that independent value. As the ratio becomes lower, we know that our current solution is more stable for that independent value. If the ratio is zero, then the results are perfectly stable at that point. Adding more measurements there would have no effect.

This seems like a very useful measure in general, so I'm sure it's been studied, but I haven't been able to find anything that matches this exactly. It seems related to sensitivity analysis, stability, and robustness, but these seem to be global measures and are related to impact on the parameters rather than impact on the optimal residuals at one particular point. It isn't really a measure of goodness-of-fit either, since I'm not trying to validate my model.

UPDATE: Thanks to @RyanVolpi, I realize that this is closely related to the concept of leverage. The only difference being that I'm determining the leverage of synthesized data points rather than observations.

  • 1
    $\begingroup$ I don't quite understand the process you describe in the second paragraph. Is it possible you're thinking of leverage? Leverage "...describes the degree by which the i-th measured value influences the i-th fitted value." en.wikipedia.org/wiki/Leverage_(statistics)#Partial_leverage $\endgroup$ – Ryan Volpi Jul 15 '20 at 3:21
  • $\begingroup$ @RyanVolpi: That does sound like what I was looking for! The only difference is that I'm synthesizing a data point using the model, and then computing its leverage, thereby giving the leverage of a particular independent value rather than the leverage of an actual observation. Can you tell me how I might improve the question? I'm sure I'm being sloppy with my terminology. $\endgroup$ – Vaughn Cato Jul 15 '20 at 5:36
  • $\begingroup$ Your procedure is at odds with the implicit assumptions of many regression analyses. In particular, when using least squares methods, the ratio of the change conflicts with the mean square loss used by the regression procedure. That would be one reason you will have difficulty finding your procedure in the literature. $\endgroup$ – whuber Jul 15 '20 at 12:46
  • $\begingroup$ @whuber: Thanks. My particular use case is trying to evaluate the quality of the measurements based on the number of measurements made and where the measurements are made. Using these leverage values allows me to ask "is there a place where a new measurement could be made that would have a significant impact on the model predicted values?" If the answer is no, then I can say that the existing measurements are good enough. Is my procedure appropriate for this? $\endgroup$ – Vaughn Cato Jul 15 '20 at 15:39
  • 1
    $\begingroup$ I think the underlying idea is a good one. There might be a disconnect, though, between the procedure used to fit the model and how you are quantifying "significant impact," so I just wanted you to be aware of that. $\endgroup$ – whuber Jul 15 '20 at 16:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.