1
$\begingroup$

I have a normally distributed continuous variable referring to an observed human behavior, and I'm interested in measuring or rather analyzing the extreme of this behavior, namely, the top 10% of the distribution as displayed in this graph. So I went ahead and replaced all the values below the 90th percentile with the value corresponding to that percentile (cutoff). This new variable is now exponentially-distributed and left-censored with a huge pile of data on the left. My goal is to conduct a multiple regression analysis with the censored variable as the DV.

Clarification: The reason why I replaced values with a cutoff is that there is no variation below the cutoff that is meaningful for the purposes of measuring the extreme behavior. For example, if the range of values is 1 to 40, and the cutoff is 20, I'm assuming that any value below 20 is not meaningfully different from 20 (and I have good reason to assume this).

Question: Someone suggested that I use tobit regression, but from what I've read tobit regression assumes that the censored data is normally distributed, whereas, in this case, I am no longer interested in the original variable--the new variable of interest represents an extreme behavior that is not normally distributed. If I am correct that tobit regression would not be appropriate, what would be an appropriate regression method to use?

Improve this question
$\endgroup$
3
  • $\begingroup$ You might be mistaken in supposing "this variable is now exponentially-distributed." If indeed the original variable was Normally distributed, then it's mathematically impossible for any part of the tail to have an exponential distribution, even though it might look like that in your censored dataset. There's a fundamental problem with using your censored variable as a DV, too: the regression will be strongly influenced by the 90% of data that you have completely screwed up! If you want to focus on the tail, then you will need to downweight or eliminate the rest, rather than changing them. $\endgroup$
    – whuber ♦
    Commented Jun 21, 2018 at 12:22
  • $\begingroup$ I agree; there are probably better approaches than to simply use a cutoff, and I would refocus the question on that if I could. Truncation is not an option as I'm interested in the variation between normal and extreme responses, but the new variable would ideally also vary in degrees of extremeness. Whichever is the best way to re-compute the variable, I'm certain that its lowest value will equal the median (most cases = normal) and as you move to the right (toward extreme values) the frequencies will decrease, hence it will look exponentially distributed, like insurance claims. $\endgroup$
    – AlexR
    Commented Jun 23, 2018 at 3:39
  • $\begingroup$ It sounds like you are using "exponentially distributed" in a loose, non-quantitative sense. There is a marked difference between the behavior of distributions with truly exponential tails and distributions with Normal-like tails, so it might be wise not to ignore the distinction. $\endgroup$
    – whuber ♦
    Commented Jun 23, 2018 at 11:46

1 Answer 1

Reset to default
0
$\begingroup$

Could you clarify what you did when you said you "replaced all the values below the 90th percentile with the value corresponding to that percentile?"

That part is not clear to me, for one. I presume you mean that you discretized the continuous values to the left of your cutoff. It's not obvious, according to econometric or statistical theory, why you should choose to replace the data to the left of that point in the first place.

Josh Angrist and Jorn-Steffen Pischke, in their book Mostly Harmless Econometrics, strongly advise against using censored data approaches when the data is not truly censored. For one, using approaches like Tobit require distributional assumptions, potential reductions in efficiency, and complicated functional forms to find marginal effects. See this discussion in Chapter 3, pages 94-102. There are other censored and zero-inflated approaches which do not require normality but still face the same criticisms I referenced above.

Hence, the recommendation is to use linear-regression and pay particular attention to the structure of your standard error.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I've added a clarification to explain. The data is not truly censored if we're talking about the original variable, but the newly created variable is because it represents an extreme that is qualitatively different from the 'normal behavior' and it's unclear what the lower bound of this behavior is. If there are other ways to analyze the variation at the end of the distribution without taking the cutoff approach described, I would welcome suggestions. (I've ruled out linear regression, as the assumptions are not met.) $\endgroup$
    – AlexR
    Commented Jun 21, 2018 at 3:39
  • $\begingroup$ Why aren't the assumptions to OLS met? I'm hesitant to suggest a non-linear, censored model when the data truly aren't of that nature. It's not clear to me that this approach is correct and imposes a number of stringent conditions. If you could outline your model, that would be better so I can offer some more specific insight. It sounds like a Wald grouping estimator might be best for you. You could also do the simple trick of adding a simple interaction term to capture nonlinearity above your cutoff value. I also don't recommend deleting data as @ whuber suggests. That will kill efficiency. $\endgroup$
    – JuliusBilly
    Commented Jun 21, 2018 at 15:20
  • $\begingroup$ Very non-normal residuals and uneven variances. If I could go back, I would re-focus my question on what acceptable steps I could take to analyze the variation between extreme and non-extreme parts of the distribution. One obvious option is to dichotomize (1 = extreme; 0 = normal), but it's too lossy. Too late to change it--I might repost it under a different title. But feel free to comment. $\endgroup$
    – AlexR
    Commented Jun 23, 2018 at 3:26
  • $\begingroup$ @AlexR That doesn't sound like a violation of OLS. Modern regression requires just three assumptions: 1. $(X,Y)$ are from an IID process, 2. $E[u|X] = 0$, and 3. The fourth moments on the error terms and covariates are finite. If anything you have a violation of the second assumption. But it sounds more like you have heterogeneity, which you can fix with the appropriate standard (likely Huber-White). The other thing might be you are seeing a non-linear causal impact. Try including interaction effects or trying quantile regression. Be specific with the model of interest for future comments. $\endgroup$
    – JuliusBilly
    Commented Jun 23, 2018 at 17:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.