# Rare value prediction in Regression

Im working on a project which is to estimate blood pressure from independent variables. The problem I have is that the Blood Pressure data is gaussian in nature since most of the people are having normal Blood Pressure, but the main aim of the project is to get the accurate estimation of extreme values so that the algorithm can cater to the hypo and hypertensive people. From looking at similar questions I've found that the common approaches to this can be to- 1. Take the log transformation of Target variable 2. Upsample the extreme case values

Are there any other methods that can be used so that the regressor model can be more accurate in the extreme ends?

• Give a larger regression weight to the data at the extremes. The more distant from the mean value, the larger the regression weight. An example approach would be to weight by the absolute value of standard deviation, with a minimum weight of 1.0. Data at +/- two standard deviations would receive a weight of 2.0. Data at +/- three standard deviations would be weighted at 3.0. I have never tried this, it seems worth testing. – James Phillips Sep 2 '19 at 17:01
• @JamesPhillips's idea is a good start. Predicting extremes is hard. One problem is discussed here: Regression predictions show far less variance than expected. Also, how often are your participants measured? If someone is measured and shows an extreme value, an additional measurement will very likely show a much less extreme value, because of regression to the mean. – Stephan Kolassa Sep 2 '19 at 17:54

I think your question can be viewed in two ways:

(1) Your data contains people of all three types: hypo, hyper and none. One way (and probably wrong) is to remove 'none' type people and use logistic regression.

(2) You don't want to remove 'none' type data because given a set of independent variables, there is a distribution of blood pressures. This is more interesting. Say for a given $$X$$, you want your regression result to give hypo or hyper even if there is 10% chance (not that the mean blood pressure is in hypo or hyper range - which the normal regression does). In second approach three pointss to be noted now:

1. Be very sure that your data actually has a normal distribution. Whatever regression method you use, do check the residuals for normality. Particularly check the Kurtosis, or better use the JB test. Your answer would be highly sensitive if the tails are thicker/thinner.

2. You can use the usual regression provided that assumption of homoskedasticity is very well followed (along with normality assumption, and careful handling of outliers - somewhat related to the comment above). If this criteria is matched: Let your model be: $$E(Y|X)=X\beta$$

$$Pr(u can be computed based on the conditional distribution that you get from your model. How good this will be depends on your assumption of homoskedasticity and normality.

1. Rarely the above assumptions perfectly match. If so is the case explore Quantile regression. Instead of modelling the $$E(Y|X)$$, in quantile regression you model the quantile. For example, you can model:

$$Q_{Y|X}(\tau)=X\beta_\tau$$

So you can take high or very low values of $$\tau$$ Here the $$\beta_\tau$$ estimated would directly give you the estimate of blood pressure for a given $$X$$ at even 10% ($$\tau=0.9$$ for hyper) chance.

Hope what I am saying makes sense in your context.