Questions tagged [leverage]
Leverage is a measure used in regression to highlight observations which are outlying in the space of the predictors.
89 questions
4
votes
0
answers
90
views
Effect of leverage on residuals
The variance of residuals in a linear regression is given by :
$$Var(e_i)=(1-h_{ii})\sigma^2$$
This means that residuals have a lower variance than the error terms, and the variance of residuals is ...
- 435
2
votes
2
answers
234
views
Leverage and residuals - Show $\frac{e_i^2}{\| (I - H)Y \|^2} \le 1 - h_{ii}$ where $e_i$ is the $i$-th residual and $h_{ii}$ is leverage
Question: Suppose that $\boldsymbol{Y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\epsilon}$, and the errors have zero mean, and are uncorrelated with constant variance. Let $\hat{\boldsymbol{\...
- 155
3
votes
0
answers
97
views
Understanding the ridge leverage scores sampling from an arXiv paper
I give a try to read the arXiv paper Distributed Adaptive Sampling for Kernel Matrix Approximation, Calandriello et al. 2017. I got a code implementation where they compute ridge leverage scores ...
- 205
0
votes
0
answers
23
views
a question about leverage and variance of the predictor variable [duplicate]
In a simple linear regression:
Assume that the leverage of a specific data point equals 1 (maximum leverage),
from my understanding, it also means that the variance of the estimated error (e) of that ...
- 45
2
votes
0
answers
36
views
How do I go about rectifying violated assumptions when more than one is violated at the same time? [duplicate]
I am currently trying to run a model analyzing the duration of the egg stage of each sex of two species of insect across five different temperatures. All independent variables are categorical. My ...
- 461
5
votes
2
answers
501
views
Prove that the leverage converges to 0. Do the residuals in a linear regression really approximate the errors?
In linear regression, $y_i = x_i^T \beta + \epsilon_i$, $i=1,\dots,n$, and $Var(\epsilon_i)=\sigma^2$. It is well known that the residuals $e_i$ have variance $Var(e_i) = \sigma^2 (1-h_{ii})$, where $...
- 263
0
votes
1
answer
718
views
Proof that leverage points are between 0 and 1 inclusive
I would really appreciate it if anyone can guide me through this.
I have a $n \times (p+1)$ matrix $X$. The projection matrix $P = X(X'X)^{-1}X'$. I want to prove that $P(i,i)$ is in $[0,1]$, where $P(...
2
votes
1
answer
198
views
Do leverage values make sense outside of linear models?
I've been looking at leverage plots, but it seems to me that they are always related to linear regression models.
For instance, this explanation of a hat matrix considers a linear regression model: ...
- 55
1
vote
0
answers
105
views
Help connecting MSE and residual variance $(=σ^2/(1-h_{ii}$)) in linear regression
In linear regression, we assume errors are normally distributed with mean 0 and variance $σ^2$.
We use MSE = (Residual sum of squares) / (n-p) as the estimator for σ, where n= number of observations ...
- 11
2
votes
1
answer
291
views
Who originally defined leverage scores to be the diagonal elements of $X(X^TX)^{-1}X^T$?
A nice description of leverage in the sense that I am using it is given here so I will not repeat it.
Who originally defined leverage scores to be the diagonal elements of $X(X^TX)^{-1}X^T$?
- 9,660
1
vote
0
answers
1k
views
Identification High Leverage for Logistic Regression
As seen below from Piet De Jong, Generalized Linear Models for Insurance Data, for linear model, we can identify high leverage, if the value of leverage>2p/n (or hii > 2p/n) then the indicates ...
1
vote
0
answers
84
views
Mistake about Covariance on Wikipedia Page for Leverage?
Wikipedia's page for leverage defines the design matrix $X \in \mathbb{R}^{n \times p}$ where $n$ is the number of data and $p$ is the dimensionality of each datum, and then, under "Relation to ...
- 1,057
1
vote
1
answer
482
views
Developing Leverage Statistics Manually in R
I've asked this question on Stack Overflow but think it might be better here.
I currently have the below data frame df with the regression equation
...
- 253
0
votes
0
answers
696
views
Cook's distance, not plotting an observation with leverage one. What is this?
I am doing some regression diagnostics in R.
I use plot() function and look at the four graphs. However, when I reach Cook's distance graph, I receive a warning ...
- 351
2
votes
0
answers
247
views
Selecting a subset of observations that retains the most information
Is there a "best" way to select a subset of data (observations, not variables) to use in a multiple regression?
The problem: A colleague of mine is planning a project for this field season. ...
- 284
2
votes
2
answers
857
views
Logistic regression with rare events, all events have large residuals and are influential points
I am running a logistic regression model where the outcome is relatively rare (250 out of 5000). My main interest is to see if there are differences between age groups, sex, educational levels, income ...
- 151
3
votes
1
answer
76
views
Why (or when) can we neglect residuals mutual correlation?
I've found in several books so far only a waving of hands explanation on why one can simply neglect the residuals mutual correlation, in a linear model context, and plot a QQ-graph to do a qualitative ...
- 5,882
3
votes
0
answers
2k
views
Understanding Residuals vs Leverage plot in terms of meeting regression assumptions
Can someone help me understand the Residuals vs Leverage plot in terms of meeting the assumption of independence/influence for multiple linear regression models?
My understanding is that the ...
- 31
0
votes
0
answers
208
views
PRESS from the hat matrix and numerical stability from statsmodels ols.fit()
Leave one out cross validation in the context of ordinary least squares regression can be done via the hat matrix:
The "hat" or projection matrix
$$
H = X(X^T X)^{-1} X^T
$$
many fit ...
- 131
0
votes
1
answer
85
views
Why does high leverage point in SLR have unusual covariate values
As stated above and also wondering what a covariate value means, is it just covariance? IF not, then what it is and what is the difference between covariance and covariate
1
vote
0
answers
427
views
How to decide between different robust standard errors?
Specifying my model I ran into some very mild heteroscedasticity problems. Given its superior small-sample properties (my dataset contains 79 observations) I used the HC3 specification of the White ...
- 101
1
vote
0
answers
187
views
Interpreting results from glm.diag.plots (package 'boot')
I'm having hard time interpreting results from glm.diag.plots (package 'boot').
I read through some questions and answers (this one was very helpful, thank you! Interpreting glm.diag.plots) but didn't ...
- 13
2
votes
1
answer
656
views
leverage() diagnostic test not supported for glmmTMB models in r
I am using a glmmTMB to look at the effect of numerous variables on how far individuals travel (distance).
The example below is ...
0
votes
0
answers
28
views
N size for not being considerated leverage
Suposse I have a dataset with two variables x and y, with the purpose to run a linear regression y ~ x.
We have all x values equal and y varying between 1 and 10.
For example (in R code):
...
- 16
0
votes
1
answer
461
views
How to check for outliers and leverage values in SPSS when conducting a multinomial logistic regression?
I am conducting a multinomial logistic regression in SPSS. I want to check for the presence of outliers and high leverage values. How do I do that in SPSS?
- 1
0
votes
0
answers
43
views
Relation between dyi/dei and its leverage (hii)
Althought I have tried in different ways, I have not been able to show that
$$\frac{\mathrm{d}Y_i}{\mathrm{d}e_i} = \frac{1}{1 - l_{ii}}$$
$e_i$ is equal to
$l_{ii}$ is the $i-th$ element of the ...
2
votes
1
answer
788
views
Simple Linear Regression: Hat-Value $h_i$
I'm trying to finish proving that in simple-regression analysis, $h_i = \frac{1}{n} + \frac{(X_i - \bar{X})^2}{\sum_{j=1}^{n}(X_j - \bar{X})^2}$, where $h_i := h_{ii} = \sum_{j=1}^nh_{ij}^2$, the ...
- 199
3
votes
1
answer
635
views
Individual significance of data points in correlation
In this question on stackoverflow, I asked about how it is possible to find the individual significance of each correlation coefficient of each node. I answered the question myself later stating that ...
- 133
4
votes
1
answer
5k
views
Proof about the diagonal element of the hat matrix
I'd really appreciate it if you could help me find the proof for the following formula:
$$h_{ii}=1/n + \frac{(x_{i}-\bar{x})^2}{\sum(x_{j}-\bar{x})^2},$$
where $j=1,\ldots,n$.
I don't really know ...
- 371
2
votes
0
answers
312
views
Outliers and influential observations in elastic net logistic regression
My dataset has many biomarkers and the boxplots of these variables show the presence of many outliers. However, these 'outliers' are real data and not misread observations. I want to use elastic net ...
- 3,625
6
votes
1
answer
175
views
Bounding residual variance with distance from mean
For a linear regression $Y = X\beta + \varepsilon$ with $\varepsilon \sim \mathcal N(0,\sigma^2 I)$, we have $\hat Y = H Y$ for $H = X(X^TX)^{-1}X^T$. This means that $Var(Y - \hat Y) = \sigma^2(I-H)$ ...
- 661
1
vote
1
answer
1k
views
Leverage and Influence
Is it possible that an outlier is neither influential nor does it have high leverage?
Or can it happen that an observation with high leverage is not an outlier and is neither influential?
- 91
0
votes
0
answers
58
views
How outliers influence your results? and what are good and bad leverage points? [duplicate]
I am confused between outliers and leverage points. And the difference between good and bad leverage points in time series analysis.
Can somebody help me?
1
vote
1
answer
217
views
Numerically Distinguish Between Real Correlation and Artifact
I'm looking at correlation for a large number of vectors, and many (about 3000) of these pairwise comparisons appear to have a significant correlation even after Bonferroni correction. Plotting these ...
- 241
2
votes
1
answer
1k
views
Why divide by 1-leverage?
I'm reading about resampling methods, and specifically leave-one-out cross-validation.
I understood the method, and how to calculate the estimate of the test MSE (Mean squared error):
In the setup ...
- 245
2
votes
0
answers
35
views
Name for spurious linear Regression Plots [duplicate]
Yesterday I was at a medical conference in which a lot of plots of Point Clouds with linear fits were shown. In many cases the fit seemed (at least to me and colleagues) to be influenced mostly by ...
- 372
1
vote
1
answer
1k
views
Why uncentered hat matrix can be used to measure the distance from the center of data?
This question is motivated from here :Why leverage measure the distance of the ith observation from the center of the x space?
which is the question related to the wonderful answer of this link :...
- 913
7
votes
2
answers
1k
views
2
votes
1
answer
3k
views
Influential observations and Outliers in Linear Regression model
What is the difference between influential observations and outliers in linear regression model?
3
votes
1
answer
369
views
Is there a specific standard error for an individual residual?
On page 97 of Introduction to Statistical Learning book, there is a paragraph on studentized residuals within the context of looking for outliers.
But in practice, it can be difficult to decide how ...
- 1,588
0
votes
0
answers
321
views
News Impact curve shift
I computed the NIC for both vanilla GARCH and EGARCH model specifications (with t(4)-student distributed innovations).
As we can see from the plot, no asymmetries are present, but the curve has ...
- 91
0
votes
0
answers
136
views
How to solve a well fitted model - Model Misspecification
I am currently writing a paper, analysing the impact of goldprice movements on the capital structure of gold mining firms. My basic model is a simple OLS model with (y=leverage and x=ln(goldprice)). ...
- 9
1
vote
0
answers
275
views
Sensitivity Measures for GEE Model
Is there any method (e.g. like Cooks D) implemented in R to identify leverage points for GEE Models? I used geepack to fit my models and would like to do a sensitivity analysis now. However, I don't ...
3
votes
1
answer
2k
views
Leverage formula/derivation and Hat matrix
1)
So I know that $h_{ii}$ is just the ith row ith column of $H=X(X^TX)^{-1}X^T$. Intuitively, why is this the case? I understand that H is the projection matrix and leverage is measuring how far away ...
- 31
8
votes
2
answers
20k
views
Cook's distance vs. hat values
What exactly does Cook's distance measure? And how is this different from what hat values measure?
I know hat values measure how distant a point it form its corresponding fitted point. I also know ...
- 129
4
votes
1
answer
2k
views
Leverage Statistic - equation explanation
I came across this equation for the leverage statistic in ISL. Does the denominator translate to the summation of all the squares of $(x_i - mean(x))^2$ excluding the current observation of x? In ...
- 341
6
votes
1
answer
1k
views
What are the leverage values for Ridge regression?
In linear least squares the parameter estimates are:
$\hat{\beta} = \left(X^{\top}X\right)^{-1}X^{\top}y$.
In Ridge regression the standardized parameter estimates are given by
$\hat{\beta}_{\Gamma} = ...
0
votes
0
answers
154
views
which residual vs fitted plot is looking better here?
This is a data analysis project we have at school where we try different ways to correct a simple linear regression model. My first attempt is just trying to transform the response variable by taking ...
- 461
1
vote
0
answers
89
views
How to Adapt Leverage vs Residual plot for Multi-variate Regression
After reading about What is the best way to identify outliers in multivariate data? I noticed that some heuristics are hard to adapt to mult-variate analysis. I'm hoping there is a way to at least ...
- 857
1
vote
0
answers
25
views
How to check if overall results from a survey are significantly affected by participation from a large company(ies)?
Suppose there is some sort of survey where people from various companies answer some questions.
Is it possible to determine whether overall results from the survey are significantly affected by ...
- 425