Skip to main content

Questions tagged [leverage]

Leverage is a measure used in regression to highlight observations which are outlying in the space of the predictors.

Filter by
Sorted by
Tagged with
0 votes
0 answers
7 views

Leverage and One-Out Model

I am currently trying to prove the following theorem, found within the course material from a Graduate Regression module. For context, here $X$ is the usual full-rank design matrix, estimators are ...
CarloPatti's user avatar
5 votes
0 answers
56 views

Effect of leverage on residuals

The variance of residuals in a linear regression is given by : $$Var(e_i)=(1-h_{ii})\sigma^2$$ This means that residuals have a lower variance than the error terms, and the variance of residuals is ...
Kilkik's user avatar
  • 355
2 votes
2 answers
199 views

Leverage and residuals - Show $\frac{e_i^2}{\| (I - H)Y \|^2} \le 1 - h_{ii}$ where $e_i$ is the $i$-th residual and $h_{ii}$ is leverage

Question: Suppose that $\boldsymbol{Y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{\epsilon}$, and the errors have zero mean, and are uncorrelated with constant variance. Let $\hat{\boldsymbol{\...
Balkys's user avatar
  • 145
3 votes
0 answers
79 views

Understanding the ridge leverage scores sampling from an arXiv paper

I give a try to read the arXiv paper Distributed Adaptive Sampling for Kernel Matrix Approximation, Calandriello et al. 2017. I got a code implementation where they compute ridge leverage scores ...
emonhossain's user avatar
0 votes
0 answers
22 views

a question about leverage and variance of the predictor variable [duplicate]

In a simple linear regression: Assume that the leverage of a specific data point equals 1 (maximum leverage), from my understanding, it also means that the variance of the estimated error (e) of that ...
Alex Il's user avatar
  • 45
2 votes
0 answers
36 views

How do I go about rectifying violated assumptions when more than one is violated at the same time? [duplicate]

I am currently trying to run a model analyzing the duration of the egg stage of each sex of two species of insect across five different temperatures. All independent variables are categorical. My ...
Insect_biologist's user avatar
5 votes
2 answers
445 views

Prove that the leverage converges to 0. Do the residuals in a linear regression really approximate the errors?

In linear regression, $y_i = x_i^T \beta + \epsilon_i$, $i=1,\dots,n$, and $Var(\epsilon_i)=\sigma^2$. It is well known that the residuals $e_i$ have variance $Var(e_i) = \sigma^2 (1-h_{ii})$, where $...
det's user avatar
  • 263
0 votes
1 answer
567 views

Proof that leverage points are between 0 and 1 inclusive

I would really appreciate it if anyone can guide me through this. I have a $n \times (p+1)$ matrix $X$. The projection matrix $P = X(X'X)^{-1}X'$. I want to prove that $P(i,i)$ is in $[0,1]$, where $P(...
Tahmid Mahmud's user avatar
2 votes
1 answer
160 views

Do leverage values make sense outside of linear models?

I've been looking at leverage plots, but it seems to me that they are always related to linear regression models. For instance, this explanation of a hat matrix considers a linear regression model: ...
Rafael L's user avatar
1 vote
0 answers
98 views

Help connecting MSE and residual variance $(=σ^2/(1-h_{ii}$)) in linear regression

In linear regression, we assume errors are normally distributed with mean 0 and variance $σ^2$. We use MSE = (Residual sum of squares) / (n-p) as the estimator for σ, where n= number of observations ...
tandeg's user avatar
  • 11
2 votes
1 answer
267 views

Who originally defined leverage scores to be the diagonal elements of $X(X^TX)^{-1}X^T$?

A nice description of leverage in the sense that I am using it is given here so I will not repeat it. Who originally defined leverage scores to be the diagonal elements of $X(X^TX)^{-1}X^T$?
Galen's user avatar
  • 8,784
1 vote
0 answers
933 views

Identification High Leverage for Logistic Regression

As seen below from Piet De Jong, Generalized Linear Models for Insurance Data, for linear model, we can identify high leverage, if the value of leverage>2p/n (or hii > 2p/n) then the indicates ...
Jasmine Helen's user avatar
1 vote
0 answers
81 views

Mistake about Covariance on Wikipedia Page for Leverage?

Wikipedia's page for leverage defines the design matrix $X \in \mathbb{R}^{n \times p}$ where $n$ is the number of data and $p$ is the dimensionality of each datum, and then, under "Relation to ...
Rylan Schaeffer's user avatar
1 vote
1 answer
411 views

Developing Leverage Statistics Manually in R

I've asked this question on Stack Overflow but think it might be better here. I currently have the below data frame df with the regression equation ...
AW27's user avatar
  • 243
0 votes
0 answers
649 views

Cook's distance, not plotting an observation with leverage one. What is this?

I am doing some regression diagnostics in R. I use plot() function and look at the four graphs. However, when I reach Cook's distance graph, I receive a warning ...
Ken Lee's user avatar
  • 351
2 votes
0 answers
224 views

Selecting a subset of observations that retains the most information

Is there a "best" way to select a subset of data (observations, not variables) to use in a multiple regression? The problem: A colleague of mine is planning a project for this field season. ...
Matt Tyers's user avatar
2 votes
2 answers
709 views

Logistic regression with rare events, all events have large residuals and are influential points

I am running a logistic regression model where the outcome is relatively rare (250 out of 5000). My main interest is to see if there are differences between age groups, sex, educational levels, income ...
Barrett's user avatar
  • 151
3 votes
1 answer
72 views

Why (or when) can we neglect residuals mutual correlation?

I've found in several books so far only a waving of hands explanation on why one can simply neglect the residuals mutual correlation, in a linear model context, and plot a QQ-graph to do a qualitative ...
An old man in the sea.'s user avatar
3 votes
0 answers
2k views

Understanding Residuals vs Leverage plot in terms of meeting regression assumptions

Can someone help me understand the Residuals vs Leverage plot in terms of meeting the assumption of independence/influence for multiple linear regression models? My understanding is that the ...
Rachel's user avatar
  • 31
0 votes
0 answers
194 views

PRESS from the hat matrix and numerical stability from statsmodels ols.fit()

Leave one out cross validation in the context of ordinary least squares regression can be done via the hat matrix: The "hat" or projection matrix $$ H = X(X^T X)^{-1} X^T $$ many fit ...
Moritz's user avatar
  • 131
0 votes
1 answer
83 views

Why does high leverage point in SLR have unusual covariate values

As stated above and also wondering what a covariate value means, is it just covariance? IF not, then what it is and what is the difference between covariance and covariate
TONGFEI ZHOU's user avatar
1 vote
0 answers
388 views

How to decide between different robust standard errors?

Specifying my model I ran into some very mild heteroscedasticity problems. Given its superior small-sample properties (my dataset contains 79 observations) I used the HC3 specification of the White ...
philipp.kn_98's user avatar
1 vote
0 answers
168 views

Interpreting results from glm.diag.plots (package 'boot')

I'm having hard time interpreting results from glm.diag.plots (package 'boot'). I read through some questions and answers (this one was very helpful, thank you! Interpreting glm.diag.plots) but didn't ...
user295393's user avatar
2 votes
1 answer
578 views

leverage() diagnostic test not supported for glmmTMB models in r

I am using a glmmTMB to look at the effect of numerous variables on how far individuals travel (distance). The example below is ...
Blundering Ecologist's user avatar
0 votes
0 answers
27 views

N size for not being considerated leverage

Suposse I have a dataset with two variables x and y, with the purpose to run a linear regression y ~ x. We have all x values equal and y varying between 1 and 10. For example (in R code): ...
flopeko's user avatar
  • 16
0 votes
1 answer
436 views

How to check for outliers and leverage values in SPSS when conducting a multinomial logistic regression?

I am conducting a multinomial logistic regression in SPSS. I want to check for the presence of outliers and high leverage values. How do I do that in SPSS?
Loulou's user avatar
  • 1
0 votes
0 answers
42 views

Relation between dyi/dei and its leverage (hii)

Althought I have tried in different ways, I have not been able to show that $$\frac{\mathrm{d}Y_i}{\mathrm{d}e_i} = \frac{1}{1 - l_{ii}}$$ $e_i$ is equal to $l_{ii}$ is the $i-th$ element of the ...
Brian González's user avatar
2 votes
1 answer
726 views

Simple Linear Regression: Hat-Value $h_i$

I'm trying to finish proving that in simple-regression analysis, $h_i = \frac{1}{n} + \frac{(X_i - \bar{X})^2}{\sum_{j=1}^{n}(X_j - \bar{X})^2}$, where $h_i := h_{ii} = \sum_{j=1}^nh_{ij}^2$, the ...
Jake's user avatar
  • 199
3 votes
1 answer
591 views

Individual significance of data points in correlation

In this question on stackoverflow, I asked about how it is possible to find the individual significance of each correlation coefficient of each node. I answered the question myself later stating that ...
Ahmed Al-haddad's user avatar
4 votes
1 answer
5k views

Proof about the diagonal element of the hat matrix

I'd really appreciate it if you could help me find the proof for the following formula: $$h_{ii}=1/n + \frac{(x_{i}-\bar{x})^2}{\sum(x_{j}-\bar{x})^2},$$ where $j=1,\ldots,n$. I don't really know ...
thenac's user avatar
  • 341
2 votes
0 answers
303 views

Outliers and influential observations in elastic net logistic regression

My dataset has many biomarkers and the boxplots of these variables show the presence of many outliers. However, these 'outliers' are real data and not misread observations. I want to use elastic net ...
Blain Waan's user avatar
  • 3,595
6 votes
1 answer
163 views

Bounding residual variance with distance from mean

For a linear regression $Y = X\beta + \varepsilon$ with $\varepsilon \sim \mathcal N(0,\sigma^2 I)$, we have $\hat Y = H Y$ for $H = X(X^TX)^{-1}X^T$. This means that $Var(Y - \hat Y) = \sigma^2(I-H)$ ...
alfalfa's user avatar
  • 641
1 vote
1 answer
1k views

Leverage and Influence

Is it possible that an outlier is neither influential nor does it have high leverage? Or can it happen that an observation with high leverage is not an outlier and is neither influential?
delete's user avatar
  • 91
0 votes
0 answers
58 views

How outliers influence your results? and what are good and bad leverage points? [duplicate]

I am confused between outliers and leverage points. And the difference between good and bad leverage points in time series analysis. Can somebody help me?
Abdu R Rahman's user avatar
1 vote
1 answer
200 views

Numerically Distinguish Between Real Correlation and Artifact

I'm looking at correlation for a large number of vectors, and many (about 3000) of these pairwise comparisons appear to have a significant correlation even after Bonferroni correction. Plotting these ...
Empiromancer's user avatar
2 votes
1 answer
1k views

Why divide by 1-leverage?

I'm reading about resampling methods, and specifically leave-one-out cross-validation. I understood the method, and how to calculate the estimate of the test MSE (Mean squared error): In the setup ...
YsfEss's user avatar
  • 245
2 votes
0 answers
35 views

Name for spurious linear Regression Plots [duplicate]

Yesterday I was at a medical conference in which a lot of plots of Point Clouds with linear fits were shown. In many cases the fit seemed (at least to me and colleagues) to be influenced mostly by ...
geo's user avatar
  • 362
1 vote
1 answer
1k views

Why uncentered hat matrix can be used to measure the distance from the center of data?

This question is motivated from here :Why leverage measure the distance of the ith observation from the center of the x space? which is the question related to the wonderful answer of this link :...
KDG's user avatar
  • 853
7 votes
2 answers
1k views

Identifying outliers in the data

Sample data ...
89_Simple's user avatar
  • 981
2 votes
1 answer
3k views

Influential observations and Outliers in Linear Regression model

What is the difference between influential observations and outliers in linear regression model?
Monisha Damodaran's user avatar
3 votes
1 answer
353 views

Is there a specific standard error for an individual residual?

On page 97 of Introduction to Statistical Learning book, there is a paragraph on studentized residuals within the context of looking for outliers. But in practice, it can be difficult to decide how ...
Doug Fir's user avatar
  • 1,568
0 votes
0 answers
309 views

News Impact curve shift

I computed the NIC for both vanilla GARCH and EGARCH model specifications (with t(4)-student distributed innovations). As we can see from the plot, no asymmetries are present, but the curve has ...
toyo10's user avatar
  • 91
0 votes
0 answers
136 views

How to solve a well fitted model - Model Misspecification

I am currently writing a paper, analysing the impact of goldprice movements on the capital structure of gold mining firms. My basic model is a simple OLS model with (y=leverage and x=ln(goldprice)). ...
Knut E.'s user avatar
1 vote
0 answers
267 views

Sensitivity Measures for GEE Model

Is there any method (e.g. like Cooks D) implemented in R to identify leverage points for GEE Models? I used geepack to fit my models and would like to do a sensitivity analysis now. However, I don't ...
geenieinabottle's user avatar
3 votes
1 answer
2k views

Leverage formula/derivation and Hat matrix

1) So I know that $h_{ii}$ is just the ith row ith column of $H=X(X^TX)^{-1}X^T$. Intuitively, why is this the case? I understand that H is the projection matrix and leverage is measuring how far away ...
Brian's user avatar
  • 31
8 votes
2 answers
19k views

Cook's distance vs. hat values

What exactly does Cook's distance measure? And how is this different from what hat values measure? I know hat values measure how distant a point it form its corresponding fitted point. I also know ...
Sara's user avatar
  • 129
4 votes
1 answer
2k views

Leverage Statistic - equation explanation

I came across this equation for the leverage statistic in ISL. Does the denominator translate to the summation of all the squares of $(x_i - mean(x))^2$ excluding the current observation of x? In ...
m_squared's user avatar
  • 311
6 votes
1 answer
1k views

What are the leverage values for Ridge regression?

In linear least squares the parameter estimates are: $\hat{\beta} = \left(X^{\top}X\right)^{-1}X^{\top}y$. In Ridge regression the standardized parameter estimates are given by $\hat{\beta}_{\Gamma} = ...
José Bayoán Santiago Calderón's user avatar
0 votes
0 answers
149 views

which residual vs fitted plot is looking better here?

This is a data analysis project we have at school where we try different ways to correct a simple linear regression model. My first attempt is just trying to transform the response variable by taking ...
Tienanh Nguyen's user avatar
1 vote
0 answers
85 views

How to Adapt Leverage vs Residual plot for Multi-variate Regression

After reading about What is the best way to identify outliers in multivariate data? I noticed that some heuristics are hard to adapt to mult-variate analysis. I'm hoping there is a way to at least ...
Arash Howaida's user avatar