Questions tagged [regression]
Techniques for analyzing the relationship between one (or more) "dependent" variables and "independent" variables.
9,548
questions with no upvoted or accepted answers
16
votes
0
answers
4k
views
Gamma hurdle model for continuous response?
I am modelling invertebrate.biomass ~ habitat.type * calendar.day + habitat.type * calendar.day ^ 2, with a random intercept of ...
- 261
15
votes
0
answers
501
views
Asymptotic property of tuning parameter in penalized regression
I'm currently working on asymptotic properties of penalized regression. I've read a myriad of papers by now, but there is an essential issue that I cannot get my head around.
To keep things simple, I'...
- 12.2k
14
votes
3
answers
654
views
Adjustments to (Linear Regression) Forecast
Full disclosure: I am not a statistician, nor do I claim to be one. I am a lowly IT administrator. Please play gentle with me. :)
I am responsible for collecting and forecasting disk storage use ...
- 241
14
votes
0
answers
11k
views
Time series regression with overlapping data
I am seeing a regression model which is regressing Year-on-Year stock index returns on lagged (12 months) Year-on-Year returns of the same stock index, credit spread (difference between monthly mean ...
- 308
11
votes
0
answers
270
views
Is autocorrelation not worth addressing with small N?
Consider a simple regression context in which there is a small set of response values, $Y$, and corresponding dates, $X$. (For simplicity, we can assume the dates are equally spaced.) We would like ...
11
votes
1
answer
524
views
Interpreting regression coefficients based on Andrew Gelman's re-scaling method
I have two predictors in a binary logistic regression model: One binary and one continuous. My primary goal is to compare the coefficients of the two predictors within the same model.
I have come ...
- 383
10
votes
0
answers
153
views
Logistic regression for classification: are there any analytical solutions for the out-of-sample accuracy?
I run a binary logistic regression, with a binary dependent variable and a continuous independent one.
Now I want to evaluate the out-of-sample performance of the classification algorithm so obtained. ...
- 1,200
10
votes
0
answers
156
views
Pope effect on pizza - Regression with presence absence and similarity data as dependent variables
I'm trying to figure out the right way to set up a regression when the dependent variables are presence absence data (of pizzas), and the similarity between the present pizzas. Bear with the story:
...
- 241
10
votes
0
answers
513
views
Bootstrap Prediction Interval: which residuals to use and which method?
I ask this question referring to the post: Bootstrap prediction interval, where a step by step method for calculating the prediction interval for linear regression models is explained.
In the ...
- 181
10
votes
0
answers
606
views
Does there exist an analogous statement to BLUE (Gauss-Markov) for GLMs?
I recall from my graduate school days that the Gauss-Markov (GM) theorem states that the Best Linear Unbiased Estimator (BLUE) in a linear regression is $\vec{\beta}=(X^TX)^{-1}X^T\vec{y}$. An amazing ...
- 3,969
10
votes
2
answers
3k
views
does serial correlation have something to do with endogeneity?
I'm a beginner of econometrics, and I've construed that endogeneity is caused by omitted variable bias, measurement error, and reverse causality, and it makes OLS estimator be biased.
And also I've ...
- 419
10
votes
1
answer
10k
views
Regress residuals in second regression
I am wondering if anyone can point me to a paper/lecture notes on the rationale behind first running an OLS on a set of variables, and then in a second regression using the residuals of that ...
- 173
9
votes
0
answers
10k
views
Calculating R-squared using standard errors
I have the following estimated model: $\hat{y} = 0.2857 + 0.8019x_1 - 0.0741x_2$ (the $t$-statistics are $1.8959$, $8.4198$, and $-3.7017$, respectively).
Furthermore, I know the sample size $N = 92$,...
- 223
9
votes
0
answers
2k
views
Is the OLS estimator the UMVUE (assuming Normality)?
Suppose
$$
\mathbf{y} = \mathbf{X} \mathbf{b} + \mathbf{e} \, ,
\\
\mathbf{e} \sim \mathcal{N}(0,\mathbf{I}_P) \, .
$$
We know that $\mathbf{\hat{b}} = (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \...
- 811
9
votes
0
answers
206
views
Efficient nonparametric estimation of confidence intervals and p-values for nonlinear regression
I'm estimating parameters for a complex, "implicit" nonlinear model $f(\mathbf{x}, \boldsymbol{\theta})$. It's "implicit" in the sense that I don't have an explicit formula for $f$: its value is the ...
- 16.1k
9
votes
0
answers
468
views
How do sufficiency statistics help in the interpretation of regression results?
One of the results why canonical link functions are widely used in GLMs is the existence of sufficiency statistics for the regression parameters, which in turn allow for:
... minimal
sufficient ...
- 3,858
9
votes
1
answer
4k
views
Using Fieller's theorem to calculate the confidence interval of a ratio (paired measurements)
If you have two means (with their own confidence intervals) and want to represent them as a ratio, how do calculate the confidence interval for the ratio?
An answer that was given to me, mentions ...
- 91
8
votes
0
answers
1k
views
Are SHAP values potentially misleading when predictors are highly correlated?
Are SHAP (SHapley Additive exPlanations) values potentially misleading when predictors are highly correlated? How and why? If so, is there any guidance on when not to use SHAP? Are there any rules ...
- 3,820
8
votes
0
answers
387
views
When using L2 regularization outside of linear regression, do the same MAP estimation assumptions hold?
Some context is shared below, and my question is bolded at the end.
In the linear regression setting, we learn model weights $\hat{\mathbf{w}}$ to make predictions $\mathbf{\hat{y}}$ from new samples ...
- 570
8
votes
0
answers
697
views
Assumptions for PCR and PLS
I am writing up a report on fitting Principal Component Regression (PCR) and Partial Least Squares (PLS) to my data-set.
A similar question: Model assumptions of partial least squares (PLS) ...
- 291
8
votes
2
answers
463
views
Logistic Regression with (Normal) Distributions for Independent Variables
Consider the logistic regression where $Y_i \in {0,1}$ are dependent variable observations and $X_i \in \mathbb{R}$ are the independent variables.
However we do not observe the $X_i$ themselves. ...
- 622
8
votes
0
answers
978
views
Why would I use ratio estimation instead of regression estimation to estimate means?
I am taking a graduate course on survey data analysis. I was recently introduced to ratio estimation and regression estimation.
I understand that using ratio estimator may be easier if we are ...
- 186
8
votes
0
answers
3k
views
Assumptions of correlation test vs regression slope test (significance testing)
If my understanding is correct, then
the test on a regression slope in a simple bivariate regression - i.e. the test of $\mathcal{H}_0$: $b = 0$ in $Y' = a + bX$
and
the test of a correlation, i.e. $...
- 331
8
votes
0
answers
2k
views
Fixed Effects vs Lagged DV vs. First Differences Regression
What are the differences between using unit fixed effects, unit fixed effects and time fixed effects, lagged DV, or first differences to analyze a time series with 4-5 time periods and 35-50 units per ...
- 390
8
votes
1
answer
1k
views
Calculating and plotting confidence interval for Theil-Sen estimator
I'm using Wilcox's R functions (specifically, regplot) to plot a Theil-Sen estimator with a single predictor.
However, regplot ...
- 81
8
votes
0
answers
916
views
Dantzig Selector, LASSO, LAD LASSO
I am wondering about this. When is it best to use Dantzig Selector (the infinity normed error measure plus the L1 regularizer) , the LASSO (the mean square error measure plus the L1 regularizer), and ...
- 391
8
votes
0
answers
5k
views
Poisson regression for binary data
I've been trying to read up on Poisson regression models, and it looks like it is possible to estimate such a model with a binary outcome. This has come up before on this site here (and somewhat here ...
- 31.5k
7
votes
0
answers
100
views
Identifying non-linearities in relationship between variables
Logistic regression is often used to identify the effect of $x$ on a binary variable $y$ after adjusting for potential confounders $x_1,...,x_n$. In the medical literature, I will sometimes encounter ...
- 440
7
votes
0
answers
316
views
Robust Gamma Regression
I am modeling some spectroscopic data where the response of the instrument to the size of the input is strictly positive and non-linear. Gamma regression seems like a good choice to explain the data, ...
- 223
7
votes
0
answers
219
views
Regression with dependent data with low dependence
Suppose you have data that is grouped in one way or another and therefore the assumption of independence is suspect. But you look at the intraclass correlation (or autocorrelation) and it is very ...
- 95k
7
votes
1
answer
259
views
Intuition: What is the difference between linear factor models and regular linear regression?
So, I have a very vexing theoretical question that I hope some experienced econometrician can help me with. Being in finance, I have recently been exposed to linear factor models, which are models ...
- 642
7
votes
0
answers
956
views
Geometrical interpretation of L1 regression
I have found the following image (or a similar version) in a lot of books related to penalized linear models. I get the insight of this image. The ellipsoids are the solution of the linear regression ...
- 1,234
7
votes
0
answers
238
views
Is there any geometric intuition on least absolute deviation regression?
There are a lot of geometric intuitions for regression with least square, e.g., projection, orthogonal, etc. (This and this answers are good examples.)
Is there similar geometric intuition for least ...
- 33.7k
7
votes
0
answers
701
views
Is lasso always outperformed by adaptive lasso?
I have been reading some papers and I understood that adaptive lasso has the Oracle properties which lasso lacks. Does that mean adaptive lasso always better than lasso (let's focus on the simple ...
- 81
7
votes
1
answer
3k
views
T-test for regression coefficients obtained from Ridge, LASSO etc
In ordinary least squares, for example in an experimental design case, I obtain the regression coefficents by:
$ \hat B = {({X^t}{X})}^{-1}X^ty$
Then, my null hypothesis for each coefficent is:
$...
- 1,432
7
votes
2
answers
852
views
Linear model with hidden variable
I have come across a somewhat unusual (I think) estimation problem. I have two "coupled" linear regression models,
$$Y = a + b x + \epsilon, \quad Z = c + d x + \nu$$
where $Y,Z,\epsilon,\nu$ are ...
- 181
7
votes
0
answers
80
views
Can I use bootstrap results at the observation level?
I have read quite a bit of bootstrapping, but the issue I want to address seem not to appear.
Consider a simple regression model:
$$ y_{i} = \beta_{0} + \beta_{1}x_{i} + e_{i}$$
I am aware that ...
- 2,549
7
votes
0
answers
685
views
Interpretation of smoothing spline
This question is about interpreting the results from non-linear regression models, especially when using regression splines. The numerical output is not very informative when interpreting the effects, ...
- 2,690
7
votes
0
answers
865
views
Can I calculate Cohen's $d$ from multiple regression coefficient?
Question: Is it appropriate to calculate Cohen's $d$ (effect size) from the regression coefficient of an independent categorical variable?
Background: My regression coefficient represents ...
- 71
7
votes
0
answers
1k
views
What techniques are there to measure goodness of fit of Deming (orthogonal) regression?
Questions:
Even if there is no "widely accepted" technique, is there a useful-and-above-average technique for estimating goodness of fit in orthogonal regressions?
What are the pros/cons of this ...
- 113
7
votes
0
answers
819
views
Rule of thumb for excluded variable in Heckman selection model?
I'm working on a project that involves the use of a Heckman selection model (more specifically a Roy or move-stay model, which is essentially a two-sided Heckman) of the following form:
$$ Y_{i1} = ...
- 226
7
votes
1
answer
390
views
minimizer weighted linear regression
In a regression problem, with $y=X\theta+\epsilon$ and $X$ is an $n$ by $p$ matrix
the ‘weighted least squares estimate is the minimizer $\theta^{*}$ of $f(\theta)=\sum_{i=1}^{n}\omega_{i}(y_i-x_i^{'}\...
- 171
7
votes
0
answers
1k
views
Bias Variance tradeoff from a Bayesian perspective
I know the general question about bias variance has been asked before. I understand the frequentist approach and the concept of model selection and the impact of bias and variance on "accuracy" of a ...
- 131
7
votes
0
answers
143
views
R packages that work with biased samples
I'm working with a biased sample of web users. I'm only able to track responses of users who have navigated my site in a certain way, and I'd like to run an analysis to determine how certain factors (...
- 605
7
votes
0
answers
2k
views
Standardizing count variables in panel data with overdispersion - R or Stata
I'm running a regression where the dependent (response) variable is a highly dispersed (slightly zero-inflated) count and the explanatory (independent or predictor) variables are continuous, counts as ...
- 505
7
votes
0
answers
1k
views
Logistic regression and maximum entropy
I have read (e.g. here) that a (multinomial) logistic regressor corresponds to a maximum entropy classifier.
My question is, how does one end up with the formula for logistic regression starting with ...
- 3,588
7
votes
1
answer
2k
views
Geometric Interpretation of Softmax Regression
I'm writing a series of blog posts on the basics of machine learning, just for fun, mostly to validate my understanding of Andrew Ng's class. As I'm currently studying generalized linear models (GLMs),...
- 613
7
votes
0
answers
270
views
Generalization of cumulative probability models for ordinal Y
There are many models in existence for ordinal $Y$, for example the proportional odds ordinal logistic model, the continuation ratio model, and the cumulative probit model. The first and third of ...
- 76.8k
7
votes
0
answers
111
views
Zero values and discontinuity in explanatory variable
One of my independent variables measures worker productivity through the variable $\frac{\log{sales}}{\text{# of workers}}$, and I'm creating one variable for skilled and another for unskilled workers....
- 104
7
votes
0
answers
720
views
Studentized residuals and goodness-of-fit with robust linear regression
Could you please advise whether studentized residuals are meaningful when computed on a robust linear regression model using an M-estimator?
I'd like to use it to detect outliers by doing something ...
- 783