Questions tagged [regression]
Techniques for analyzing the relationship between one (or more) "dependent" variables and "independent" variables.
8,537
questions with no upvoted or accepted answers
25
votes
0answers
1k views
How does a Relevance Vector Machine (RVM) work?
Relevance Vector Machines (RVMs) are really interesting models when contrasted with the highly geometrical (and popular) SVMs.
In the light of a question like How does a Support Vector Machine (SVM) ...
13
votes
0answers
3k views
Gamma hurdle model for continuous response
I am modelling invertebrate.biomass ~ habitat.type * calendar.day + habitat.type * calendar.day ^ 2, with a random intercept of transect.id (50 transects were repeated 5 times)
My response is zero-...
13
votes
0answers
436 views
Asymptotic property of tuning parameter in penalized regression
I'm currently working on asymptotic properties of penalized regression. I've read a myriad of papers by now, but there is an essential issue that I cannot get my head around.
To keep things simple, I'...
12
votes
1answer
9k views
Cause of a high condition number in a python statsmodels regression?
I'm pretty new to regression analysis, and I'm using python's statsmodels to look at the relationship between GDP/health/social services spending and health outcomes (DALYs) across the OECD. Just to ...
12
votes
2answers
536 views
Adjustments to (Linear Regression) Forecast
Full disclosure: I am not a statistician, nor do I claim to be one. I am a lowly IT administrator. Please play gentle with me. :)
I am responsible for collecting and forecasting disk storage use ...
11
votes
0answers
6k views
“Zero-inflated” predictors in regression?
I know that zero-inflated models (e.g. zero-inflated Poisson or negative binomial models) can be used for dependent variables. I also know that in general there are no assumptions for the independent ...
10
votes
0answers
267 views
When/why not to use studentized residuals for regression diagnostics?
Consider a linear regression
$$
y=X\beta+\varepsilon.
$$
Residuals $e:=y-X\hat\beta$ are often used as substitutes for the unobserved model errors $\varepsilon$ for validating assumptions such as ...
10
votes
1answer
2k views
Hidden state models vs. stateless models for time series regression
This is a quite generic question: assume I want to build a model to predict the next observation based on the previous $N$ observations ($N$ can be a parameter to optimize experimentally). So we ...
10
votes
1answer
388 views
Interpreting regression coefficients based on Andrew Gelman's re-scaling method
I have two predictors in a binary logistic regression model: One binary and one continuous. My primary goal is to compare the coefficients of the two predictors within the same model.
I have come ...
9
votes
0answers
126 views
Pope effect on pizza - Regression with presence absence and similarity data as dependent variables
I'm trying to figure out the right way to set up a regression when the dependent variables are presence absence data (of pizzas), and the similarity between the present pizzas. Bear with the story:
...
9
votes
0answers
176 views
Bootstrap Prediction Interval: which residuals to use and which method?
I ask this question referring to the post: Bootstrap prediction interval, where a step by step method for calculating the prediction interval for linear regression models is explained.
In the ...
9
votes
0answers
1k views
Are there unbiased, non-linear estimators with lower variance than the OLS estimator?
Consider an ordinary least squares model,
$$y = \beta X + \epsilon \qquad \epsilon\sim N(0, \sigma)$$
The Gauss-Markov theorem tells us that the ordinary least-squares (OLS) estimator is the minimum-...
8
votes
0answers
154 views
Understanding equation used by Hastie et al
I am trying to recreate FIGURE 3.6 from Elements of Statistical Learning. The only information about the figure is included in the caption.
I am not clear on what the equation on the Y-axis means ...
8
votes
1answer
2k views
does serial correlation have something to do with endogeneity?
I'm a beginner of econometrics, and I've construed that endogeneity is caused by omitted variable bias, measurement error, and reverse causality, and it makes OLS estimator be biased.
And also I've ...
8
votes
0answers
397 views
How do sufficiency statistics help in the interpretation of regression results?
One of the results why canonical link functions are widely used in GLMs is the existence of sufficiency statistics for the regression parameters, which in turn allow for:
... minimal
sufficient ...
8
votes
0answers
198 views
Is autocorrelation not worth addressing with small N?
Consider a simple regression context in which there is a small set of response values, $Y$, and corresponding dates, $X$. (For simplicity, we can assume the dates are equally spaced.) We would like ...
7
votes
0answers
179 views
Efficient nonparametric estimation of confidence intervals and p-values for nonlinear regression
I'm estimating parameters for a complex, "implicit" nonlinear model $f(\mathbf{x}, \boldsymbol{\theta})$. It's "implicit" in the sense that I don't have an explicit formula for $f$: its value is the ...
7
votes
0answers
580 views
Assumptions for PCR and PLS
I am writing up a report on fitting Principal Component Regression (PCR) and Partial Least Squares (PLS) to my data-set.
A similar question: Model assumptions of partial least squares (PLS) ...
7
votes
1answer
265 views
Logistic Regression with (Normal) Distributions for Independent Variables
Consider the logistic regression where $Y_i \in {0,1}$ are dependent variable observations and $X_i \in \mathbb{R}$ are the independent variables.
However we do not observe the $X_i$ themselves. ...
7
votes
0answers
4k views
Poisson regression for binary data
I've been trying to read up on Poisson regression models, and it looks like it is possible to estimate such a model with a binary outcome. This has come up before on this site here (and somewhat here ...
7
votes
1answer
3k views
Using Fieller's theorem to calculate the confidence interval of a ratio (paired measurements)
If you have two means (with their own confidence intervals) and want to represent them as a ratio, how do calculate the confidence interval for the ratio?
An answer that was given to me, mentions ...
6
votes
0answers
37 views
Interactions in Multiple Linear Regression (Divide Vs Multiply)
My question is about the difference (in general) between the interaction terms $x_1x_2$ and $x_1/x_2$ in multiple linear regression.
Suppose you are performing multiple linear regression and you have ...
6
votes
0answers
74 views
In reality, there is almost always measurement error in the independent variable(s), so why is this ignored in almost every linear regression model?
In the vast majority of cases, linear regression models are used in practice as opposed to the more complicated errors-in-variables models. For the sake of example, consider modelling height $Y$ vs ...
6
votes
0answers
212 views
Robust Gamma Regression
I am modeling some spectroscopic data where the response of the instrument to the size of the input is strictly positive and non-linear. Gamma regression seems like a good choice to explain the data, ...
6
votes
0answers
210 views
Regression with dependent data with low dependence
Suppose you have data that is grouped in one way or another and therefore the assumption of independence is suspect. But you look at the intraclass correlation (or autocorrelation) and it is very ...
6
votes
0answers
5k views
Calculating R-squared using standard errors
I have the following estimated model: $\hat{y} = 0.2857 + 0.8019x_1 - 0.0741x_2$ (the $t$-statistics are $1.8959$, $8.4198$, and $-3.7017$, respectively).
Furthermore, I know the sample size $N = 92$,...
6
votes
0answers
697 views
Geometrical interpretation of L1 regression
I have found the following image (or a similar version) in a lot of books related to penalized linear models. I get the insight of this image. The ellipsoids are the solution of the linear regression ...
6
votes
0answers
1k views
Is the OLS estimator the UMVUE (assuming Normality)?
Suppose
$$
\mathbf{y} = \mathbf{X} \mathbf{b} + \mathbf{e} \, ,
\\
\mathbf{e} \sim \mathcal{N}(0,\mathbf{I}_P) \, .
$$
We know that $\mathbf{\hat{b}} = (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \...
6
votes
0answers
201 views
Is there any geometric intuition on least absolute deviation regression?
There are a lot of geometric intuitions for regression with least square, e.g., projection, orthogonal, etc. (This and this answers are good examples.)
Is there similar geometric intuition for least ...
6
votes
0answers
575 views
Is lasso always outperformed by adaptive lasso?
I have been reading some papers and I understood that adaptive lasso has the Oracle properties which lasso lacks. Does that mean adaptive lasso always better than lasso (let's focus on the simple ...
6
votes
0answers
326 views
Does there exist an analogous statement to BLUE (Gauss-Markov) for GLMs?
I recall from my graduate school days that the Gauss-Markov (GM) theorem states that the Best Linear Unbiased Estimator (BLUE) in a linear regression is $\vec{\beta}=(X^TX)^{-1}X^T\vec{y}$. An amazing ...
6
votes
0answers
949 views
Signatures of underfitting and overfitting in logistic regression calibration curves
My confusion stems from reading the following paper
http://www.bmj.com/content/351/bmj.h3868
It states in its abstract (and they later show an empirical study that conforms to the claim) - "...
6
votes
0answers
2k views
When is oversampling poor practice?
For my particular domain and problem, I have data on the entire population. However, my "event" only occurs in 0.5% of the cases. I want my model to be able to pick up on significant characteristics ...
6
votes
0answers
483 views
Cumulative hazard in the setting of Cox regression of repeated events
Cox regression is commonly extended to estimate repeated events processes (for a quick review see [ 1 ] and [ 2 ]).
In Clark et al's first article in their excellent review series of survival ...
6
votes
0answers
867 views
Does there exist zero-inflated linear regression?
I have a non-count data with huge number of zeros in the target variable. I need to fit a model being a mixture of Dirac delta function and normal distribution parametrized by mean $X\beta$ and ...
6
votes
0answers
1k views
How to test a linear relationship between log odds and predictors before performing logistic regression?
In case of a linear regression, it's easy to test a linear relationship between a continuous dependent variable and each independent variable. For example, I can plot a scatter plot between the ...
6
votes
0answers
74 views
Can I use bootstrap results at the observation level?
I have read quite a bit of bootstrapping, but the issue I want to address seem not to appear.
Consider a simple regression model:
$$ y_{i} = \beta_{0} + \beta_{1}x_{i} + e_{i}$$
I am aware that ...
6
votes
0answers
898 views
Why would I use ratio estimation instead of regression estimation to estimate means?
I am taking a graduate course on survey data analysis. I was recently introduced to ratio estimation and regression estimation.
I understand that using ratio estimator may be easier if we are ...
6
votes
0answers
719 views
Can I calculate Cohen's $d$ from multiple regression coefficient?
Question: Is it appropriate to calculate Cohen's $d$ (effect size) from the regression coefficient of an independent categorical variable?
Background: My regression coefficient represents ...
6
votes
0answers
2k views
Fixed Effects vs Lagged DV vs. First Differences Regression
What are the differences between using unit fixed effects, unit fixed effects and time fixed effects, lagged DV, or first differences to analyze a time series with 4-5 time periods and 35-50 units per ...
6
votes
0answers
7k views
Regress residuals in second regression
I am wondering if anyone can point me to a paper/lecture notes on the rationale behind first running an OLS on a set of variables, and then in a second regression using the residuals of that ...
6
votes
1answer
702 views
Calculating and plotting confidence interval for Theil-Sen estimator
I'm using Wilcox's R functions (specifically, regplot) to plot a Theil-Sen estimator with a single predictor.
However, regplot ...
6
votes
0answers
680 views
Rule of thumb for excluded variable in Heckman selection model?
I'm working on a project that involves the use of a Heckman selection model (more specifically a Roy or move-stay model, which is essentially a two-sided Heckman) of the following form:
$$ Y_{i1} = ...
6
votes
1answer
334 views
minimizer weighted linear regression
In a regression problem, with $y=X\theta+\epsilon$ and $X$ is an $n$ by $p$ matrix
the ‘weighted least squares estimate is the minimizer $\theta^{*}$ of $f(\theta)=\sum_{i=1}^{n}\omega_{i}(y_i-x_i^{'}\...
6
votes
0answers
949 views
Bias Variance tradeoff from a Bayesian perspective
I know the general question about bias variance has been asked before. I understand the frequentist approach and the concept of model selection and the impact of bias and variance on "accuracy" of a ...
6
votes
0answers
134 views
R packages that work with biased samples
I'm working with a biased sample of web users. I'm only able to track responses of users who have navigated my site in a certain way, and I'd like to run an analysis to determine how certain factors (...
6
votes
0answers
1k views
Logistic regression and maximum entropy
I have read (e.g. here) that a (multinomial) logistic regressor corresponds to a maximum entropy classifier.
My question is, how does one end up with the formula for logistic regression starting with ...
6
votes
0answers
247 views
Generalization of cumulative probability models for ordinal Y
There are many models in existence for ordinal $Y$, for example the proportional odds ordinal logistic model, the continuation ratio model, and the cumulative probit model. The first and third of ...
6
votes
0answers
101 views
Zero values and discontinuity in explanatory variable
One of my independent variables measures worker productivity through the variable $\frac{\log{sales}}{\text{# of workers}}$, and I'm creating one variable for skilled and another for unskilled workers....
6
votes
0answers
631 views
Studentized residuals and goodness-of-fit with robust linear regression
Could you please advise whether studentized residuals are meaningful when computed on a robust linear regression model using an M-estimator?
I'd like to use it to detect outliers by doing something ...