Questions tagged [regression]

Techniques for analyzing the relationship between one (or more) "dependent" variables and "independent" variables.

9,997 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
16 votes
0 answers
554 views

Asymptotic property of tuning parameter in penalized regression

I'm currently working on asymptotic properties of penalized regression. I've read a myriad of papers by now, but there is an essential issue that I cannot get my head around. To keep things simple, I'...
Nick Sabbe's user avatar
  • 12.8k
16 votes
0 answers
13k views

Time series regression with overlapping data

I am seeing a regression model which is regressing Year-on-Year stock index returns on lagged (12 months) Year-on-Year returns of the same stock index, credit spread (difference between monthly mean ...
Vishal Belsare's user avatar
15 votes
0 answers
4k views

Interpreting and reporting gamm4 result

I am new to generalized additive models (GAMs), and most of my knowledge comes from this document. Now I am using a generalized additive model with random effects to model some data, where I want to ...
nan's user avatar
  • 1,125
13 votes
0 answers
231 views

Logistic regression for classification: are there any analytical solutions for the out-of-sample accuracy?

I run a binary logistic regression, with a binary dependent variable and a continuous independent one. Now I want to evaluate the out-of-sample performance of the classification algorithm so obtained. ...
robertspierre's user avatar
13 votes
0 answers
694 views

Interpreting regression coefficients based on Andrew Gelman's re-scaling method

I have two predictors in a binary logistic regression model: One binary and one continuous. My primary goal is to compare the coefficients of the two predictors within the same model. I have come ...
ksroogl's user avatar
  • 413
13 votes
0 answers
362 views

Is autocorrelation not worth addressing with small N?

Consider a simple regression context in which there is a small set of response values, $Y$, and corresponding dates, $X$. (For simplicity, we can assume the dates are equally spaced.) We would like ...
gung - Reinstate Monica's user avatar
10 votes
0 answers
176 views

Pope effect on pizza - Regression with presence absence and similarity data as dependent variables

I'm trying to figure out the right way to set up a regression when the dependent variables are presence absence data (of pizzas), and the similarity between the present pizzas. Bear with the story: ...
elsherbini's user avatar
10 votes
0 answers
551 views

When using L2 regularization outside of linear regression, do the same MAP estimation assumptions hold?

Some context is shared below, and my question is bolded at the end. MLE from observation noise In the linear regression setting, we learn model weights $\mathbf{w}$ to make scalar predictions $\hat{y}...
kdbanman's user avatar
  • 857
10 votes
0 answers
1k views

Bootstrap Prediction Interval: which residuals to use and which method?

I ask this question referring to the post: Bootstrap prediction interval, where a step by step method for calculating the prediction interval for linear regression models is explained. In the ...
user2683832's user avatar
10 votes
2 answers
3k views

Does serial correlation have something to do with endogeneity?

I'm a beginner of econometrics, and I've construed that endogeneity is caused by omitted variable bias, measurement error, and reverse causality, and it makes OLS estimator be biased. And also I've ...
Kevin Kang's user avatar
9 votes
0 answers
4k views

T-test for regression coefficients obtained from Ridge, LASSO etc

In ordinary least squares, for example in an experimental design case, I obtain the regression coefficents by: $ \hat B = {({X^t}{X})}^{-1}X^ty$ Then, my null hypothesis for each coefficent is: $...
gunakkoc's user avatar
  • 1,522
9 votes
0 answers
214 views

Efficient nonparametric estimation of confidence intervals and p-values for nonlinear regression

I'm estimating parameters for a complex, "implicit" nonlinear model $f(\mathbf{x}, \boldsymbol{\theta})$. It's "implicit" in the sense that I don't have an explicit formula for $f$: its value is the ...
DeltaIV's user avatar
  • 17.8k
9 votes
0 answers
497 views

How do sufficiency statistics help in the interpretation of regression results?

One of the results why canonical link functions are widely used in GLMs is the existence of sufficiency statistics for the regression parameters, which in turn allow for: ... minimal sufficient ...
Alex's user avatar
  • 4,340
9 votes
0 answers
1k views

Why would I use ratio estimation instead of regression estimation to estimate means?

I am taking a graduate course on survey data analysis. I was recently introduced to ratio estimation and regression estimation. I understand that using ratio estimator may be easier if we are ...
Ozan Öğreden's user avatar
9 votes
0 answers
1k views

What techniques are there to measure goodness of fit of Deming (orthogonal) regression?

Questions: Even if there is no "widely accepted" technique, is there a useful-and-above-average technique for estimating goodness of fit in orthogonal regressions? What are the pros/cons of this ...
NOTM's user avatar
  • 133
9 votes
3 answers
515 views

Minimizer weighted linear regression

In a regression problem, with $y=X\theta+\epsilon$ and $X$ is an $n$ by $p$ matrix the ‘weighted least squares estimate is the minimizer $\theta^{*}$ of $f(\theta)=\sum_{i=1}^{n}\omega_{i}(y_i-x_i^{'}\...
M.Krov's user avatar
  • 191
9 votes
0 answers
6k views

Poisson regression for binary data

I've been trying to read up on Poisson regression models, and it looks like it is possible to estimate such a model with a binary outcome. This has come up before on this site here (and somewhat here ...
dimitriy's user avatar
  • 35.2k
9 votes
1 answer
5k views

Using Fieller's theorem to calculate the confidence interval of a ratio (paired measurements)

If you have two means (with their own confidence intervals) and want to represent them as a ratio, how do calculate the confidence interval for the ratio? An answer that was given to me, mentions ...
Marc's user avatar
  • 91
8 votes
1 answer
87 views

How did Auguste Bravais come up with the regression line?

I am new to statistics and linear regression and I came across the face that auguste bravais discovered regression line but didn't realize it. Auguste Bravais (1811-1863), professor of astronomy and ...
Alexander's user avatar
  • 181
8 votes
0 answers
2k views

Fitting a Logistic Regression via Brier Score or Mean Squared Error

Is there a name for a logistic regression model that has been fit using the Brier score (or equivalently the mean-squared error) rather than the cross-entropy? I realise this isn't maximum-likelihood, ...
Dikran Marsupial's user avatar
8 votes
0 answers
225 views

Regression with dependent data with low dependence

Suppose you have data that is grouped in one way or another and therefore the assumption of independence is suspect. But you look at the intraclass correlation (or autocorrelation) and it is very ...
Peter Flom's user avatar
  • 117k
8 votes
0 answers
2k views

Multi-target Regression Neural Network: Trade Off

Suppose you have a number of input features, for example: x1 - temperature x2 - day of the week x3 - quantity of rainfall ... You are trying to predict a number of output targets - using neural ...
Mike Tauber's user avatar
  • 1,037
8 votes
1 answer
887 views

Assumptions for PCR and PLS

I am writing up a report on fitting Principal Component Regression (PCR) and Partial Least Squares (PLS) to my data-set. A similar question: Model assumptions of partial least squares (PLS) ...
Kivis's user avatar
  • 291
8 votes
0 answers
3k views

Assumptions of correlation test vs regression slope test (significance testing)

If my understanding is correct, then the test on a regression slope in a simple bivariate regression - i.e. the test of $\mathcal{H}_0$: $b = 0$ in $Y' = a + bX$ and the test of a correlation, i.e. $...
J Taylor's user avatar
  • 381
8 votes
0 answers
3k views

Fixed Effects vs Lagged DV vs. First Differences Regression

What are the differences between using unit fixed effects, unit fixed effects and time fixed effects, lagged DV, or first differences to analyze a time series with 4-5 time periods and 35-50 units per ...
Michael's user avatar
  • 402
8 votes
0 answers
286 views

Generalization of cumulative probability models for ordinal Y

There are many models in existence for ordinal $Y$, for example the proportional odds ordinal logistic model, the continuation ratio model, and the cumulative probit model. The first and third of ...
Frank Harrell's user avatar
8 votes
0 answers
976 views

Dantzig Selector, LASSO, LAD LASSO

I am wondering about this. When is it best to use Dantzig Selector (the infinity normed error measure plus the L1 regularizer) , the LASSO (the mean square error measure plus the L1 regularizer), and ...
Meenakshi's user avatar
  • 391
8 votes
1 answer
153 views

What do the terms "nearly-optimal rate", "near-minimax rate", "minimax optimal rate" and "minimax rate" mean in the context of posterior consistency?

Definition: A sequence $\epsilon_n$ is a posterior contraction rate at the parameter $θ_0$ if $$\Pi_n(θ: d(θ, θ_0) ≥ M_n \epsilon_n| X^{(n)}) → 0$$ in $P^{(n)}_{θ_0}$-probability, for every $M_n → ∞$. ...
user3911153's user avatar
8 votes
1 answer
770 views

How to subset alternatives in nested multinomial logistic regression?

I am trying to predict whether or not captains in a particular groundfish fishery choose to fish on any given day and what variables may influence that decision. Originally I had planned on using ...
Trevor Gratz's user avatar
7 votes
0 answers
117 views

Can you tell if a linear regression slope is significant just by plotting its confidence interval?

When doing a hypothesis test for whether some value $a$ is significantly different from some other value $b$, if the confidence interval for $a$ contains $b$, then we cannot reject the null hypothesis ...
Benedict Clarkson's user avatar
7 votes
1 answer
165 views

Posterior consistency for scale-mixture shrinkage priors in low dimension?

Consider the model [1] $$y_n=X_n\beta_n+\epsilon_n$$ $$\beta_i|\sigma^2,v_i \sim \mathcal{N}(0,\sigma^2 v_i), i=1,\ldots,p$$ $$v_i \sim \beta^\prime(a,b)$$ $$\sigma^2 \sim \mathcal{IG}(c,d)$$ where $\...
MrDi's user avatar
  • 129
7 votes
0 answers
403 views

In Ordinary Least Square (OLS) estimation: is the slope actually an "Inverse-variance weighting" estimator?

I am suspecting the answer is yes, but I'd appreciate help in proving it (even though we know that the estimator is BLUE, so it should probably hold). For context: An Inverse-variance weighting is ...
Tal Galili's user avatar
  • 21.4k
7 votes
1 answer
344 views

Xgboost: does data need to be standardized when using shrinkage (parameters lambda, gamma)?

In this similar question about the implications of standardizing the features of data, the answer is that it is not important. However, (as pointed out in the comments on that post) I am interested in ...
PaulG's user avatar
  • 1,245
7 votes
0 answers
149 views

Identifying non-linearities in relationship between variables

Logistic regression is often used to identify the effect of $x$ on a binary variable $y$ after adjusting for potential confounders $x_1,...,x_n$. In the medical literature, I will sometimes encounter ...
The_Anomaly's user avatar
7 votes
0 answers
273 views

Minimizing MISE to find consistent estimator

Consider kernel regression estimation of the mean function $m$ of the process $$y_t = m(x_t) + \epsilon_t,$$ where $\epsilon_t$' s are correlated with covariance function $R(s,t) = \exp \{-\lambda|s-...
Shanks's user avatar
  • 755
7 votes
0 answers
402 views

Robust Gamma Regression

I am modeling some spectroscopic data where the response of the instrument to the size of the input is strictly positive and non-linear. Gamma regression seems like a good choice to explain the data, ...
udushu's user avatar
  • 223
7 votes
0 answers
1k views

Geometrical interpretation of L1 regression

I have found the following image (or a similar version) in a lot of books related to penalized linear models. I get the insight of this image. The ellipsoids are the solution of the linear regression ...
Álvaro Méndez Civieta's user avatar
7 votes
0 answers
253 views

Is there any geometric intuition on least absolute deviation regression?

There are a lot of geometric intuitions for regression with least square, e.g., projection, orthogonal, etc. (This and this answers are good examples.) Is there similar geometric intuition for least ...
Haitao Du's user avatar
  • 36.6k
7 votes
0 answers
792 views

Is lasso always outperformed by adaptive lasso?

I have been reading some papers and I understood that adaptive lasso has the Oracle properties which lasso lacks. Does that mean adaptive lasso always better than lasso (let's focus on the simple ...
MZ75's user avatar
  • 81
7 votes
2 answers
1k views

Linear model with hidden variable

I have come across a somewhat unusual (I think) estimation problem. I have two "coupled" linear regression models, $$Y = a + b x + \epsilon, \quad Z = c + d x + \nu$$ where $Y,Z,\epsilon,\nu$ are ...
Roland's user avatar
  • 181
7 votes
0 answers
87 views

Can I use bootstrap results at the observation level?

I have read quite a bit of bootstrapping, but the issue I want to address seem not to appear. Consider a simple regression model: $$ y_{i} = \beta_{0} + \beta_{1}x_{i} + e_{i}$$ I am aware that ...
luchonacho's user avatar
  • 2,717
7 votes
0 answers
780 views

Interpretation of smoothing spline

This question is about interpreting the results from non-linear regression models, especially when using regression splines. The numerical output is not very informative when interpreting the effects, ...
JonB's user avatar
  • 2,870
7 votes
0 answers
1k views

Can I calculate Cohen's $d$ from multiple regression coefficient?

Question: Is it appropriate to calculate Cohen's $d$ (effect size) from the regression coefficient of an independent categorical variable? Background: My regression coefficient represents ...
user89128's user avatar
7 votes
0 answers
4k views

Testing for conditional independence: What's the correct way?

My goal is to check if two variables $X$ and $Y$ are conditionally independent given $Z$. For simplicity, let's assume the joint distribution is multivariate normal. In this case, we can compute ...
Vimal's user avatar
  • 1,107
7 votes
0 answers
1k views

Rule of thumb for excluded variable in Heckman selection model?

I'm working on a project that involves the use of a Heckman selection model (more specifically a Roy or move-stay model, which is essentially a two-sided Heckman) of the following form: $$ Y_{i1} = ...
NickCHK's user avatar
  • 271
7 votes
0 answers
1k views

Bias Variance tradeoff from a Bayesian perspective

I know the general question about bias variance has been asked before. I understand the frequentist approach and the concept of model selection and the impact of bias and variance on "accuracy" of a ...
dazedandconfused's user avatar
7 votes
0 answers
154 views

R packages that work with biased samples

I'm working with a biased sample of web users. I'm only able to track responses of users who have navigated my site in a certain way, and I'd like to run an analysis to determine how certain factors (...
user1956609's user avatar
7 votes
0 answers
2k views

Standardizing count variables in panel data with overdispersion - R or Stata

I'm running a regression where the dependent (response) variable is a highly dispersed (slightly zero-inflated) count and the explanatory (independent or predictor) variables are continuous, counts as ...
SJDS's user avatar
  • 525
7 votes
1 answer
2k views

Geometric Interpretation of Softmax Regression

I'm writing a series of blog posts on the basics of machine learning, just for fun, mostly to validate my understanding of Andrew Ng's class. As I'm currently studying generalized linear models (GLMs),...
cjauvin's user avatar
  • 613
7 votes
0 answers
117 views

Zero values and discontinuity in explanatory variable

One of my independent variables measures worker productivity through the variable $\frac{\log{sales}}{\text{# of workers}}$, and I'm creating one variable for skilled and another for unskilled workers....
user62117's user avatar
  • 104

1
2 3 4 5
200