Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [regression]

Techniques for analyzing the relationship between one (or more) "dependent" variables and "independent" variables.

15
votes
0answers
581 views

How does a Relevance Vector Machine (RVM) work?

Relevance Vector Machines (RVMs) are really interesting models when contrasted with the highly geometrical (and popular) SVMs. In the light of a question like How does a Support Vector Machine (SVM) ...
10
votes
0answers
1k views

Ratios in Regression, aka Questions on Kronmal

Recently, randomly browsing questions triggered a memory of on off-hand comment from one of my professors a few years back warning about the usage of ratios in regression models. So I started reading ...
9
votes
0answers
360 views

Asymptotic property of tuning parameter in penalized regression

I'm currently working on asymptotic properties of penalized regression. I've read a myriad of papers by now, but there is an essential issue that I cannot get my head around. To keep things simple, I'...
7
votes
0answers
136 views

Efficient nonparametric estimation of confidence intervals and p-values for nonlinear regression

I'm estimating parameters for a complex, "implicit" nonlinear model $f(\mathbf{x}, \boldsymbol{\theta})$. It's "implicit" in the sense that I don't have an explicit formula for $f$: its value is the ...
7
votes
0answers
229 views

Should I use unpenalized logistic regression, lasso or ridge for explanatory modelling?

When using logistic regression for predictive modelling, the choice between 'standard' logistic regression vs ridge vs LASSO versions of logistic regression seems relatively straightforward - just ...
7
votes
0answers
2k views

Gamma hurdle model for continuous response

I am modelling invertebrate.biomass ~ habitat.type * calendar.day + habitat.type * calendar.day ^ 2, with a random intercept of transect.id (50 transects were repeated 5 times) My response is zero-...
6
votes
0answers
123 views

Sparse linear regression 0-norm and 1-norm

We have a response $Y \in \Bbb R^n$ and predictors $X = (x_1, x_2, \cdots, x_m)^T \in \Bbb R^{n \times m}$ The problem we want to solve is $$\text{argmin}_{k \in \Bbb R^{m}} (\Vert Y - Xk \Vert_2^2 +...
6
votes
0answers
131 views

Is there any geometric intuition on least absolute deviation regression?

There are a lot of geometric intuitions for regression with least square, e.g., projection, orthogonal, etc. (This and this answers are good examples.) Is there similar geometric intuition for least ...
6
votes
0answers
65 views

Can I use bootstrap results at the observation level?

I have read quite a bit of bootstrapping, but the issue I want to address seem not to appear. Consider a simple regression model: $$ y_{i} = \beta_{0} + \beta_{1}x_{i} + e_{i}$$ I am aware that ...
6
votes
0answers
521 views

Rule of thumb for excluded variable in Heckman selection model?

I'm working on a project that involves the use of a Heckman selection model (more specifically a Roy or move-stay model, which is essentially a two-sided Heckman) of the following form: $$ Y_{i1} = ...
6
votes
0answers
4k views

“Zero-inflated” predictors in regression?

I know that zero-inflated models (e.g. zero-inflated Poisson or negative binomial models) can be used for dependent variables. I also know that in general there are no assumptions for the independent ...
6
votes
0answers
205 views

Generalization of cumulative probability models for ordinal Y

There are many models in existence for ordinal $Y$, for example the proportional odds ordinal logistic model, the continuation ratio model, and the cumulative probit model. The first and third of ...
6
votes
0answers
96 views

Zero values and discontinuity in explanatory variable

One of my independent variables measures worker productivity through the variable $\frac{\log{sales}}{\text{# of workers}}$, and I'm creating one variable for skilled and another for unskilled workers....
6
votes
0answers
782 views

Dantzig Selector, LASSO, LAD LASSO

I am wondering about this. When is it best to use Dantzig Selector (the infinity normed error measure plus the L1 regularizer) , the LASSO (the mean square error measure plus the L1 regularizer), and ...
6
votes
0answers
3k views

Poisson regression for binary data

I've been trying to read up on Poisson regression models, and it looks like it is possible to estimate such a model with a binary outcome. This has come up before on this site here (and somewhat here ...
6
votes
0answers
496 views

Average Structural Function Calculation

EDIT: I have solved this problem myself. The problem with the simulation below is that the omitted variable should not be included in the 'true model'. I have written a blog post with a more detailed ...
6
votes
0answers
249 views

Making new variable instead of correcting for temporal autocorrelation in a GLMM. Is it a valid alternative?

I am doing some forest disturbance research, in which the aim is to predict the probabilities of wind damage occurrence in forest stands of different site (altitude, slope steepness) and stand ...
5
votes
0answers
44 views

Why does “mixtools” return the model with highest AIC as the “winner” if lower AIC is better?

Mixtools package is used to fit mixtures of normal/regressions. The package documentation is given here The regmixmodel.sel fits the mixture model for varying ...
5
votes
0answers
141 views

Is linear regression equivalent to the support vector regression with a linear kernel?

I notice the objective function of the linear regression and the support vector regression (SVR) with a linear kernel could be the same, except for the SVR has two error constraints for each data ...
5
votes
0answers
223 views

Predictive modeling using GAM (mgcv)

I have seven years of continuous insect population data, along with temperature and humidity parameters. I’d like to use this data to predict future populations in a given year using a generalized ...
5
votes
0answers
498 views

Signatures of underfitting and overfitting in logistic regression calibration curves

My confusion stems from reading the following paper http://www.bmj.com/content/351/bmj.h3868 It states in its abstract (and they later show an empirical study that conforms to the claim) - "...
5
votes
0answers
497 views

When is oversampling poor practice?

For my particular domain and problem, I have data on the entire population. However, my "event" only occurs in 0.5% of the cases. I want my model to be able to pick up on significant characteristics ...
5
votes
0answers
472 views

Are there unbiased, non-linear estimators with lower variance than the OLS estimator?

Consider an ordinary least squares model, $$y = \beta X + \epsilon \qquad \epsilon\sim N(0, \sigma)$$ The Gauss-Markov theorem tells us that the ordinary least-squares (OLS) estimator is the minimum-...
5
votes
0answers
167 views

Orthogonality in ANOVA and Regression Analysis

I read the following (Wikipedia) regarding contrast coding of categorical variables: Unlike when used in ANOVA, where it is at the researcher’s discretion whether they choose coefficient values ...
5
votes
0answers
151 views

Wishart Conditional

I am looking for the conditional probability of a Wishart distribution, i.e., if I have a Wishart distributed variable $ S \sim W(\Sigma,n), $ where $$ S = \begin{bmatrix} S_1 \quad S_{12} \\ S_{21} ...
5
votes
0answers
425 views

How to test a linear relationship between log odds and predictors before performing logistic regression?

In case of a linear regression, it's easy to test a linear relationship between a continuous dependent variable and each independent variable. For example, I can plot a scatter plot between the ...
5
votes
0answers
6k views

Comparing models using the deviance and log-likelihood ratio tests

This is an update to a previous question that I have posted. I am looking for clarification on comparing glm models using deviance and log-likelihood ratio tests (I have updated my question to make it ...
5
votes
0answers
84 views

Is autocorrelation not worth addressing with small N?

Consider a simple regression context in which there is a small set of response values, $Y$, and corresponding dates, $X$. (For simplicity, we can assume the dates are equally spaced.) We would like ...
5
votes
0answers
211 views

Is there an appropriate order to apply bagging and filter feature selection?

I'm training a (regression) learner on a $p \gg n$ problem, including bagging and filter feature selection (information gain). I'm in doubt though regarding the order of the procedures: Apply the ...
5
votes
0answers
668 views

Why would I use ratio estimation instead of regression estimation to estimate means?

I am taking a graduate course on survey data analysis. I was recently introduced to ratio estimation and regression estimation. I understand that using ratio estimator may be easier if we are ...
5
votes
0answers
184 views

Interpretation of smoothing spline

This question is about interpreting the results from non-linear regression models, especially when using regression splines. The numerical output is not very informative when interpreting the effects, ...
5
votes
0answers
451 views

Can I calculate Cohen's $d$ from multiple regression coefficient?

Question: Is it appropriate to calculate Cohen's $d$ (effect size) from the regression coefficient of an independent categorical variable? Background: My regression coefficient represents ...
5
votes
0answers
681 views

What techniques are there to measure goodness of fit of Deming (orthogonal) regression?

Questions: Even if there is no "widely accepted" technique, is there a useful-and-above-average technique for estimating goodness of fit in orthogonal regressions? What are the pros/cons of this ...
5
votes
0answers
389 views

Regression on large samples: can aggregation of the dependent variable by covariate pattern increase speed of estimation?

Is there a way to build a regression model for continuous output using aggregate data instead of individual data points when all input variables are categorical? I have a moderately large dataset (...
5
votes
0answers
3k views

Regress residuals in second regression

I am wondering if anyone can point me to a paper/lecture notes on the rationale behind first running an OLS on a set of variables, and then in a second regression using the residuals of that ...
5
votes
0answers
130 views

How to evaluate uncertainty estimates in regression?

Some regression algorithms (e.g. Gaussian process regression) can produce uncertainties along with point predictions at test time. These should also be evaluated. How about calculating the Pearson ...
5
votes
0answers
389 views

Ideal statistical or machine learning technique to model highly cross-correlated data

I'm trying to build a model that can predict streamflow for an alpine (snowmelt-fed) watershed using snow albedo (roughly, the energy reflectance of the snow) data. Albedo controls the melt of the ...
5
votes
0answers
231 views

Restriction matrix for a VAR

In New Introduction to Multiple Time Series Analysis by Luetkepohl (2005), section 5.2.1, it says that one can specify linear restraints for a VAR, $Y = \beta X + U$, in the form $$ \operatorname{vec}{...
5
votes
0answers
837 views

Model diagnostics for a glmmPQL in R mixed-effects model

Several texts (both online and published books) have been reviewed prior to asking this. What diagnostics are accepted as best practise for a generalised linear mixed-effects model fitted in R using ...
5
votes
0answers
93 views

R packages that work with biased samples

I'm working with a biased sample of web users. I'm only able to track responses of users who have navigated my site in a certain way, and I'd like to run an analysis to determine how certain factors (...
5
votes
0answers
1k views

Standardizing count variables in panel data with overdispersion - R or Stata

I'm running a regression where the dependent (response) variable is a highly dispersed (slightly zero-inflated) count and the explanatory (independent or predictor) variables are continuous, counts as ...
5
votes
0answers
462 views

Is it ever appropriate to put dependent variables in a model when they calculate the independent variable directly?

Is it ever appropriate to fit a multivariable regression model with dependent variables that directly calculate the independent variable? For example, I know that fitting a model to predict BMI by ...
5
votes
0answers
879 views

Logistic regression and maximum entropy

I have read (e.g. here) that a (multinomial) logistic regressor corresponds to a maximum entropy classifier. My question is, how does one end up with the formula for logistic regression starting with ...
5
votes
0answers
475 views

Studentized residuals and goodness-of-fit with robust linear regression

Could you please advise whether studentized residuals are meaningful when computed on a robust linear regression model using an M-estimator? I'd like to use it to detect outliers by doing something ...
5
votes
0answers
861 views

Rule of thumb - number of predictors - Poisson regression rates

I am interested in estimating a Poisson regression for mortality rates, with number of deaths as the dependent variable and log(population size) as the offset. I have 50 observations (states). I am ...
5
votes
0answers
2k views

Time series forecasting using autoregressive and linear terms in R

I have real daily market data which I'm looking at to create a model for forecasting. The model that I created (below) used autoregressive terms within a linear regression. I was sharing this with a ...
5
votes
0answers
1k views

Log-Ratio \ Compositional analysis

I am not a trained statistician but I am trying to improve on my own MBA thesis which was essentially regression analysis of the factors affecting opening cinema box-office in the UK. I am now ...
5
votes
0answers
392 views

Regression line envelope from Census ACS data

Context: I’m working with the Census Bureau’s American Community Survey (ACS) data which are samples (not complete enumerations) aggregated at different spatial scales. Each ACS estimate is provided ...
5
votes
0answers
95 views

Simultaneous Non-parametric regression and Non-parametric density estimation

Given a collection of $(x,y)$ data, one might do a non/semiparametric regression of $y$ on $x$ to understand how to predict the $E(Y|X)$. Similarly, if one has enough data, it might be useful take a ...
5
votes
0answers
1k views

Methods to best test lead/lag relationships

I was wondering if you can share your experiences on what you feel is the best method to test lead / lag relationships between I(1) time series variables (i.e stock prices) and advantages and ...