Questions tagged [regression]

Techniques for analyzing the relationship between one (or more) "dependent" variables and "independent" variables.

18
votes
0answers
730 views

How does a Relevance Vector Machine (RVM) work?

Relevance Vector Machines (RVMs) are really interesting models when contrasted with the highly geometrical (and popular) SVMs. In the light of a question like How does a Support Vector Machine (SVM) ...
14
votes
0answers
206 views

Does LASSO suffer from the same problems stepwise regression does?

Stepwise algorithmic variable-selection methods tend to select for models which bias more or less every estimate in regression models ($\beta$s and their SEs, p-values, F statistics, etc.), and are ...
10
votes
0answers
382 views

Asymptotic property of tuning parameter in penalized regression

I'm currently working on asymptotic properties of penalized regression. I've read a myriad of papers by now, but there is an essential issue that I cannot get my head around. To keep things simple, I'...
9
votes
0answers
2k views

Gamma hurdle model for continuous response

I am modelling invertebrate.biomass ~ habitat.type * calendar.day + habitat.type * calendar.day ^ 2, with a random intercept of transect.id (50 transects were repeated 5 times) My response is zero-...
7
votes
0answers
155 views

Efficient nonparametric estimation of confidence intervals and p-values for nonlinear regression

I'm estimating parameters for a complex, "implicit" nonlinear model $f(\mathbf{x}, \boldsymbol{\theta})$. It's "implicit" in the sense that I don't have an explicit formula for $f$: its value is the ...
7
votes
0answers
272 views

Should I use unpenalized logistic regression, lasso or ridge for explanatory modelling?

When using logistic regression for predictive modelling, the choice between 'standard' logistic regression vs ridge vs LASSO versions of logistic regression seems relatively straightforward - just ...
6
votes
0answers
45 views

Robust Gamma Regression

I am modeling some spectroscopic data where the response of the instrument to the size of the input is strictly positive and non-linear. Gamma regression seems like a good choice to explain the data, ...
6
votes
0answers
196 views

Regression with dependent data with low dependence

Suppose you have data that is grouped in one way or another and therefore the assumption of independence is suspect. But you look at the intraclass correlation (or autocorrelation) and it is very ...
6
votes
0answers
152 views

Is there any geometric intuition on least absolute deviation regression?

There are a lot of geometric intuitions for regression with least square, e.g., projection, orthogonal, etc. (This and this answers are good examples.) Is there similar geometric intuition for least ...
6
votes
0answers
67 views

Can I use bootstrap results at the observation level?

I have read quite a bit of bootstrapping, but the issue I want to address seem not to appear. Consider a simple regression model: $$ y_{i} = \beta_{0} + \beta_{1}x_{i} + e_{i}$$ I am aware that ...
6
votes
0answers
578 views

Rule of thumb for excluded variable in Heckman selection model?

I'm working on a project that involves the use of a Heckman selection model (more specifically a Roy or move-stay model, which is essentially a two-sided Heckman) of the following form: $$ Y_{i1} = ...
6
votes
0answers
5k views

“Zero-inflated” predictors in regression?

I know that zero-inflated models (e.g. zero-inflated Poisson or negative binomial models) can be used for dependent variables. I also know that in general there are no assumptions for the independent ...
6
votes
0answers
212 views

Generalization of cumulative probability models for ordinal Y

There are many models in existence for ordinal $Y$, for example the proportional odds ordinal logistic model, the continuation ratio model, and the cumulative probit model. The first and third of ...
6
votes
0answers
99 views

Zero values and discontinuity in explanatory variable

One of my independent variables measures worker productivity through the variable $\frac{\log{sales}}{\text{# of workers}}$, and I'm creating one variable for skilled and another for unskilled workers....
6
votes
0answers
831 views

Dantzig Selector, LASSO, LAD LASSO

I am wondering about this. When is it best to use Dantzig Selector (the infinity normed error measure plus the L1 regularizer) , the LASSO (the mean square error measure plus the L1 regularizer), and ...
6
votes
0answers
4k views

Poisson regression for binary data

I've been trying to read up on Poisson regression models, and it looks like it is possible to estimate such a model with a binary outcome. This has come up before on this site here (and somewhat here ...
6
votes
0answers
540 views

Average Structural Function Calculation

EDIT: I have solved this problem myself. The problem with the simulation below is that the omitted variable should not be included in the 'true model'. I have written a blog post with a more detailed ...
6
votes
0answers
261 views

Making new variable instead of correcting for temporal autocorrelation in a GLMM. Is it a valid alternative?

I am doing some forest disturbance research, in which the aim is to predict the probabilities of wind damage occurrence in forest stands of different site (altitude, slope steepness) and stand ...
5
votes
0answers
44 views

Compare if two regression model behaves similarly

this is my first asking question here... forgive me if my question is not clear enough. I have two datasets; one is ground truth dataset and another is perturbed version of the same dataset. I want ...
5
votes
0answers
44 views

“Return values” of univariate logistic regression

I read an interesting article on an approach to calibrate probabilistic classifiers (Kull et al. 2017, "Beyond sigmoids: How to obtain well-calibrated probabilities from binary classifiers with beta ...
5
votes
0answers
102 views

Why does “mixtools” return the model with highest AIC as the “winner” if lower AIC is better?

Mixtools package is used to fit mixtures of normal/regressions. The package documentation is given here The regmixmodel.sel fits the mixture model for varying ...
5
votes
0answers
198 views

Is linear regression equivalent to the support vector regression with a linear kernel?

I notice the objective function of the linear regression and the support vector regression (SVR) with a linear kernel could be the same, except for the SVR has two error constraints for each data ...
5
votes
0answers
337 views

Predictive modeling using GAM (mgcv)

I have seven years of continuous insect population data, along with temperature and humidity parameters. I’d like to use this data to predict future populations in a given year using a generalized ...
5
votes
0answers
332 views

Bayesian linear regression - posterior distribution

This is about bayesian linear regression. In this link http://fourier.eng.hmc.edu/e161/lectures/gaussianprocess/node2.html there's a derivation for = The part that I don't understand is how it is ...
5
votes
0answers
632 views

Signatures of underfitting and overfitting in logistic regression calibration curves

My confusion stems from reading the following paper http://www.bmj.com/content/351/bmj.h3868 It states in its abstract (and they later show an empirical study that conforms to the claim) - "...
5
votes
0answers
853 views

When is oversampling poor practice?

For my particular domain and problem, I have data on the entire population. However, my "event" only occurs in 0.5% of the cases. I want my model to be able to pick up on significant characteristics ...
5
votes
0answers
575 views

Are there unbiased, non-linear estimators with lower variance than the OLS estimator?

Consider an ordinary least squares model, $$y = \beta X + \epsilon \qquad \epsilon\sim N(0, \sigma)$$ The Gauss-Markov theorem tells us that the ordinary least-squares (OLS) estimator is the minimum-...
5
votes
0answers
514 views

Does there exist zero-inflated linear regression?

I have a non-count data with huge number of zeros in the target variable. I need to fit a model being a mixture of Dirac delta function and normal distribution parametrized by mean $X\beta$ and ...
5
votes
0answers
218 views

Orthogonality in ANOVA and Regression Analysis

I read the following (Wikipedia) regarding contrast coding of categorical variables: Unlike when used in ANOVA, where it is at the researcher’s discretion whether they choose coefficient values ...
5
votes
0answers
167 views

Wishart Conditional

I am looking for the conditional probability of a Wishart distribution, i.e., if I have a Wishart distributed variable $ S \sim W(\Sigma,n), $ where $$ S = \begin{bmatrix} S_1 \quad S_{12} \\ S_{21} ...
5
votes
0answers
4k views

Cause of a high condition number in a python statsmodels regression?

I'm pretty new to regression analysis, and I'm using python's statsmodels to look at the relationship between GDP/health/social services spending and health outcomes (DALYs) across the OECD. Just to ...
5
votes
0answers
7k views

Comparing models using the deviance and log-likelihood ratio tests

This is an update to a previous question that I have posted. I am looking for clarification on comparing glm models using deviance and log-likelihood ratio tests (I have updated my question to make it ...
5
votes
0answers
101 views

Is autocorrelation not worth addressing with small N?

Consider a simple regression context in which there is a small set of response values, $Y$, and corresponding dates, $X$. (For simplicity, we can assume the dates are equally spaced.) We would like ...
5
votes
0answers
260 views

Is there an appropriate order to apply bagging and filter feature selection?

I'm training a (regression) learner on a $p \gg n$ problem, including bagging and filter feature selection (information gain). I'm in doubt though regarding the order of the procedures: Apply the ...
5
votes
0answers
852 views

How to deal with failing the proportional odds assumption in ordinal logistic regression

I am attempting to do ordinal logistic regression but I keep failing to pass the proportional odds assumption. Almost all of my features are shown to have high significance, but the only model that I ...
5
votes
0answers
765 views

Why would I use ratio estimation instead of regression estimation to estimate means?

I am taking a graduate course on survey data analysis. I was recently introduced to ratio estimation and regression estimation. I understand that using ratio estimator may be easier if we are ...
5
votes
0answers
2k views

Assumptions of correlation test vs regression slope test (significance testing)

If my understanding is correct, then the test on a regression slope in a simple bivariate regression - i.e. the test of $\mathcal{H}_0$: $b = 0$ in $Y' = a + bX$ and the test of a correlation, i.e. $...
5
votes
0answers
258 views

Interpretation of smoothing spline

This question is about interpreting the results from non-linear regression models, especially when using regression splines. The numerical output is not very informative when interpreting the effects, ...
5
votes
0answers
501 views

Can I calculate Cohen's $d$ from multiple regression coefficient?

Question: Is it appropriate to calculate Cohen's $d$ (effect size) from the regression coefficient of an independent categorical variable? Background: My regression coefficient represents ...
5
votes
0answers
736 views

What techniques are there to measure goodness of fit of Deming (orthogonal) regression?

Questions: Even if there is no "widely accepted" technique, is there a useful-and-above-average technique for estimating goodness of fit in orthogonal regressions? What are the pros/cons of this ...
5
votes
0answers
450 views

Regression on large samples: can aggregation of the dependent variable by covariate pattern increase speed of estimation?

Is there a way to build a regression model for continuous output using aggregate data instead of individual data points when all input variables are categorical? I have a moderately large dataset (...
5
votes
0answers
4k views

Regress residuals in second regression

I am wondering if anyone can point me to a paper/lecture notes on the rationale behind first running an OLS on a set of variables, and then in a second regression using the residuals of that ...
5
votes
0answers
150 views

How to evaluate uncertainty estimates in regression?

Some regression algorithms (e.g. Gaussian process regression) can produce uncertainties along with point predictions at test time. These should also be evaluated. How about calculating the Pearson ...
5
votes
0answers
393 views

Ideal statistical or machine learning technique to model highly cross-correlated data

I'm trying to build a model that can predict streamflow for an alpine (snowmelt-fed) watershed using snow albedo (roughly, the energy reflectance of the snow) data. Albedo controls the melt of the ...
5
votes
0answers
254 views

Restriction matrix for a VAR

In New Introduction to Multiple Time Series Analysis by Luetkepohl (2005), section 5.2.1, it says that one can specify linear restraints for a VAR, $Y = \beta X + U$, in the form $$ \operatorname{vec}{...
5
votes
0answers
719 views

Bias Variance tradeoff from a Bayesian perspective

I know the general question about bias variance has been asked before. I understand the frequentist approach and the concept of model selection and the impact of bias and variance on "accuracy" of a ...
5
votes
0answers
99 views

R packages that work with biased samples

I'm working with a biased sample of web users. I'm only able to track responses of users who have navigated my site in a certain way, and I'd like to run an analysis to determine how certain factors (...
5
votes
0answers
1k views

Standardizing count variables in panel data with overdispersion - R or Stata

I'm running a regression where the dependent (response) variable is a highly dispersed (slightly zero-inflated) count and the explanatory (independent or predictor) variables are continuous, counts as ...
5
votes
0answers
482 views

Is it ever appropriate to put dependent variables in a model when they calculate the independent variable directly?

Is it ever appropriate to fit a multivariable regression model with dependent variables that directly calculate the independent variable? For example, I know that fitting a model to predict BMI by ...
5
votes
0answers
930 views

Logistic regression and maximum entropy

I have read (e.g. here) that a (multinomial) logistic regressor corresponds to a maximum entropy classifier. My question is, how does one end up with the formula for logistic regression starting with ...