Questions tagged [regression]

Techniques for analyzing the relationship between one (or more) "dependent" variables and "independent" variables.

Filter by
Sorted by
Tagged with
38
votes
6answers
18k views

Why do we need multivariate regression (as opposed to a bunch of univariate regressions)?

I just browsed through this wonderful book: Applied multivariate statistical analysis by Johnson and Wichern. The irony is, I am still not able to understand the motivation for using multivariate (...
38
votes
2answers
17k views

Gradient Boosting for Linear Regression - why does it not work?

While learning about Gradient Boosting, I haven't heard about any constraints regarding the properties of a "weak classifier" that the method uses to build and ensemble model. However, I could not ...
38
votes
2answers
35k views

How do I know which method of cross validation is best?

I am trying to figure out which cross validation method is best for my situation. The following data are just an example for working through the issue (in R), but my real ...
38
votes
1answer
20k views

Proof that the coefficients in an OLS model follow a t-distribution with (n-k) degrees of freedom

Background Suppose we have an Ordinary Least Squares model where we have $k$ coefficients in our regression model, $$\mathbf{y}=\mathbf{X}\mathbf{\beta} + \mathbf{\epsilon}$$ where $\mathbf{\beta}$ ...
38
votes
3answers
5k views

Variance of $K$-fold cross-validation estimates as $f(K)$: what is the role of “stability”?

TL,DR: It appears that, contrary to oft-repeated advice, leave-one-out cross validation (LOO-CV) -- that is, $K$-fold CV with $K$ (the number of folds) equal to $N$ (the number of training ...
37
votes
5answers
94k views

How to derive the least square estimator for multiple linear regression?

In the simple linear regression case $y=\beta_0+\beta_1x$, you can derive the least square estimator $\hat\beta_1=\frac{\sum(x_i-\bar x)(y_i-\bar y)}{\sum(x_i-\bar x)^2}$ such that you don't have to ...
37
votes
3answers
39k views

Significance contradiction in linear regression: significant t-test for a coefficient vs non-significant overall F-statistic

I'm fitting a multiple linear regression model between 4 categorical variables (with 4 levels each) and a numerical output. My dataset has 43 observations. Regression gives me the following $p$-...
37
votes
2answers
40k views

Purpose of the link function in generalized linear model

What is the purpose of the link function as a component of the generalized linear model? Why do we need it? Wikipedia states: It can be convenient to match the domain of the link function to the ...
37
votes
2answers
6k views

Theory behind partial least squares regression

Can anyone recommend a good exposition of the theory behind partial least squares regression (available online) for someone who understands SVD and PCA? I have looked at many sources online and have ...
37
votes
2answers
10k views

Is there any algorithm combining classification and regression?

I'm wondering if there's any algorithm could do classification and regression at the same time. For example, I'd like to let the algorithm learn a classifier, and at the same time within each label, ...
36
votes
6answers
35k views

How does cross-validation overcome the overfitting problem?

Why does a cross-validation procedure overcome the problem of overfitting a model?
36
votes
4answers
45k views

How is the cost function from Logistic Regression derivated

I am doing the Machine Learning Stanford course on Coursera. In the chapter on Logistic Regression, the cost function is this: Then, it is derivated here: I tried getting the derivative of the cost ...
36
votes
2answers
11k views

Do we need gradient descent to find the coefficients of a linear regression model?

I was trying to learn machine learning using the Coursera material. In this lecture, Andrew Ng uses gradient descent algorithm to find the coefficients of the linear regression model that will ...
36
votes
2answers
30k views

Understanding shape and calculation of confidence bands in linear regression

I am trying to understand the origin of the curved shaped of confidence bands associated with an OLS linear regression and how it relates to the confidence intervals of the regression parameters (...
35
votes
4answers
13k views

Why does logistic regression become unstable when classes are well-separated?

Why is it that logistic regression becomes unstable when classes are well-separated? What does well-separated classes mean? I would really appreciate if someone can explain with an example.
35
votes
4answers
43k views

Why squared residuals instead of absolute residuals in OLS estimation? [duplicate]

Why are we using the squared residuals instead of the absolute residuals in OLS estimation? My idea was that we use the square of the error values, so that residuals below the fitted line (which are ...
35
votes
3answers
94k views

Polynomial regression using scikit-learn

I am trying to use scikit-learn for polynomial regression. From what I read polynomial regression is a special case of linear regression. I was hopping that maybe one of scikit's generalized linear ...
35
votes
5answers
3k views

What if my linear regression data contains several co-mingled linear relationships?

Let's say I am studying how daffodils respond to various soil conditions. I have collected data on the pH of the soil versus the mature height of the daffodil. I'm expecting a linear relationship, ...
35
votes
3answers
20k views

Why is RSS distributed chi square times n-p?

I would like to understand why, under the OLS model, the RSS (residual sum of squares) is distributed $$\chi^2\cdot (n-p)$$ ($p$ being the number of parameters in the model, $n$ the number of ...
35
votes
4answers
101k views

How to translate the results from lm() to an equation?

We can use lm() to predict a value, but we still need the equation of the result formula in some cases. For example, add the equation to plots.
35
votes
5answers
52k views

How does linear regression use the normal distribution?

In linear regression, each predicted value is assumed to have been picked from a normal distribution of possible values. See below. But why is each predicted value assumed to have come from a normal ...
34
votes
4answers
46k views

X and Y are not correlated, but X is significant predictor of Y in multiple regression. What does it mean?

X and Y are not correlated (-.01); however, when I place X in a multiple regression predicting Y, alongside three (A, B, C) other (related) variables, X and two other variables (A, B) are significant ...
34
votes
2answers
11k views

Is Tikhonov regularization the same as Ridge Regression?

Tikhonov regularization and ridge regression are terms often used as if they were identical. Is it possible to specify exactly what the difference is?
34
votes
7answers
20k views

Are there algorithms for computing “running” linear or logistic regression parameters?

A paper "Accurately computing running variance" at http://www.johndcook.com/standard_deviation.html shows how to compute running mean, variance and standard deviations. Are there algorithms where the ...
34
votes
3answers
3k views

Datasets constructed for a purpose similar to that of Anscombe's quartet

I've just come across Anscombe's quartet (four datasets that have almost indistinguishable descriptive statistics but look very different when plotted) and I am curious if there are other more or less ...
34
votes
4answers
44k views

How do you Interpret RMSLE (Root Mean Squared Logarithmic Error)?

I've been doing a machine learning competition where they use RMSLE (Root Mean Squared Logarithmic Error) to evaluate the performance predicting the sale price of a category of equipment. The problem ...
34
votes
2answers
16k views

Why is Laplace prior producing sparse solutions?

I was looking through the literature on regularization, and often see paragraphs that links L2 regulatization with Gaussian prior, and L1 with Laplace centered on zero. I know how these priors look ...
34
votes
4answers
19k views

(Why) do overfitted models tend to have large coefficients?

I imagine that the larger a coefficient on a variable is, the more ability the model has to "swing" in that dimension, providing an increased opportunity to fit noise. Although I think I've got a ...
33
votes
11answers
9k views

Regression to the mean vs gambler's fallacy

On the one hand, I have the regression to the mean and on the other hand I have the gambler´s fallacy. Gambler’s fallacy is defined by Miller and Sanjurjo (2019) as “the mistaken belief that random ...
33
votes
5answers
7k views

Why use regularisation in polynomial regression instead of lowering the degree?

When doing regression, for example, two hyper parameters to choose are often the capacity of the function (eg. the largest exponent of a polynomial), and the amount of regularisation. What I'm ...
33
votes
3answers
35k views

What does the logit value actually mean?

I have a logit model that comes up with a number between 0 and 1 for many cases, but how can we interprete this? Lets take a case with a logit of 0.20 Can we assert that there is 20% probability ...
33
votes
4answers
29k views

Ensemble of different kinds of regressors using scikit-learn (or any other python framework)

I am trying to solve the regression task. I found out that 3 models are working nicely for different subsets of data: LassoLARS, SVR and Gradient Tree Boosting. I noticed that when I make predictions ...
33
votes
6answers
5k views

Data mining: How should I go about finding the functional form?

I'm curious about repeatable procedures that can be used to discover the functional form of the function y = f(A, B, C) + error_term where my only input is a set of ...
33
votes
2answers
27k views

Interpretation of simple predictions to odds ratios in logistic regression

I'm somewhat new to using logistic regression, and a bit confused by a discrepancy between my interpretations of the following values which I thought would be the same: exponentiated beta values ...
33
votes
3answers
53k views

Regression coefficients that flip sign after including other predictors

Imagine You run a linear regression with four numeric predictors (IV1, ..., IV4) When only IV1 is included as a predictor the standardised beta is +.20 When you ...
33
votes
6answers
29k views

What's the difference between logistic regression and perceptron?

I'm going through Andrew Ng's lecture notes on Machine Learning. The notes introduce us to logistic regression and then to perceptron. While describing Perceptron, the notes say that we just change ...
33
votes
6answers
14k views

Can deep neural network approximate multiplication function without normalization?

Let say we want to do regression for simple f = x * y using standart deep neural network. I remember that there are reseraches that tells that NN with one hiden ...
33
votes
1answer
3k views

Negative binomial regression question - is it a poor model?

I am reading a very interesting article by Sellers and Shmueli on regression models for count data. Near the beginning (p. 944) they cite McCullaugh and Nelder (1989) saying that negative binomial ...
33
votes
2answers
4k views

Degrees of freedom of $\chi^2$ in Hosmer-Lemeshow test

The test statistic for the Hosmer-Lemeshow test (HLT) for goodness of fit (GOF) of a logistic regression model is defined as follows: The sample is then split into $d=10$ deciles, $D_1, D_2, \dots ...
32
votes
4answers
114k views

McFadden's Pseudo-$R^2$ Interpretation

I have a binary logistic regression model with a McFadden's pseudo R-squared of 0.192 with a dependent variable called payment (1 = payment and 0 = no payment). What is the interpretation of this ...
32
votes
2answers
21k views

Cost function in OLS linear regression

I'm a bit confused with a lecture on linear regression given by Andrew Ng on Coursera about machine learning. There, he gave a cost function that minimises the sum-of-squares as: $$ \frac{1}{2m} \sum ...
32
votes
3answers
12k views

Why is logistic regression a linear model?

I want to know why logistic regression is called a linear model. It uses a sigmoid function, which is not linear. So why is logistic regression a linear model?
32
votes
2answers
23k views

What are the assumptions of negative binomial regression?

I'm working with a large data set (confidential, so I can't share too much), and came to the conclusion a negative binomial regression would be necessary. I've never done a glm regression before, and ...
32
votes
1answer
10k views

Does Cox Regression have an underlying Poisson distribution?

Our small team was having a discussion and got stuck. Does anyone know whether Cox regression has an underlying Poisson distribution. We had a debate that maybe Cox regression with constant time at ...
32
votes
6answers
12k views

How to deal with hierarchical / nested data in machine learning

I'll explain my problem with an example. Suppose you want to predict the income of an individual given some attributes: {Age, Gender, Country, Region, City}. You have a training dataset like so <...
31
votes
4answers
78k views

Choosing the best model from among different “best” models

How do you choose a model from among different models chosen by different methods (e.g. backwards or forwards selection)? Also what is a parsimonious model?
31
votes
5answers
38k views

Neural network with skip-layer connections

I am interested in regression with neural networks. Neural networks with zero hidden nodes + skip-layer connections are linear models. What about the same neural nets but with hidden nodes ? I am ...
31
votes
7answers
5k views

In Regression Analysis, why do we call independent variables “independent”?

I mean some of those variables are strongly correlated among themselves. How / why / in what context do we define them as independent variables?
31
votes
3answers
34k views

What is theta in a negative binomial regression fitted with R?

I've got a question concerning a negative binomial regression: Suppose that you have the following commands: ...
31
votes
5answers
12k views

Detecting significant predictors out of many independent variables

In a dataset of two non-overlapping populations (patients & healthy, total $n=60$) I would like to find (out of $300$ independent variables) significant predictors for a continuous dependent ...