# Questions tagged [regression]

Techniques for analyzing the relationship between one (or more) "dependent" variables and "independent" variables.

23,365 questions
Filter by
Sorted by
Tagged with
18k views

### Why do we need multivariate regression (as opposed to a bunch of univariate regressions)?

I just browsed through this wonderful book: Applied multivariate statistical analysis by Johnson and Wichern. The irony is, I am still not able to understand the motivation for using multivariate (...
17k views

### Gradient Boosting for Linear Regression - why does it not work?

While learning about Gradient Boosting, I haven't heard about any constraints regarding the properties of a "weak classifier" that the method uses to build and ensemble model. However, I could not ...
35k views

### How do I know which method of cross validation is best?

I am trying to figure out which cross validation method is best for my situation. The following data are just an example for working through the issue (in R), but my real ...
20k views

### Proof that the coefficients in an OLS model follow a t-distribution with (n-k) degrees of freedom

Background Suppose we have an Ordinary Least Squares model where we have $k$ coefficients in our regression model, $$\mathbf{y}=\mathbf{X}\mathbf{\beta} + \mathbf{\epsilon}$$ where $\mathbf{\beta}$ ...
5k views

### Variance of $K$-fold cross-validation estimates as $f(K)$: what is the role of “stability”?

TL,DR: It appears that, contrary to oft-repeated advice, leave-one-out cross validation (LOO-CV) -- that is, $K$-fold CV with $K$ (the number of folds) equal to $N$ (the number of training ...
94k views

### How to derive the least square estimator for multiple linear regression?

In the simple linear regression case $y=\beta_0+\beta_1x$, you can derive the least square estimator $\hat\beta_1=\frac{\sum(x_i-\bar x)(y_i-\bar y)}{\sum(x_i-\bar x)^2}$ such that you don't have to ...
39k views

### Significance contradiction in linear regression: significant t-test for a coefficient vs non-significant overall F-statistic

I'm fitting a multiple linear regression model between 4 categorical variables (with 4 levels each) and a numerical output. My dataset has 43 observations. Regression gives me the following $p$-...
40k views

### Purpose of the link function in generalized linear model

What is the purpose of the link function as a component of the generalized linear model? Why do we need it? Wikipedia states: It can be convenient to match the domain of the link function to the ...
6k views

### Theory behind partial least squares regression

Can anyone recommend a good exposition of the theory behind partial least squares regression (available online) for someone who understands SVD and PCA? I have looked at many sources online and have ...
10k views

### Is there any algorithm combining classification and regression?

I'm wondering if there's any algorithm could do classification and regression at the same time. For example, I'd like to let the algorithm learn a classifier, and at the same time within each label, ...
35k views

### How does cross-validation overcome the overfitting problem?

Why does a cross-validation procedure overcome the problem of overfitting a model?
45k views

### How is the cost function from Logistic Regression derivated

I am doing the Machine Learning Stanford course on Coursera. In the chapter on Logistic Regression, the cost function is this: Then, it is derivated here: I tried getting the derivative of the cost ...
11k views

### Do we need gradient descent to find the coefficients of a linear regression model?

I was trying to learn machine learning using the Coursera material. In this lecture, Andrew Ng uses gradient descent algorithm to find the coefficients of the linear regression model that will ...
30k views

### Understanding shape and calculation of confidence bands in linear regression

I am trying to understand the origin of the curved shaped of confidence bands associated with an OLS linear regression and how it relates to the confidence intervals of the regression parameters (...
13k views

### Why does logistic regression become unstable when classes are well-separated?

Why is it that logistic regression becomes unstable when classes are well-separated? What does well-separated classes mean? I would really appreciate if someone can explain with an example.
43k views

### Why squared residuals instead of absolute residuals in OLS estimation? [duplicate]

Why are we using the squared residuals instead of the absolute residuals in OLS estimation? My idea was that we use the square of the error values, so that residuals below the fitted line (which are ...
94k views

### Polynomial regression using scikit-learn

I am trying to use scikit-learn for polynomial regression. From what I read polynomial regression is a special case of linear regression. I was hopping that maybe one of scikit's generalized linear ...
3k views

### What if my linear regression data contains several co-mingled linear relationships?

Let's say I am studying how daffodils respond to various soil conditions. I have collected data on the pH of the soil versus the mature height of the daffodil. I'm expecting a linear relationship, ...
20k views

### Why is RSS distributed chi square times n-p?

I would like to understand why, under the OLS model, the RSS (residual sum of squares) is distributed $$\chi^2\cdot (n-p)$$ ($p$ being the number of parameters in the model, $n$ the number of ...
101k views

### How to translate the results from lm() to an equation?

We can use lm() to predict a value, but we still need the equation of the result formula in some cases. For example, add the equation to plots.
52k views

### How does linear regression use the normal distribution?

In linear regression, each predicted value is assumed to have been picked from a normal distribution of possible values. See below. But why is each predicted value assumed to have come from a normal ...
46k views

### X and Y are not correlated, but X is significant predictor of Y in multiple regression. What does it mean?

X and Y are not correlated (-.01); however, when I place X in a multiple regression predicting Y, alongside three (A, B, C) other (related) variables, X and two other variables (A, B) are significant ...
11k views

### Is Tikhonov regularization the same as Ridge Regression?

Tikhonov regularization and ridge regression are terms often used as if they were identical. Is it possible to specify exactly what the difference is?
20k views

### Are there algorithms for computing “running” linear or logistic regression parameters?

A paper "Accurately computing running variance" at http://www.johndcook.com/standard_deviation.html shows how to compute running mean, variance and standard deviations. Are there algorithms where the ...
3k views

### Datasets constructed for a purpose similar to that of Anscombe's quartet

I've just come across Anscombe's quartet (four datasets that have almost indistinguishable descriptive statistics but look very different when plotted) and I am curious if there are other more or less ...
44k views

### How do you Interpret RMSLE (Root Mean Squared Logarithmic Error)?

I've been doing a machine learning competition where they use RMSLE (Root Mean Squared Logarithmic Error) to evaluate the performance predicting the sale price of a category of equipment. The problem ...
16k views

### Why is Laplace prior producing sparse solutions?

I was looking through the literature on regularization, and often see paragraphs that links L2 regulatization with Gaussian prior, and L1 with Laplace centered on zero. I know how these priors look ...
19k views

### (Why) do overfitted models tend to have large coefficients?

I imagine that the larger a coefficient on a variable is, the more ability the model has to "swing" in that dimension, providing an increased opportunity to fit noise. Although I think I've got a ...
9k views

### Regression to the mean vs gambler's fallacy

On the one hand, I have the regression to the mean and on the other hand I have the gambler´s fallacy. Gambler’s fallacy is defined by Miller and Sanjurjo (2019) as “the mistaken belief that random ...
7k views

### Why use regularisation in polynomial regression instead of lowering the degree?

When doing regression, for example, two hyper parameters to choose are often the capacity of the function (eg. the largest exponent of a polynomial), and the amount of regularisation. What I'm ...
35k views

### What does the logit value actually mean?

I have a logit model that comes up with a number between 0 and 1 for many cases, but how can we interprete this? Lets take a case with a logit of 0.20 Can we assert that there is 20% probability ...
29k views

### Ensemble of different kinds of regressors using scikit-learn (or any other python framework)

I am trying to solve the regression task. I found out that 3 models are working nicely for different subsets of data: LassoLARS, SVR and Gradient Tree Boosting. I noticed that when I make predictions ...
5k views

### Data mining: How should I go about finding the functional form?

I'm curious about repeatable procedures that can be used to discover the functional form of the function y = f(A, B, C) + error_term where my only input is a set of ...
27k views

### Interpretation of simple predictions to odds ratios in logistic regression

I'm somewhat new to using logistic regression, and a bit confused by a discrepancy between my interpretations of the following values which I thought would be the same: exponentiated beta values ...
53k views

### Regression coefficients that flip sign after including other predictors

Imagine You run a linear regression with four numeric predictors (IV1, ..., IV4) When only IV1 is included as a predictor the standardised beta is +.20 When you ...
29k views

### What's the difference between logistic regression and perceptron?

I'm going through Andrew Ng's lecture notes on Machine Learning. The notes introduce us to logistic regression and then to perceptron. While describing Perceptron, the notes say that we just change ...
14k views

### Can deep neural network approximate multiplication function without normalization?

Let say we want to do regression for simple f = x * y using standart deep neural network. I remember that there are reseraches that tells that NN with one hiden ...