Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [least-squares]

Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.

0
votes
0answers
12 views

Measure of error for non-linear systems?

I am trying to model a system that is ideally explained by the formula Q=Qi*(1+Di*b*t)^(-1/b), where: ...
1
vote
0answers
18 views

Confidence interval around weighted sum of regression coefficient estimates?

Let's say you have $N$ random variables $Y_i$, where $Y_i = \beta_i X + \epsilon_i$. $X$ values are the same for all $Y_i$, but the error terms have different variance. I estimate each $\beta_i$ with ...
2
votes
0answers
42 views

divergence of beta estimates between OLS and regression with ARIMA error

I have physiological time-series data: ~60k observations per channel, ~100 Hz sampling. I will model individual channels with ~20 regressors. Under OLS, given temporal autocorrelation in the data, ...
0
votes
0answers
13 views

confusion with frisch lovell type situation

Suppose I am trying to run a regression as follows: $y= a_1+ b_1*x_1 +b_2*x_2 + e_1$ To study the relation between $y$ and $x_2$ net of their correlation with $x_1$, I run the following model first: ...
0
votes
0answers
34 views

How can I find the k-nearest neighbors for a collection of linear time series data?

I need to figure out how to determine the nearest neighbors of an "optimal" line, as illustrated in a simplified figure, linked below. The blue, orange, green, and purple lines represent the best fit ...
1
vote
1answer
30 views

Interpret orthogonal design

I am dealing with orthogonal matrices in regressions, so every regressor has $x_j'x_k=0, if \ j \neq k \\ x_j'x_k=1, if \ \ j=k $ The first one told us that the correlation between two predictors ...
2
votes
1answer
70 views

Ridge/Lasso for correlated response

I want to try a penalised linear regression (ridge/lasso) as a comparison to standard OLS for its predictive ability. My response variable is a continuous measure of an eye parameter, so there is (...
2
votes
1answer
28 views

How can I look for correlations between variables with large deviations?

I'm researching the correlation between the magnitude (a measure of brightness) and redshift ($z$ - a measure of distance) for a variety of galaxies called quasars. Plotting the magnitude against $log(...
1
vote
1answer
33 views

Nonlinear least squares transformation

Suppose that I wish to estimate the parametes $\alpha$ and $\beta$ in the following regression model: $$ Y=K^{\alpha}L^{\beta}\epsilon $$ A standard procedure is to take logs and estimate $$ \text{...
2
votes
1answer
26 views

OLS with Lagged DV

I am interested in building an OLS model with a lagged (lag 1) DV as a right-side explanatory variable. This is relatively straightforward in R, however my problem is the rest of the data. I have six ...
0
votes
0answers
123 views

Multicollinearity in the data with categorical variables

I want to calculate the vif to check for multicollinearity in my data set. I read that a values of >10 tells me that I could have a problem with multicollinearity in my data set. I run an ols ...
0
votes
1answer
39 views

The meaning of coefficients in Multiple Linear Regression

So I am learning about linear regression. The coefficient is the slope of the function, which means how much the dependent variable change due to change of the independent variable. So I make an ...
0
votes
0answers
29 views

EDA and cross-validation suggest no interaction but OLS suggests otherwise

Say that we have two variables, $S$ is a score that is used to predict your $I$, an income. We have two groups in our dataset, $A$ and $B$. Members of group $B$ are expected to have higher income. ...
2
votes
2answers
45 views

Alternative to using $R^2$ to assign data categories?

A background to my problem: I use survey data on firms, where I want to measure the relationship between a binary variable (perceived growth barriers) and firm size. However, I cannot treat "firm size"...
6
votes
4answers
178 views

Is R-squared truly an invalid metric for non-linear models?

I have read that R-squared is invalid for non-linear models, because the relationship that SSR + SSE = SSTotal no longer holds. Can somebody explain why this is true? SSR and SSE are just the ...
0
votes
0answers
14 views

Why does instrument exogeneity imply conditional mean zero?

On slide 14 here: https://www.uio.no/studier/emner/sv/oekonomi/ECON4150/v14/undervisningsmateriale/lecture16_instrumental_variables.pdf it says that "instrument exogeneity implies $E[u_i \mid Z_i]=0$" ...
0
votes
0answers
15 views

How to obtain expressions for coefficients from OLS formula?

Consider the standard linear regression model: $y_i = \alpha + \beta D_i + e_i$ where the coefficients are defined by linear projections and $D_i$ is a dummy variable. In the population, the ...
1
vote
0answers
25 views

Proving First Difference is more efficient than OLS

I am trying to prove that the First Difference method is asymptotically more efficient than OLS when the error term follows a random walk. Assume the following model $$\begin{align}y_i&= \mathbf{...
1
vote
1answer
74 views

Residualizing dependent variable and two step linear regression

Assume we have a DGP of the form $y = \beta_0 + \beta_1 * x_1 + \beta_2 * x_2 + \beta_3 * x_3 + \epsilon$ where $\epsilon$ is a standard i.i.d. error term. Does residualizing $y$ using a linear ...
2
votes
1answer
52 views

Ridge Regression as Robust Optimization

We were told to assume in class that the below optimization formulations are equivalent- $$\min_w\max_{\delta:||\delta||_F\leq\epsilon}||(X+\delta)w-y||_2^2$$ $$\min_{w}||Xw-y||_2^2+\lambda||w||_2^2 ...
6
votes
1answer
759 views

How Ridge or Lasso regression really work?

Very basic question here, but I would like to understand (not mathematically) how the fact to add a "penalty" (sum of squared coeff. times a scalar) to the residual sum of square can reduce big ...
0
votes
0answers
25 views

Linear OLS regression with aggregates and components

A linear OLS regression is specified as Y = a + b*∑(O+R) + c*R + e, i.e. ∑(O+R) is an aggregate and one of the components, R, is added separately. Results for the regression show that both b and c are ...
1
vote
1answer
31 views

Selection of sample on X or Y

In an OLS regression where Y is the dependent variable, X the independent variable and u the error term: Selecting our sample on Y creates a bias: If we have a Y variable that is zero mean and we ...
1
vote
0answers
54 views

Linear Regression When x is Random and Gaussian

Let X denote the design matrix. Our regression is y = $X\hat\beta +\hat\epsilon$. Under the most stringent assumptions i.e. x is assumed to be nonrandom, error terms are iid Gaussian, E[$\epsilon$ | ...
4
votes
0answers
109 views

Prediction with OLS better then prediction with lasso or ridge

I did a regression on a train data set with 7000 observations and 50 explenatory variables with ols ridge and lasso. The lambda was chosen via cross validation. After that i wanted to compare the ...
3
votes
1answer
70 views

Are “feasible generalized least squares” and “iteratively reweighted least squares” the same thing?

These two techniqies seem closely related: Iteratively reweighted least squares (IRLS) Feasible generalized least squares (FGLS) Are the mathematics the same, just different communities (math or ...
1
vote
1answer
305 views

OLS, Fixed effects or Random effects Model?

I am a little bit confused about type of model to apply because my type of data. I am interesting in get regression parameters for time (dependent variable) with independent variables= sex + age+ ...
0
votes
1answer
56 views

MLE with unbalanced system of regressions

I want to estimate the following system of regressions simultaneously: $$ \begin{align} y_1 &=\alpha_1 + \beta\ x_1 + \gamma\ z_2 + \epsilon_1 \\ y_2 &=\alpha_2 + \beta\ x_2 + \gamma\ z_1 + \...
0
votes
1answer
36 views

When use multiple Regression and is linear Regression legit in this case?

I have some trouble understanding the use of multiple regression. I made a survey which has 3 variables (simplified): A, Skill, Money. The participants made choice ...
3
votes
1answer
40 views

Fitted values of a simple regression with intercept and dummy

Why are the fitted values of a simple regression with intercept and dummy, estimated by OLS, just the group means belonging to the two groups of observations? I.e., why do we have that the fitted ...
0
votes
0answers
18 views

Skewed dependend variable, residual assumption violations, appropriate model

I am working with a survey variable which asks respondents to place themselves on a scale from 0 to 10 (integer) (N=1850), where both ends have a specific meaning. Thus, treating the variable as ...
0
votes
1answer
82 views

Why may a matrix be singular or ill-conditioned with standard learning algorithm for linear classification?

In the learning algorithm for linear classification by least square method, which find a weight vector $\hat w\in R^d$ and bias $\hat b\in R$ for a linear scoring function $f(x) = \hat w ^T x +\hat b$ ...
1
vote
0answers
26 views

Formutaion of Least Squares Problem

In general, to use the method of least squares, a linear stochastic system is modeled as: \begin{equation} y = ax + \eta \end{equation} where, $y$, is an observed variable, $x$ is an input while $\...
12
votes
2answers
726 views

Is there any theoretical problem with averaging regression coefficients to build a model?

I want to build a regression model that is an average of multiple OLS models, each based on a subset of the full data. The idea behind this is based on this paper. I create k folds and build k OLS ...
1
vote
0answers
26 views

Partial Residual in OLS or Lasso case

I have a similar question as in this this post. Assume I've a regression $y_i=\beta_1x_1+\beta_2x_2+\beta_3x_3+\epsilon_i$ And the partial residual is defined as: $r_i^{(3)}=y_i-\beta_1x_1+\...
3
votes
1answer
24 views

How to perform Least Squares with constraints on a subset of the model coefficients?

For solving an unconstrained LS regression $$\hat{y}=w_1.x_1+w_2.x_2+w_3.x_3+w_4.x_4 + \epsilon$$ I use the following normal equation: $$W^*=(X^{\top}X)^{-1}X^{\top}Y $$ If I want to introduce a ...
2
votes
1answer
248 views

Linear regression with multiple y values per x and their arithmetic means

I noticed that if there are multiple values of $y$ for each value of $x$, I can replace the values of $y$ with their arithmetic mean at each value of $x$ and still get the same regression model (...
0
votes
0answers
11 views

OLS with shares as outcome

I have a regression where my outcome are share measures within a household (share of underweight for example) and am pondering whether this is a case of censored regression? My intuition is to think ...
0
votes
0answers
93 views

Least squares - why multiply both sides by the transpose? [duplicate]

The formula: $A^T(b - Ax) = 0\\ A^Tb = A^TAx\\ x = (A^TA)^{-1}A^Tb$ What is the reason for multiplying both sides by the transpose of A?
1
vote
0answers
521 views

Find RMSE from StatsModels OLS Results

I playing around with some regression analyses in Python using StatsModels. I am getting a little confused with some terminology and just wanted to clarify. I have run a regression and get the ...
4
votes
3answers
127 views

When is there a difference between a normal likelihood loss and a least squares loss?

My understanding is that if the errors follow a normal distribution, then using a maximum likelihood loss or a least squares loss to train a model amounts to the same thing. However, I am looking at ...
0
votes
0answers
19 views

Interpretation of coefficient of an index whose value lies between 0 and 1

I am running a simple ordinary least squares regression to understand the effect of parental attitude ($PA$) on Math scores of children. $Math Score_{i}= \beta_{0} + PA_{i}\beta_{1} + Controls + \...
7
votes
3answers
73 views

Converting the beta coefficient from matrix to scalar notation in OLS regression

I've found for my econometrics exams that if I forget the scalar notation, I can often save myself by remembering the matrix notation and working backwards. However, the following confused me. Given ...
1
vote
1answer
33 views

OLS and Probit possible on large sample enough?

I think I understood that normality of residuals may not be a problem if the sample is large enough (cf, here). My question is: Would my sample be large enough to be analysed using a probit and an ...
1
vote
1answer
48 views

Sum of squared residuals and MSE

It seems that minimizing the sum of squared residuals (SSR) in linear regression is equivalent to minimizing MSE (both use true value - prediction) and OLS is the best estimator for minimizing SSR. I ...
0
votes
0answers
21 views

Least Squares with coefficient constraints

I need to perform a least squares problem with constraints on some (but not all) of my coefficients. For example say I am fitting the following model: $$\hat{y} = \beta_0 + \beta_1 x_1 + \beta_2 x_2 ...
3
votes
1answer
38 views

Correlation between the j-feature and partial residual

I am reading a book for the lasso-regression. However, I think that this is a more general question regarding the derivative of the OLS-term: Can someone explain me why this $c_j$ term is ...
2
votes
0answers
19 views

How to interpret A VAR model, Method: OLS

I am working on a VAR using an OLS model as can be seen below. I am however having difficulties interpreting the results. I think to understand the basic concept of an equation being significant below ...
1
vote
0answers
10 views

Learning rate: Normalised Leat Mean Squares

I was wondering why when performing NLMS regression the learning rate $\eta>0$ for the following update rule: $$w_t=w_{t-1}-\frac{\eta}{1+\eta||x_t||_2^2}(x_t'w_{t-1}-y_t)x_t$$ If I set $\eta<...
0
votes
0answers
16 views

Test for Homoskedasticity and endogeneity in Panel data with Pooled OLS estimation

Which tests are valid to test for: 1) Homoskedasticity 2) Endogeneity in a Pooled OLS model built on Panel data?