Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [least-squares]

Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.

1
vote
1answer
60 views

Multicollinearity confusion

so for my master's thesis, I am examining the influence of union density (% of the workforce in a union) and top marginal tax rates on pre-tax CEO pay. These two independent variables are very highly ...
3
votes
1answer
358 views

How do we know $X'X$ is nonsingular in OLS?

I am currently working through understanding the mechanics of OLS estimates and the hat matrix. One thing I have been searching for without luck is how we know that the term $X'X$ is invertible where $...
0
votes
1answer
30 views

Choice of deflator in OLS

I am running a pooled OLS model and am not able to internalize change in coefficients/ significance due to change in deflator. The OLS specification is as follows: Market Value = Constant + A1*Profit ...
0
votes
0answers
13 views

Individual level DVs with only Group-level IVs?

This is a pooled cross-sectional design where the IVs are at the district-level and the DV is at the individual-level. I also use district- and time-dummies to control for district-specific ...
5
votes
0answers
44 views

Is a visual estimate of homoscedasticity rigorous enough?

As part of my research in astronomy (quasar magnitudes at various wavelengths), I've been producing graphs such as the following: The bottom plot on each graph shows the distribution of the residuals ...
1
vote
1answer
267 views

What is the relationship of long and short regression when we have an intercept?

Consider the linear model estimated by OLS: $$ y = X\hat{\beta} + \hat{u} = X_1 \hat{\beta}_1 + X_2 \hat{\beta}_2 + \hat{u} $$ We say that the above equation is the long regression, Consider also ...
0
votes
1answer
51 views

Why is it unacceptable to use binary or count dependent variables in OLS?

I know this is a basic question but I really want to make sure I fully understand the reason's why this is the case. If possible, can someone help explain to me as simply as possible why it is bad to ...
1
vote
1answer
116 views

when is it possible to have OLS fits better than random forest and LASSO?

I ran several different models on a mini data set of about 100 observations with 90 features. When I tried OLS with backward selection the model is significant with many features significant (82 ...
10
votes
3answers
518 views

Intuition behind $(X^TX)^{-1}$ in closed form of w in Linear Regression

The closed form of w in Linear regression can be written as $\hat{w}=(X^TX)^{-1}X^Ty$ How can we intuitively explain the role of $(X^TX)^{-1}$ in this equation?
0
votes
0answers
37 views

binary least square classification and labelling

I'm trying to do least square method on a set $X \in \mathbb{R}^{100 \times (2+1)}$ (the $+1$ is for the dummy bias feature) for a classification task on 2 classes (NB: no multiclass) and I found that ...
2
votes
1answer
55 views

Relationship eigenvalues of $X'X$ and $(X'X)^{-1}$

I've a question regarding to eigenvalues, sinve I am not very familiar with the concept. Suppose I've a matrix $X'X$ in the case of an OLS regession. And lets assume that the regarding eigenvalues are ...
0
votes
0answers
252 views

Constrained Linear Regression with coefficients related by inequality

I have a (slightly simplified) model of the following form: $Y=c_1X_1 + c_2X_2 + \epsilon$ subject to the constraint $0\leq c_1\leq c_2$. The distribution of $\epsilon$ is actually not important to ...
0
votes
0answers
17 views

Formulation for multiple regression, but with the bias term taken out and treated separately [duplicate]

Does anyone have a reference to an explicit formulation of multiple regression, but in which the bias term is taken out and treated separately? I would especially be interested if either ridge ...
0
votes
0answers
39 views

Pooled regression vs. average over individual time series models

I have a standard prediction problem in social science where the data set is a pooled data set (i.e., time series of cross sections where each sample can differ and is de-trended, $y_{t+1} = \beta X_t ...
1
vote
1answer
16 views

Parameter errors in linear least squares

I would like to help my (Chemistry) students understand the math behind linear regression.(1) The generally accepted approach for my discipline is to omit any use of calculus and introduce matrix ...
0
votes
0answers
24 views

Enzyme kinetics: Bootstrapping parameter estimates

First things first - this is kind of a mixed problem from biological data analysis, I hope I am in the right place to ask my question(s). Context I have data from a fluorescence assay of binding of ...
0
votes
0answers
39 views

Correlation coefficients and range of residuals

I've generated these plots: The upper two scatter plots show linear regressions of two different combinations of three variables ($W2$, $K$ and $R$), both against $log(z)$, with their ...
2
votes
0answers
184 views

What are the advantages of a random effects model versus a pooled OLS regression with cluster–robust standard errors?

Both models allow for explanatory variables that are time-invariant. I had thought that the advantage of a random effects model might be related to the fact that random effects models mitigate ...
0
votes
0answers
13 views

Measure of error for non-linear systems?

I am trying to model a system that is ideally explained by the formula Q=Qi*(1+Di*b*t)^(-1/b), where: ...
1
vote
0answers
24 views

Confidence interval around weighted sum of regression coefficient estimates?

Let's say you have $N$ random variables $Y_i$, where $Y_i = \beta_i X + \epsilon_i$. $X$ values are the same for all $Y_i$, but the error terms have different variance. I estimate each $\beta_i$ with ...
2
votes
0answers
47 views

divergence of beta estimates between OLS and regression with ARIMA error

I have physiological time-series data: ~60k observations per channel, ~100 Hz sampling. I will model individual channels with ~20 regressors. Under OLS, given temporal autocorrelation in the data, ...
0
votes
0answers
14 views

confusion with frisch lovell type situation

Suppose I am trying to run a regression as follows: $y= a_1+ b_1*x_1 +b_2*x_2 + e_1$ To study the relation between $y$ and $x_2$ net of their correlation with $x_1$, I run the following model first: ...
0
votes
0answers
43 views

How can I find the k-nearest neighbors for a collection of linear time series data?

I need to figure out how to determine the nearest neighbors of an "optimal" line, as illustrated in a simplified figure, linked below. The blue, orange, green, and purple lines represent the best fit ...
1
vote
1answer
33 views

Interpret orthogonal design

I am dealing with orthogonal matrices in regressions, so every regressor has $x_j'x_k=0, if \ j \neq k \\ x_j'x_k=1, if \ \ j=k $ The first one told us that the correlation between two predictors ...
2
votes
1answer
93 views

Ridge/Lasso for correlated response

I want to try a penalised linear regression (ridge/lasso) as a comparison to standard OLS for its predictive ability. My response variable is a continuous measure of an eye parameter, so there is (...
2
votes
1answer
28 views

How can I look for correlations between variables with large deviations?

I'm researching the correlation between the magnitude (a measure of brightness) and redshift ($z$ - a measure of distance) for a variety of galaxies called quasars. Plotting the magnitude against $log(...
1
vote
1answer
37 views

Nonlinear least squares transformation

Suppose that I wish to estimate the parametes $\alpha$ and $\beta$ in the following regression model: $$ Y=K^{\alpha}L^{\beta}\epsilon $$ A standard procedure is to take logs and estimate $$ \text{...
2
votes
1answer
33 views

OLS with Lagged DV

I am interested in building an OLS model with a lagged (lag 1) DV as a right-side explanatory variable. This is relatively straightforward in R, however my problem is the rest of the data. I have six ...
0
votes
0answers
187 views

Multicollinearity in the data with categorical variables

I want to calculate the vif to check for multicollinearity in my data set. I read that a values of >10 tells me that I could have a problem with multicollinearity in my data set. I run an ols ...
0
votes
1answer
49 views

The meaning of coefficients in Multiple Linear Regression

So I am learning about linear regression. The coefficient is the slope of the function, which means how much the dependent variable change due to change of the independent variable. So I make an ...
2
votes
2answers
45 views

Alternative to using $R^2$ to assign data categories?

A background to my problem: I use survey data on firms, where I want to measure the relationship between a binary variable (perceived growth barriers) and firm size. However, I cannot treat "firm size"...
6
votes
4answers
192 views

Is R-squared truly an invalid metric for non-linear models?

I have read that R-squared is invalid for non-linear models, because the relationship that SSR + SSE = SSTotal no longer holds. Can somebody explain why this is true? SSR and SSE are just the ...
0
votes
0answers
19 views

Why does instrument exogeneity imply conditional mean zero?

On slide 14 here: https://www.uio.no/studier/emner/sv/oekonomi/ECON4150/v14/undervisningsmateriale/lecture16_instrumental_variables.pdf it says that "instrument exogeneity implies $E[u_i \mid Z_i]=0$" ...
0
votes
0answers
17 views

How to obtain expressions for coefficients from OLS formula?

Consider the standard linear regression model: $y_i = \alpha + \beta D_i + e_i$ where the coefficients are defined by linear projections and $D_i$ is a dummy variable. In the population, the ...
1
vote
0answers
30 views

Proving First Difference is more efficient than OLS

I am trying to prove that the First Difference method is asymptotically more efficient than OLS when the error term follows a random walk. Assume the following model $$\begin{align}y_i&= \mathbf{...
1
vote
1answer
104 views

Residualizing dependent variable and two step linear regression

Assume we have a DGP of the form $y = \beta_0 + \beta_1 * x_1 + \beta_2 * x_2 + \beta_3 * x_3 + \epsilon$ where $\epsilon$ is a standard i.i.d. error term. Does residualizing $y$ using a linear ...
2
votes
1answer
62 views

Ridge Regression as Robust Optimization

We were told to assume in class that the below optimization formulations are equivalent- $$\min_w\max_{\delta:||\delta||_F\leq\epsilon}||(X+\delta)w-y||_2^2$$ $$\min_{w}||Xw-y||_2^2+\lambda||w||_2^2 ...
7
votes
1answer
767 views

How Ridge or Lasso regression really work?

Very basic question here, but I would like to understand (not mathematically) how the fact to add a "penalty" (sum of squared coeff. times a scalar) to the residual sum of square can reduce big ...
0
votes
0answers
26 views

Linear OLS regression with aggregates and components

A linear OLS regression is specified as Y = a + b*∑(O+R) + c*R + e, i.e. ∑(O+R) is an aggregate and one of the components, R, is added separately. Results for the regression show that both b and c are ...
1
vote
1answer
31 views

Selection of sample on X or Y

In an OLS regression where Y is the dependent variable, X the independent variable and u the error term: Selecting our sample on Y creates a bias: If we have a Y variable that is zero mean and we ...
1
vote
0answers
54 views

Linear Regression When x is Random and Gaussian

Let X denote the design matrix. Our regression is y = $X\hat\beta +\hat\epsilon$. Under the most stringent assumptions i.e. x is assumed to be nonrandom, error terms are iid Gaussian, E[$\epsilon$ | ...
4
votes
0answers
178 views

Prediction with OLS better then prediction with lasso or ridge

I did a regression on a train data set with 7000 observations and 50 explenatory variables with ols ridge and lasso. The lambda was chosen via cross validation. After that i wanted to compare the ...
3
votes
1answer
88 views

Are “feasible generalized least squares” and “iteratively reweighted least squares” the same thing?

These two techniqies seem closely related: Iteratively reweighted least squares (IRLS) Feasible generalized least squares (FGLS) Are the mathematics the same, just different communities (math or ...
1
vote
1answer
396 views

OLS, Fixed effects or Random effects Model?

I am a little bit confused about type of model to apply because my type of data. I am interesting in get regression parameters for time (dependent variable) with independent variables= sex + age+ ...
0
votes
1answer
62 views

MLE with unbalanced system of regressions

I want to estimate the following system of regressions simultaneously: $$ \begin{align} y_1 &=\alpha_1 + \beta\ x_1 + \gamma\ z_2 + \epsilon_1 \\ y_2 &=\alpha_2 + \beta\ x_2 + \gamma\ z_1 + \...
0
votes
1answer
39 views

When use multiple Regression and is linear Regression legit in this case?

I have some trouble understanding the use of multiple regression. I made a survey which has 3 variables (simplified): A, Skill, Money. The participants made choice ...
3
votes
1answer
48 views

Fitted values of a simple regression with intercept and dummy

Why are the fitted values of a simple regression with intercept and dummy, estimated by OLS, just the group means belonging to the two groups of observations? I.e., why do we have that the fitted ...
0
votes
0answers
21 views

Skewed dependend variable, residual assumption violations, appropriate model

I am working with a survey variable which asks respondents to place themselves on a scale from 0 to 10 (integer) (N=1850), where both ends have a specific meaning. Thus, treating the variable as ...
1
vote
1answer
103 views

Why may a matrix be singular or ill-conditioned with standard learning algorithm for linear classification?

In the learning algorithm for linear classification by least square method, which find a weight vector $\hat w\in R^d$ and bias $\hat b\in R$ for a linear scoring function $f(x) = \hat w ^T x +\hat b$ ...
1
vote
0answers
26 views

Formutaion of Least Squares Problem

In general, to use the method of least squares, a linear stochastic system is modeled as: \begin{equation} y = ax + \eta \end{equation} where, $y$, is an observed variable, $x$ is an input while $\...