Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
Join us in building a kind, collaborative learning community via our updated Code of Conduct.

Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least ...

0
votes
0answers
10 views

How can I find the k-nearest neighbors for a collection of linear time series data?

I need to figure out how to determine the nearest neighbors of an "optimal" line, as illustrated in a simplified figure, linked below. The blue, orange, green, and purple lines represent the best fit ...
1
vote
1answer
15 views

Interpret orthogonal design

I am dealing with orthogonal matrices in regressions, so every regressor has $x_j'x_k=0, if \ j \neq k \\ x_j'x_k=1, if \ \ j=k $ The first one told us that the correlation between two predictors ...
0
votes
0answers
27 views

Ridge/Lasso for correlated response

I'm wanting to try a penalised linear regression (ridge/lasso) as a comparison to standard OLS for it's predictive ability. My response variable is a continuous measure of an eye parameter - so there ...
0
votes
1answer
14 views

Keeping the intercept in an OLS model [on hold]

There are lots of questions that deal with keeping versus dropping the intercept in an OLS model, and most of them conclude never to remove the intercept (such as here, here and here, among many ...
2
votes
1answer
25 views

How can I look for correlations between variables with large deviations?

I'm researching the correlation between the magnitude (a measure of brightness) and redshift ($z$ - a measure of distance) for a variety of galaxies called quasars. Plotting the magnitude against $log(...
1
vote
1answer
26 views

Nonlinear least squares transformation

Suppose that I wish to estimate the parametes $\alpha$ and $\beta$ in the following regression model: $$ Y=K^{\alpha}L^{\beta}\epsilon $$ A standard procedure is to take logs and estimate $$ \text{...
2
votes
1answer
19 views

OLS with Lagged DV

I am interested in building an OLS model with a lagged (lag 1) DV as a right-side explanatory variable. This is relatively straightforward in R, however my problem is the rest of the data. I have six ...
-1
votes
0answers
13 views

R: Representing RSS as ellipse equaiton [duplicate]

since I am not the best in algebra I didnt find a solution to get from the RSS equation to: $RSS=||y-X\beta||_{2}^{2}=(\beta-\hat\beta)X'X(\beta-\hat\beta )$ Can someone help me to find the solution ...
0
votes
0answers
27 views

Multicollinearity in the data with categorical variables

I want to calculate the vif to check for multicollinearity in my data set. I read that a values of >10 tells me that I could have a problem with multicollinearity in my data set. I run an ols ...
2
votes
0answers
27 views

Can the maximum-likelihood method be derived from something else?

I am an author of a paper, in which we show that the maximum-likelihood (ML) method can be derived a limiting case of an iterated weighted least-squares fit. https://arxiv.org/abs/1807.07911 We, the ...
0
votes
1answer
26 views

The meaning of coefficients in Multiple Linear Regression

So I am learning about linear regression. The coefficient is the slope of the function, which means how much the dependent variable change due to change of the independent variable. So I make an ...
0
votes
0answers
24 views

EDA and cross-validation suggest no interaction but OLS suggests otherwise

Say that we have two variables, $S$ is a score that is used to predict your $I$, an income. We have two groups in our dataset, $A$ and $B$. Members of group $B$ are expected to have higher income. ...
2
votes
2answers
36 views

Alternative to using $R^2$ to assign data categories?

A background to my problem: I use survey data on firms, where I want to measure the relationship between a binary variable (perceived growth barriers) and firm size. However, I cannot treat "firm size"...
5
votes
3answers
125 views

Is R-squared truly an invalid metric for non-linear models?

I have read that R-squared is invalid for non-linear models, because the relationship that SSR + SSE = SSTotal no longer holds. Can somebody explain why this is true? SSR and SSE are just the ...
0
votes
0answers
11 views

Why does instrument exogeneity imply conditional mean zero?

On slide 14 here: https://www.uio.no/studier/emner/sv/oekonomi/ECON4150/v14/undervisningsmateriale/lecture16_instrumental_variables.pdf it says that "instrument exogeneity implies $E[u_i \mid Z_i]=0$" ...
0
votes
0answers
13 views

How to obtain expressions for coefficients from OLS formula?

Consider the standard linear regression model: $y_i = \alpha + \beta D_i + e_i$ where the coefficients are defined by linear projections and $D_i$ is a dummy variable. In the population, the ...
1
vote
0answers
18 views

Proving First Difference is more efficient than OLS

I am trying to prove that the First Difference method is asymptotically more efficient than OLS when the error term follows a random walk. Assume the following model $$\begin{align}y_i&= \mathbf{...
1
vote
1answer
28 views

Residualizing dependent variable and two step linear regression

Assume we have a DGP of the form $y = \beta_0 + \beta_1 * x_1 + \beta_2 * x_2 + \beta_3 * x_3 + \epsilon$ where $\epsilon$ is a standard i.i.d. error term. Does residualizing $y$ using a linear ...
2
votes
1answer
34 views

Ridge Regression as Robust Optimization

We were told to assume in class that the below optimization formulations are equivalent- $$\min_w\max_{\delta:||\delta||_F\leq\epsilon}||(X+\delta)w-y||_2^2$$ $$\min_{w}||Xw-y||_2^2+\lambda||w||_2^2 ...
6
votes
1answer
649 views

How Ridge or Lasso regression really work?

Very basic question here, but I would like to understand (not mathematically) how the fact to add a "penalty" (sum of squared coeff. times a scalar) to the residual sum of square can reduce big ...
0
votes
0answers
20 views

Linear OLS regression with aggregates and components

A linear OLS regression is specified as Y = a + b*∑(O+R) + c*R + e, i.e. ∑(O+R) is an aggregate and one of the components, R, is added separately. Results for the regression show that both b and c are ...
1
vote
1answer
31 views

Selection of sample on X or Y

In an OLS regression where Y is the dependent variable, X the independent variable and u the error term: Selecting our sample on Y creates a bias: If we have a Y variable that is zero mean and we ...
1
vote
0answers
53 views

Linear Regression When x is Random and Gaussian

Let X denote the design matrix. Our regression is y = $X\hat\beta +\hat\epsilon$. Under the most stringent assumptions i.e. x is assumed to be nonrandom, error terms are iid Gaussian, E[$\epsilon$ | ...
4
votes
0answers
39 views

Prediction with OLS better then prediction with lasso or ridge

I did a regression on a train data set with 7000 observations and 50 explenatory variables with ols ridge and lasso. The lambda was chosen via cross validation. After that i wanted to compare the ...
3
votes
1answer
30 views

Are “feasible generalized least squares” and “iteratively reweighted least squares” the same thing?

These two techniqies seem closely related: Iteratively reweighted least squares (IRLS) Feasible generalized least squares (FGLS) Are the mathematics the same, just different communities (math or ...
1
vote
1answer
73 views

OLS, Fixed effects or Random effects Model?

I am a little bit confused about type of model to apply because my type of data. I am interesting in get regression parameters for time (dependent variable) with independent variables= sex + age+ ...
0
votes
1answer
31 views

MLE with unbalanced system of regressions

I want to estimate the following system of regressions simultaneously: $$ \begin{align} y_1 &=\alpha_1 + \beta\ x_1 + \gamma\ z_2 + \epsilon_1 \\ y_2 &=\alpha_2 + \beta\ x_2 + \gamma\ z_1 + \...
0
votes
1answer
29 views

When use multiple Regression and is linear Regression legit in this case?

I have some trouble understanding the use of multiple regression. I made a survey which has 3 variables (simplified): A, Skill, Money. The participants made choice ...
2
votes
1answer
30 views

Fitted values of a simple regression with intercept and dummy

Why are the fitted values of a simple regression with intercept and dummy, estimated by OLS, just the group means belonging to the two groups of observations? I.e., why do we have that the fitted ...
0
votes
0answers
18 views

Skewed dependend variable, residual assumption violations, appropriate model

I am working with a survey variable which asks respondents to place themselves on a scale from 0 to 10 (integer) (N=1850), where both ends have a specific meaning. Thus, treating the variable as ...
0
votes
1answer
48 views

Why may a matrix be singular or ill-conditioned with standard learning algorithm for linear classification?

In the learning algorithm for linear classification by least square method, which find a weight vector $\hat w\in R^d$ and bias $\hat b\in R$ for a linear scoring function $f(x) = \hat w ^T x +\hat b$ ...
1
vote
0answers
24 views

Formutaion of Least Squares Problem

In general, to use the method of least squares, a linear stochastic system is modeled as: \begin{equation} y = ax + \eta \end{equation} where, $y$, is an observed variable, $x$ is an input while $\...
12
votes
2answers
712 views

Is there any theoretical problem with averaging regression coefficients to build a model?

I want to build a regression model that is an average of multiple OLS models, each based on a subset of the full data. The idea behind this is based on this paper. I create k folds and build k OLS ...
1
vote
0answers
14 views

Partial Residual in OLS or Lasso case

I have a similar question as in this this post. Assume I've a regression $y_i=\beta_1x_1+\beta_2x_2+\beta_3x_3+\epsilon_i$ And the partial residual is defined as: $r_i^{(3)}=y_i-\beta_1x_1+\...
3
votes
1answer
21 views

How to perform Least Squares with constraints on a subset of the model coefficients?

For solving an unconstrained LS regression $$\hat{y}=w_1.x_1+w_2.x_2+w_3.x_3+w_4.x_4 + \epsilon$$ I use the following normal equation: $$W^*=(X^{\top}X)^{-1}X^{\top}Y $$ If I want to introduce a ...
2
votes
1answer
74 views

Linear regression with multiple y values per x and their arithmetic means

I noticed that if there are multiple values of $y$ for each value of $x$, I can replace the values of $y$ with their arithmetic mean at each value of $x$ and still get the same regression model (...
0
votes
0answers
10 views

OLS with shares as outcome

I have a regression where my outcome are share measures within a household (share of underweight for example) and am pondering whether this is a case of censored regression? My intuition is to think ...
0
votes
0answers
32 views

Least squares - why multiply both sides by the transpose? [duplicate]

The formula: $A^T(b - Ax) = 0\\ A^Tb = A^TAx\\ x = (A^TA)^{-1}A^Tb$ What is the reason for multiplying both sides by the transpose of A?
1
vote
0answers
92 views

Find RMSE from StatsModels OLS Results

I playing around with some regression analyses in Python using StatsModels. I am getting a little confused with some terminology and just wanted to clarify. I have run a regression and get the ...
3
votes
3answers
110 views

When is there a difference between a normal likelihood loss and a least squares loss?

My understanding is that if the errors follow a normal distribution, then using a maximum likelihood loss or a least squares loss to train a model amounts to the same thing. However, I am looking at ...
0
votes
0answers
13 views

Interpretation of coefficient of an index whose value lies between 0 and 1

I am running a simple ordinary least squares regression to understand the effect of parental attitude ($PA$) on Math scores of children. $Math Score_{i}= \beta_{0} + PA_{i}\beta_{1} + Controls + \...
7
votes
3answers
64 views

Converting the beta coefficient from matrix to scalar notation in OLS regression

I've found for my econometrics exams that if I forget the scalar notation, I can often save myself by remembering the matrix notation and working backwards. However, the following confused me. Given ...
1
vote
1answer
31 views

OLS and Probit possible on large sample enough?

I think I understood that normality of residuals may not be a problem if the sample is large enough (cf, here). My question is: Would my sample be large enough to be analysed using a probit and an ...
1
vote
1answer
38 views

Sum of squared residuals and MSE

It seems that minimizing the sum of squared residuals (SSR) in linear regression is equivalent to minimizing MSE (both use true value - prediction) and OLS is the best estimator for minimizing SSR. I ...
0
votes
0answers
15 views

Least Squares with coefficient constraints

I need to perform a least squares problem with constraints on some (but not all) of my coefficients. For example say I am fitting the following model: $$\hat{y} = \beta_0 + \beta_1 x_1 + \beta_2 x_2 ...
3
votes
1answer
33 views

Correlation between the j-feature and partial residual

I am reading a book for the lasso-regression. However, I think that this is a more general question regarding the derivative of the OLS-term: Can someone explain me why this $c_j$ term is ...
2
votes
0answers
13 views

How to interpret A VAR model, Method: OLS

I am working on a VAR using an OLS model as can be seen below. I am however having difficulties interpreting the results. I think to understand the basic concept of an equation being significant below ...
1
vote
0answers
8 views

Learning rate: Normalised Leat Mean Squares

I was wondering why when performing NLMS regression the learning rate $\eta>0$ for the following update rule: $$w_t=w_{t-1}-\frac{\eta}{1+\eta||x_t||_2^2}(x_t'w_{t-1}-y_t)x_t$$ If I set $\eta<...
0
votes
0answers
12 views

Test for Homoskedasticity and endogeneity in Panel data with Pooled OLS estimation

Which tests are valid to test for: 1) Homoskedasticity 2) Endogeneity in a Pooled OLS model built on Panel data?
2
votes
0answers
26 views

Kernel and regularization parameter of James–Stein estimator

Consider a FIR model of the form $y= Ug_0+e$ with $e$ white noise with variance $\sigma^2$. We assume that we have collected N input-output measurements $y$ and $U$. The James–Stein estimator is ...