Questions tagged [least-squares]

Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.

Filter by
Sorted by
Tagged with
1
vote
1answer
26 views

Regressor contribution in OLS regression

Assume I have the following model, estimated using OLS: $Y_{it}=β0+β1∗X1_{it}+β2∗X2_{it}+β3∗X3_{it}+ϵ_{it}$ I know that some methods exist to compute the relative contribution of each variable to the ...
0
votes
1answer
66 views

Why is $E(\hat{\beta}_1) = \beta_1 + 1$ also an unbiased estimator of true $\beta_1$

let regression model be $$Y_i = \beta_0 + \beta_1X_i +u_i$$ If the ordinary least square estimator of $\beta_1$, $\hat{\beta}1 = \frac{\sum^n_{i=1}(x_i -\bar{X})(y_i-\bar{Y})}{\sum^n_{i=1}(x_i -\bar{X}...
0
votes
0answers
4 views

On the use of FEs in repeated cross section

I have a repeated cross-section of state representative household surveys in the United States (20 years). Note the data is not longitudinal. For each year, the data was collected on a brand new ...
0
votes
0answers
21 views

Least square estimation and nonlinear model

Suppoese X is the attribute and Y is the response as random variables. We observe (X,Y) jointly as a bi-variate normal variable, then least square estimation of Y as a regression function ...
1
vote
0answers
19 views

Derive least squares solution [duplicate]

I have the equation $$ R_{\text{emp}}(w) = \frac{1}{2N} \lVert \Phi w -y \rVert ^2 = \frac{1}{2N} \sum_{i=1}^N \left( w^T \phi(x_i)- y_i \right)^2 $$ and am supposed to derive the least squares ...
0
votes
0answers
19 views

Efficiency of GLS over OLS, proving through semi-definite and positive semi-definite

JHi all! Just browsing through some materials on proving efficiency of GLS over OLS, vice versa, and one of the methodology was to compare their respective variances, that for OLS to be more efficient,...
0
votes
1answer
33 views

OLS Regression equation [closed]

This question is primarily for my understanding; say we have a regression equation of the form Y = Xb, where X is a matrix of a few explanatory variables. If you are told that the vector 'b' does not ...
1
vote
2answers
27 views

Independence between variance and regression coefficients

In a regression model with 3 regressors, how can I prove that $\hat{\beta_1}-\hat{\beta_3}$ and $s^2$ are statistically independent? My idea is to start with $Cov((\hat{\beta_1}-\hat{\beta_3}), s^2)=...
4
votes
1answer
76 views

Zero Covariance vs Independence of Slope and Intercept Estimators in Linear Models with Least Squares

$\newcommand{\Cov}{\operatorname{Cov}}$Problem Statement: Under the assumptions of Exercise 11.16, find $\Cov\big(\hat\beta_0,\hat\beta_1\big).$ Use this answer to show that $\hat\beta_0$ and $\hat\...
1
vote
1answer
22 views

Bias-variance trade-off in case of biased estimators: is the bias zero?

Consider a data generating process (DGP) that is AR(1): $y_t=\varphi_1 y_{t-1}+\varepsilon_t$ with $\varepsilon_t\sim i.i.D(0,\sigma^2)$ for some distribution $D$ with mean zero and variance $\sigma^2$...
0
votes
0answers
41 views

Is it possibe to find an estimator for variance of interest using least squared estimation in a linear model?

In a linear model, maximum likelihood estimation (MLE) provides estimator for both mean of insterest and variance of interest, but least squared estimation (LSE) just provides estimator for the mean ...
2
votes
0answers
34 views

Parameter estimation in linear regression

This is a regression problem. Let $r_i=f_iu_i+e_i,i=1,...,n$, where $u_i\sim^{i.i.d}N(0,1)$, and $e_i=O_p(\alpha_n),\alpha_n\rightarrow 0$ as $n\rightarrow 0$. When I take the log transformation of $...
0
votes
0answers
35 views

Understanding difference between Maximum Likelihood and Levenberg Marquardt result

In some of my regression results I noticed a deviation between Maximum Likelihood (via Monte Carlo Markov Chain, initialised by parameter result of Nelder-Mead, median value pictured) result and ...
0
votes
1answer
44 views

How do we interpret the "natural log of income"

I'm just confused as to how we interpret a natural log in relation to a regressor, say years of training thanks
4
votes
0answers
41 views

What is scikit-learn's LinearRegression doing when there are more features than observations? [duplicate]

I'm trying to understand what sklearn's LinearRegression (which should be using ordinary least squares) is doing when there are more features than observations. ...
1
vote
0answers
18 views

What are the calculations or maths behind least-squares-minimizing in linear regression used by sklearn [duplicate]

I'm relatively new in the ML field, and this question came up when working with linear regression from sklearn library. After a bit of looking up in the documentation, it states Compute least-squares ...
2
votes
1answer
77 views

What if the prior not be conjugate with posterior in Bayesian learning?

I know that when the prior is conjugate with posterior then one can get an analytical representation for the posterior distribution, but what if these two are not to be conjugate? For example, I would ...
1
vote
1answer
28 views

OLSR model: high negative correlation between 2 predictors but low vif - which one decides if there is multicollinearity?

Date, age, mrt and shops are all predictors in a dataset of 414 observations. Pearson's product-moment correlation shows a sizeable negative correlation between mrt and shops (-0.6 so definitely ...
1
vote
0answers
35 views

Unit of measure in the OLS regression

Suppose I estimate by OLS the following linear model $$ Y_i=\beta_0+\beta_1 X_i+U_i $$ where $Y_i$ denotes the weight of individual $i$ (in pounds), and $X_i$ denotes the height of individual $i$ (in ...
0
votes
1answer
24 views

Is there a simple generalization of the OLS "weighted average of the slopes" interpretation of the slope coefficient to multivariate predictors?

The slope coefficient of univariate (single predictor, single response) OLS can be written as a weighted average of the pairwise slopes between $(x,y)$ pairs, specifically: $$ \hat{\beta} = \frac{\...
1
vote
0answers
19 views

How can I find the uncertainty of derivatives? [duplicate]

Suppose I have a quadratic (weighted) least-square fit result obtained from a given set of data: $$ f(x) = \underbrace{-0.243(\pm0.3324)}_{quad\_a}x^2\underbrace{-0.921(\pm0.061)}_{quad\_b}x\...
0
votes
1answer
15 views

Cross-validation for Ordinary least squares

I want to do a ordinary least-square (OLS) fit to points (no error bars, they are not measurements) to find linear coefficients. I know that the model is imperfect, and want to quantify the model ...
0
votes
0answers
10 views

Change in coefficient β in log linear model

I'm studying econometrics with Wooldridge's manual. In a problem from Chapter 2, you have to prove that the coefficient β1 does not change when you transform a simple linear regression into a log-...
4
votes
0answers
103 views

Fitting a Logistic Regression via Brier Score or Mean Squared Error

Is there a name for a logistic regression model that has been fit using the Brier score (or equivalently the mean-squared error) rather than the cross-entropy? I realise this isn't maximum-likelihood, ...
1
vote
0answers
35 views

Solving a probabilistic optimization problem similar to (constrained) least squares

A simple constrained least squares problem could be the following: For a given matrix $A \in \mathbb{R^{m,n}}$ find the vector $v \in \mathbb{R^n}$ that $$\text{minimizes: }\quad \| Az \|_2^2 \\ \...
0
votes
0answers
13 views

Same results with Linear Regression and Lasso

I have fit the data with Linear Regression model, and OLS regression results look like below: Here is how these insignificant betas look like: When I apply Lasso to the same data, I get almost the ...
0
votes
0answers
24 views

What transformation to apply before linear regression?

I'm trying to remove the autocorrelation from a time series with linear regression based on past values. But as this plot shows (y is 1-step head values averaged ...
2
votes
1answer
105 views

Using OLS over Logit regression

I was hoping someone could clarify this for me. I have used OLS regressions using the PISA dataset where most of my outcome variables are categorical in nature and measured on 4 point Likert scales, ...
1
vote
0answers
23 views

Prediction from linear regression when R2 is low

I would like to ask you: What is the releationship between low R2 (let's say 0,15) and user's ability to predict outcome values of Y given input values if X. I've read that the low R2 is not ...
0
votes
0answers
47 views

No significant interaction in OLS, but significant in pairwise comparison

I have two independent categorical variables — (i) A: {1, 2, 3}, (ii) B: {0, 1} — with a continuous outcome variable ...
0
votes
0answers
27 views

Literature on residual analysis

I am looking for literature (papers, books, etc) that could have information about residuals analysis on non-linear regression. I have done a residual analysis in python following the MATLAB Guides ...
0
votes
0answers
16 views

Alternative for multicollinearity problem

I am a bit unsure how to specify my model and therefore look for some references / ideas. I am interested in a model along the following lines: $pay_i = \alpha + \gamma X_i + \beta pop_i + \epsilon_i ...
2
votes
0answers
23 views

Least square fit for data with uncertainty [closed]

I have a set of data $\{x_i,y_i\}$ that came from my measurement, so each of them has some uncertainty. I wonder is there a method to fit them using a model (like Gaussian), while taking care of those ...
0
votes
0answers
23 views

Linear regression encoding for categorical variable

Suppose we have a variable with $k+1$ categories, which we can decompose it with $k$ binary variables: $$x_i = \begin{cases} 1, & \text{category i occurs} \\ 0, & \text{otherwise} \end{cases}$...
0
votes
0answers
16 views

Least squares with non-orthogonal basis

Suppose I have a set of basis functions $f_i(x)$, $i = 1, \dots, n$ which are linearly independent but not orthogonal. I want to fit some data $(x_j,y_j)$, $j = 1, \dots, m$ with $m > n$ to the ...
2
votes
3answers
365 views

Why use MSE instead of SSE as cost function in linear regression?

I am studying linear regression and I solved some problems analytically. For that I used the normal and intuitive sum of squared error function. Looking at this function, it makes all sense why it ...
2
votes
2answers
57 views

Matrix representation of the OLS of an AR(1) process,

Is there any precise way to express the OLS estimator of the centred error terms $\{u_t\} _{t=1}^{n}$ that follows an AR(1) process? In other words, for \begin{equation} u_t=\rho u_{t-1}+\varepsilon_t,...
0
votes
0answers
10 views

Partial regression plot from multiple regression - residual analysis

For the most part, the residuals seem normally distributed and linear model seems appropriate for the data that I am trying to fit. However, for one independent variable, they don't look normal and ...
0
votes
0answers
23 views

How to implement the OLS regression model on my data

I want to find out if economic sanctions effect the educations system of sanctioned countries. There are some studies that have worked already on providing evidence that sanctions negatively effect ...
0
votes
1answer
26 views

Problem dealing with OLS estimator

I'm an econometrics student and I'm having a little trouble with lineal algebra. I have seen that the OLS estimator, given the following regression in matrix form: $$ y=X \beta+u $$ Is: $$ \hat{\beta}=...
1
vote
1answer
17 views

We know beta for univariate OLS; what can we say about beta after switching the independent and dependent variables?

I was asked this in an interview, and I'm curious if (1) my reasoning was correct and (2) if I could have been more precise in my answer. Consider standard OLS with a single predictor: $$ \mathbf{y} = ...
1
vote
1answer
60 views

Invertibility of $E(X^\top X)$ and $\tilde{X}^\top \tilde{X}$

Consider the linear regression model with 3 regressors $$ Y=\beta_1 Q+\beta_2 W+\beta_3 Z+\epsilon $$ Let $X\equiv (Q, W, Z)$ and $\beta\equiv (\beta_1,\beta_2,\beta_3)^\top$. Also suppose that $Q,W,...
0
votes
0answers
19 views

minimize sum of MSEs under orthogonality constraint

Suppose we want to minimize the objective $L=\frac{1}{2}||y-X\beta_1||^2 + \frac{1}{2}||y-X\beta_2||^2$ over $\beta_1, \beta_2$, under the constraint $\beta_1^T\beta_2=0$. Is this equivalent to first ...
2
votes
1answer
30 views

Where does this formula for the OLS estimate as a function of the true parameters come from?

Suppose that in the population: $$ y = \alpha + \beta_1 x_1 + \epsilon $$ We now estimate the model: $$ \hat{y} = \hat{\alpha} + \hat{\beta_1} x_1 $$ I have seen the following formula which writes the ...
0
votes
0answers
10 views

Writing OLS estimates as a function of real parameters in case of multivariate regression

Suppose that in the population: $$ y = \alpha + \beta_1 x_1 + \epsilon $$ We now estimate the model: $$ \hat{y} = \hat{\alpha} + \hat{\beta_1} x_1 $$ If we try to estimate $\beta$ using OLS, we have ...
1
vote
0answers
48 views

Differences between least-squares estimate and weighted least-squares estimate when I focus on only one coefficient in a multiple linear regression?

Suppose we have $n$ samples from the following linear model: $$y=x_1 \beta_1+x_2 \beta_2 + e,$$ where $e_i$ i.i.d. comes from $N(0,\sigma^2)$; $x_1$, $x_2$ are centralized with mean $0$. I am only ...
2
votes
2answers
42 views

If a fitted OLS regression model is mis-specified, is it possible to produce a second model that is unbiased?

Let’s say I want to build a linear regression model to conduct some sort of statistical inference. I plan on using the least squares method to fit the model. My understanding is that you need to ...
0
votes
0answers
15 views

understanding R2 in probit

I try to create a model to predict football (socker) results with a performance variable. It doesn't really matter how this performance is calculated since any performance variable is an adequote ...
0
votes
0answers
13 views

Weak orthogonality, consistency, and unbiasedness of the OLS estimator

My question is based on this question. Suppose we assume the sample is iid (so time series data is out) and $E[e_i X_i ] = 0$ but we're not sure about $E[e_i \mid X_i]$ = 0. Can you provide a ...
2
votes
1answer
49 views

Efficiency of IV vs GMM

I am trying to understand how IV/just identified GMM and overidentified GMM compare when it comes to efficiency. The way I understand it, we are able to identify the vector of coefficients in IV and ...

1
2 3 4 5
48