Questions tagged [least-squares]

Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.

Filter by
Sorted by
Tagged with
0
votes
0answers
11 views

PRESS from the hat matrix and numerical stability from statsmodels ols.fit()

Leave one out cross validation in the context of ordinary least squares regression can be done via the hat matrix: The "hat" or projection matrix $$ H = X(X^T X)^{-1} X^T $$ many fit ...
2
votes
1answer
22 views

Relation between OLS, MM and ML

What is the relation between OLS, MM (method of moments) and ML (maximum likelihood)? During my studies, the three concepts got taught completely separated from each other. However, they seem to be ...
2
votes
1answer
30 views

In OLS regression, when the assumption of normally distributed residuals is rejected, is bootstrap (and block bootstrap) the way to deal with it?

In OLS regression, when the assumption of normally distributed residuals is rejected, is bootstrap (and block bootstrap) the way to deal with it? Is this the right way to go or non-normally ...
1
vote
0answers
8 views

Curious results with t statistic with GARCH errors in a linear regression

So, I was playing around with an odd specification in a simulation experiment: \begin{align} y_t &= x_t \beta + \sqrt{h_t} \epsilon_t \\ h_t &= \sigma^2 + \pi \left( h_{t-1} - \sigma^2 \...
1
vote
0answers
9 views

Question about resulting decision boundary in a classification task

I have 1000 data points from the bivariate normal distribution $\mathcal{N}$ with mean $(0,0)$ and variance $\sigma_1^2=\sigma_2^2=10$ with the covariances being $0$. Also there are 20 more points ...
0
votes
0answers
30 views

Square loss for “big data”

Let’s set up a supervised learning problem with $p$ predictors and $n$ observations. The response variable is univariate. The problem can be regression or classification, though I think a ...
3
votes
1answer
63 views

If an OLS model is estimated without an intercept (no constant term) but the average residual is close to zero, are we OK?

Let's suppose an OLS model is estimated without an intercept (no constant term), but the mean residual is very close to zero (2.2E-11) does that mean the model is OK to have been estimated without a ...
0
votes
0answers
23 views

OLS regression problem in R, where intercept is function of the other parameters [duplicate]

I'm having problems doing OLS in R using the lm() function on the following linear model: $Y_t = \bar{Y} \cdot (1-a-b-c) + a \cdot X_{1t} + b \cdot X_{2t} + c \cdot ...
1
vote
0answers
19 views

Question about heteroscedascity and point estimates

Let's say I am running a regression: $y_{j}=\beta x_{j} + \eta_j$ and $var(y_i|x_i)=var(\eta_i|x_i)=f(x_i)$, say the variance is increasing in x. assume $\eta$ is fully independent of x, so ...
0
votes
0answers
14 views

Regression : Variable transformation necessary? If yes, why?

I am working on a real estate project and have historical rental prices and vacancy data. Interested in exploring the relationship between vacancy rates and change in rental prices. The unit of ...
0
votes
0answers
23 views

Optional least-squares assumption: e_i|x ~ N, what does it do?

Among the least-squares assumptions, the optional assumption that e_i|x~N is not required but is used to derive the estimated coefficients. I was wondering what will happen to these estimated ...
0
votes
0answers
9 views

CNLRM: Testing linear restriction - F statistic in terms of regression residuals of the unrestricted and restricted regressions

In order to test linear restriction Rβ=q of the Classical Normal Linear Regression we use F=1/p(Rβ ̂-q)'(〖σ ̂^2 R(X^' X)^(-1) R^')〗^(-1) (Rβ ̂-q)~F(p,n-k) How can I write this F statistic as F_0=((ε ̃^...
0
votes
0answers
28 views

Least Squares Regression to Solve a Non-Linear System

I'm trying to solve the following non-linear system; $\sqrt{(x-x_1)^2 + (y-y_1)^2} + s(t_2-t_1) = \sqrt{(x-x_2)^2 + ( y-y_2)^2}$ $\sqrt{(x-x_2)^2 + (y-y_2)^2} + s(t_3-t_2) = \sqrt{(x-x_3)^2 + ( y-y_3)^...
0
votes
1answer
19 views

Subtracting a constant from the OLS summation

On a proof for the OLS of $\beta$, I have seen this step: $\sum x_i (y_i - \alpha - \beta x_i) = \sum (x_i - k) (y_i - \alpha - \beta x_i) $ for any constant $k$. Why is this true?
2
votes
1answer
38 views

What is the difference between least squares line and the regression line?

It is common to plot the line of best fit on a scatter plot when there is a linear association between two variables. One method of doing this is with the line of best fit found using the least-...
0
votes
0answers
22 views

Multiply beta coefficients from two different models

I have a count outcome with a heavy right skew that is modeled with a negative binomial. I have a continuous mediator that is modeled with OLS. We're attempting a method of causal mediation analyses ...
0
votes
1answer
17 views

Existence of least squares and maximum likelihood estimators?

In statistical parameter estimation where there is a deterministic and stochastic component to the observation-generating model, do least squares and maximum likelihood estimators always exist? ...
0
votes
0answers
12 views

The OLS and CSS deterministic trend with AutoRegressive Model

I use two method to get the coefficient of AR(1)+trend Model, one of the method is "CSS" and another "OLS". First i generate a time series ar(1) with trend and mean = 10. ...
0
votes
0answers
12 views

How do you check whether heteroscedastic-consistent estimators fix your error variance problem?

I am running an OLS regression, and have non-constant error variance (residuals vs fitted looks like a fan opening up to the right). I have tried a number of power transformation but they seem to make ...
2
votes
1answer
50 views

Interpreting Estimated Coefficients of Linear Regression

I have data that requires interpretation of the below coefficients. Description of variables: "region" = the beneficiary’s residential area in the US; a ...
0
votes
0answers
10 views

In multivariate regression, under what condtions is $var(X_i\epsilon_i')$ positive definite?

Suppose we have $(Y_i, X_i)$, with $Y_i$ an r.v. in $\mathbb{R}^k$ and $X_i$ an r.v. in $\mathbb{R}^p$ and suppose the covariance matrix of $X_i$, $E(XX')$ is positive definite. Now we can estimate ...
0
votes
1answer
32 views

Why is the Dual Formulation a valid reparametrization of a regression model

In polynomial regression problems, in which an input vector $\underline{\phi}(\underline{x})$ is used to map a feature vector to a higher dimensional space (an example of this being $(x_{1}, x_{2}) \...
0
votes
1answer
27 views

CLT with inconsistent estimator

So I have the OLS estimator that is inconsistent due to the mean independence assumption being violated. I'm asked whether $\sqrt{n}(\hat{\beta}-\beta)$ converges when the sample size $n$ goes to ...
5
votes
2answers
274 views

least square estimator of regression x onto y [duplicate]

I've been reading linear regression and least square estimator. Suppose we have i.i.d data $(x_1, y_q), (x_2, y_2), ..., (x_n,y_n)$ such that we use a linear regression model $y_i = \beta x_i + \...
1
vote
0answers
53 views

Robust Covariance in Multivariate / Multi-response OLS

Assume we are in the OLS setting with $y = X\beta + \epsilon$. When $y$ is a response vector, and $X$ are covariates, we can get two types of covariance estimates: The homoskedastic covariance $cov(\...
1
vote
0answers
20 views

Non linear least squares with non vertical residuals

Disclaimer: my scholar math level is quite basic, and english is not my native language, so please forgive me for the words I will use. I have a point cloud (~50000 points) and I would like to fit a ...
1
vote
1answer
42 views

what does the normality assumption in OLS and glm imply

I am a bit confused about the normality assumption of the error term in linear regression models. Several textbooks write that one of the Least Squares assumptions is that the (conditional) ...
0
votes
0answers
12 views

How does the Zeiger-McEwin & Kung algorithm work for fitting a sum of exponentials?

I am trying to understand this paper fit sum of exponentials but am having a bit of difficulty. Let me go through what I have understood so far. One has a certain time-series data set and want's to ...
1
vote
0answers
24 views

Using a standard OLS linear regression or a panel model for small sample size

I have created a dataset involving certain NLP variables for news-websites over time. The data involves six news-websites which are observed over 27 time periods with some missing data. As I only have ...
0
votes
0answers
12 views

Estimate $β$s and $σ^2$ using Least squares and newton's method for model $y_i=β_1 e^{-β_2 t_i+β_3 t_i^2}+ϵ_i$

The model: $y_i=β_1e^{(-β_2 t_i+β_3 t_i^2)}+ϵ_i$, and $ϵ_i $ is $ N (0,σ^2) $ iid how to estimate above parameters $β_1,β_2,β_3$ and $σ^2$ using Least square and newton's rule? the optimization should ...
0
votes
1answer
26 views

Derivation of the conditional variance of OLS

I was reading the notes by Matthew Blackwell from lecture 10 of his Quantitative Research Methodology course and ran into a couple moments that I do not understand: 1. How does he go from line 2 to ...
1
vote
0answers
18 views

What is so special about the least norm solution in case of an undetermined system of equations

In particular this is the go to approach in case of solving a least squares problem that lacks a unique solution, how does being the closest point to the origin among all the solutions make it any ...
0
votes
0answers
14 views

Logistic regression and balanced treatment groups [duplicate]

When using logistic regression how critical is to have balanced treatment groups. I have a binary response variable (pregnant and non-pregnant) from two treatments at n=40 and n=14? How does the need ...
0
votes
0answers
35 views

Least Squares for projection matrix estimation produces unwanted behaviour

I am estimating a non-linear correspondence between 3D and 2D points using scipy.optimize.least_squares. I found that setting the method to be Trust Region ...
1
vote
1answer
39 views

What is the correct way to write the model equation for a linear probability model?

I'm trying to write down the equation describing a linear probability model. If I was writing out the equation for an OLS model with continuous y with observation unit i , I would write: $$y_i = \...
0
votes
0answers
21 views

distribution of least squares estimates for random design regression

Suppose we consider the least squares problem where the objective is to find $\beta$ that minimizes $(\boldsymbol{y} - \boldsymbol{X} \beta)^T (\boldsymbol{y} - \boldsymbol{X} \beta)$. We know that an ...
1
vote
2answers
60 views

Confused with the fundamental assumptions of Frequentist and Bayesian Linear Regression

In Frequentist Linear Regression, I have seen 2 approaches which lead to basically similar models. We have $W,y,X,\epsilon$ related as $y=W^TX+\epsilon$, where $y$ is the dependent random variable, ...
1
vote
0answers
33 views

replace variable in a linear model with new variable with same covariance that yields the same least-sqares parameter estimate

Consider the following linear model, which explains the relation between a $d$-dimensional set of explanatory variables $\{\mathbf{X},D \}$ and a 1-dimensional effect variable $Y$ ($\{\mathbf{X},D \}$ ...
1
vote
1answer
25 views

Why are interactions harder to estimate at high values of the modulator?

Suppose you have the following model, where all variables are continuous: $$y = \alpha + \beta_0x_0 + \beta_1x_1 + \beta_2 x_0 x_1 + \epsilon$$ The standard error for the effect of $x_1$ is $$\text{se}...
0
votes
1answer
24 views

If I add an extra variable to a regression model I already have, and the R squared increases, does this usually mean the new model is better?

Lets say I'm estimating a model with ordinary least squares, and that my initial model (with an $R^2$ of 0.5) is $$ y_i = \beta_o + \beta_1x_{i1} + \beta_2x_{i2} + \varepsilon_i $$ Lets say I add a ...
0
votes
0answers
13 views

Fitting OLS spread in ARIMA AR (1) process

'm a newbie to econometrics. I've simply ran a regression and have coefficient values of the variables. I'm running a regression for a crypto data, and I've gotten the Spread of the variables. To ...
0
votes
1answer
66 views

How to derive the covariance matrix between $\bar{y}$ and $\hat{\beta_c}$ where $\hat{\beta_c}$ is the OLS estimator of a linear model?

$$cov\begin{bmatrix}\bar{y}\\\hat{\beta}_c\end{bmatrix}=\sigma^2\begin{bmatrix}\frac{1}{n} & 0^T\\0 & (X_c^TX_c)^{-1}\end{bmatrix}$$ with $\hat{\beta}_c=(X_c^TX)^{-1}X^T_cy$. I am supposed to ...
0
votes
0answers
15 views

What exactly is explained sum of squares, and why do we focus on minimizing error rather than maximizing explained variability?

In the third of my questions about decomposing the total sum of squares, I want to focus on the sum of squares of the regression. I can make sense of what the sum of squares of the residuals means: ...
0
votes
3answers
36 views

Why does pre-summing variables lead to a different OLS fit?

Suppose I have an OLS model like this: $$y = \beta_1x_1 + \beta_2x_2 + \epsilon$$ If you sum the variables first, $x_3 = x_1 + x_2$ and fit $$y = \beta_3x_3 +\epsilon$$ I expect that $\beta_3$ is the ...
0
votes
1answer
24 views

Fitting a Survival Function with OLS - What Points To Use?

I'm trying to fit a survival function that is generated from discrete data. Is it correct to use Ordinary Least Squares, and if so what points do I use to do the fit? A, B or C? Or other? Or am I ...
0
votes
0answers
13 views

Compare goodness of fit for GAMs and (multiple) ordinary least squares

I'd be super grateful for any help. I have a dataset with a continuous response variable and various predictors. I want to fit two models. A "simple" linear multiple least squares regression....
1
vote
0answers
28 views

The similarity between Mallows Cp and AIC?

It is possible to compute the log-likelihood used for AIC as $n /log(RSS/n) + const$ or even as $RSS/\sigma^2 + n\log(\sigma) + const$ considering the least-square or MLE scenario for linear and non-...
39
votes
6answers
7k views

Why don't linear regression assumptions matter in machine learning?

When I learned linear regression in my statistics class, we are asked to check for a few assumptions which need to be true for linear regression to make sense. I won't delve deep into those ...
0
votes
1answer
33 views

Angle between $\hat{y}$ and $y$ stays the same as $\lambda$ in ridge regression is adjusted

I was given a thought experiment a while back to think about, but I haven't been able to come up with a solution. The question is For some dataset $X$ with response $Y$, you apply ridge regression. ...
1
vote
0answers
18 views

Can I use logistic regressions and least squares coefficients in a meta-analysis?

I am trying to write a Economics meta-analysis. The papers that I have collected so far contain a 50/50 split between OLS models and Logistic regressions (coefficients reported as odds ratios). If we ...

1
2 3 4 5
42