Questions tagged [least-squares]

Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.

Filter by
Sorted by
Tagged with
1
vote
0answers
13 views

How to prove that optimal solution for ridge regression can be expressed in a following form? [closed]

We know that for ridge regression: $$\beta = (X^TX + \lambda I)^{-1}X^Ty.$$ How can I prove that the optimal solution for beta can be expressed as $$\beta = X^T*V.$$ for some V
0
votes
0answers
17 views

Fitting when variables have different variances?

I am trying to fit a function y = f(x1,x2,x3). The function is non-linear and variables x1,x2, and x3 have different variances. In such a case how do I weigh different variables while performing a ...
0
votes
0answers
7 views

Should there be a transpose in calculating the parameters of SVM

I am trying to code SVM from scratch using a small toy problem that involves five support vector values. In the code below, there are 5 support vectors arbitrary chosen and denoted by the variables <...
0
votes
1answer
13 views

Hausman test fails to reject the H0 (and the instrument is not weak), should I still use the 2SLS based on knowledge that the variable is endogenous?

I have two questions. I am conducting the Hausman test to check the endogeneity of a variable. If the Hausman test fails to reject the null hypothesis, there is no difference between OLS (my reference ...
0
votes
0answers
5 views

How to combine the estimate and confidence intervals of the fitted parameters that you obtain in every run of cross validation?

I am fitting quite a complex model to experimental data points (lots of equations, but just two parameters). The nonlinear least square regression tool (I use fitnlm in Matlab) outputs the estimates ...
1
vote
0answers
26 views

Can I always defend using ols, (for example when my dependent variable is ordinal), if I satisfy all CLM assumptions?

Previous reading Let me first say that I went through this post: (How to determine which distribution fits my data best?) and this post: Assumptions of linear models and what to do if the residuals ...
0
votes
0answers
48 views

Panel Data Regression: Confusion about Random Effects Model

I have a panel data set covering 42 countries over a period of 14 years. My data set contains country sustainability scores (combined, economic, environmental, social) as well as the mean firm-level ...
0
votes
0answers
7 views

Dependent variable inspired independent variable

I have a regression which looks like the standard OLS regression y=Bx+e The independent variable of interest is a continuous variable set by law (think of a tax rate) which I expect to have an effect ...
0
votes
1answer
20 views

Dealing with Skewed Normality for the Standardized Residuals to meet Normality Assumption (OLS)

I am required to build a OLS model. Currently, My model of log(response) against a number of predictors have fulfilled homogeneity assumption (constant variance) and low multicollinearity (based on ...
0
votes
0answers
12 views

Formulating the null hypothesis when the depended variable is a ratio involving the independent variable

I face the following simple regression model: $$ \widehat{rdexp}=\hat{\beta}_{0}+\hat{\beta}_{1}log(sales) $$ where $$ \widehat{rdexp}=\frac{RD}{sales} $$ and $RD$ is some positive number. I am having ...
0
votes
1answer
25 views

Interaction of Gender and Income categorical variable

I ran the following model, with exam scores in science as my outcome variable, and parental income group divided into 5 groups and a binary gender variable with the results below: ...
0
votes
1answer
22 views

Proof of B1by OLS in a multiple regression model with dummies

Given the model Y=B0 + B1C + B2A + e, where C=(0,1) and A=(0,1), how can I derive the formula for the estimator of B1 by OLS? I already know the formula for the general case in which A and C are not ...
5
votes
2answers
76 views

What is the MSE of a model when the true data generating process is known?

Say I fit an OLS model to get an estimate of the coefficients $\hat{\beta}$. For some reason, I also happen to know with complete certainty: That the predictors come from a multivariate normal with ...
0
votes
1answer
56 views

Setting up a linear regression: plant height as a function of time period and soil pH

The following is from Hoff's "A First Course in Bayesian Statistical Methods" book. The data set tplant mentioned above appears below. I am trying to use an ordinary least squares to fit a ...
0
votes
0answers
6 views

standar error of non-linear regression

I'm trying to compute the standar error of a non-linear regression fit, I find the answer in this post Non-linear regression confidence interval But i did't find that formula in any other place,and I ...
2
votes
1answer
34 views

Biased OLS coefficient of endogenous interaction term

Assume we have the following system of equations Eq. 1: $z = x\gamma + \epsilon$ Eq. 2: $y = z\theta + x\beta + \eta$ where the error terms fulfill standard assumptions. Simulating data from Eq. 1 and ...
0
votes
0answers
14 views

statsmodels h-leverage vs leverage terminology

I have been using the python statsmodels api to perform OLS analysis. The API allows you to create several standardized plots. The Influence plot has axes of studentized residuals and h leverage. ...
0
votes
0answers
10 views

Interpretation of WLS estimates

Why are we able to interpret GLS/WLS estimates as though they were regular OLS estimates? For example, I have been studying the use of Feasible Generalised Least Squares to correct heteroskedasticity. ...
0
votes
0answers
14 views

What is the intuition behind the assumption of homoskedasticity in OLS?

I am trying to understand the intuition as to why heteroskedasticity is bad. I have been constantly told that we want to correct heteroskedasticity simply because "we need to satisfy the ...
1
vote
2answers
74 views

If $\hat{e}$ are the OLS residuals, what is random in $\hat{\beta}_{OLS}|\hat{e} = e_0$?

Suppose $y \sim N(\mu, \Sigma)$, where $y \in \mathbb{R}^n$. Let $X \in \mathbb{R}^{n \times p}$ denote a full rank design matrix. By ordinary least squares, the residuals are $$\hat{e} = (I - X(X^TX)^...
0
votes
0answers
29 views

Why does PanelOLS.from_formula from Python returns way more parameters than what the model used for it works with?

I tried to run a panel OLS on a panel data (comprised of 275 time entries for 12 individuals) in Python using: ...
2
votes
1answer
43 views

What are the consequences of “duplicating” a subset of data for OLS?

Suppose I have a sample $\{X_i,Y_i\}_{i=1}^n$. Then the OLS estimator of the slope coefficient is given by $$\hat{\beta}=\frac{Cov(X,Y)}{Var(X)}$$ Now suppose I take my data set and replicate a subset ...
0
votes
1answer
24 views

How to show least square estimator is not BLUE when residual is dependent of each other?

Suppose we have $y=X\beta+\varepsilon$, where $\varepsilon \sim (0,\sigma^2V)$, $\sigma^2$ unknown but $V$ known (we can assume a valid $V$ for this model). Then, by general least square, we can find $...
0
votes
0answers
12 views

Difference between leverage plot, partial regression plot/added-variable plot, and component-plus-residual plot?

What is the difference between leverage plot, partial regression plot/added-variable plot, and component-plus-residual plot? In my intro stats course, I was only taught leverage plots, and not the ...
0
votes
1answer
36 views

OLS with regression with a constant predictor

Imagine for the sake of simplicity that I am regressing $Y$ on $X$ with the model $Y = \beta_0 + \beta_1X + \epsilon$ Now imagine that observations on my $X$ are constant, e.g. take $Y = \{2, 7, 9\}$ ...
0
votes
0answers
14 views

Requesting model for which $E[\epsilon]=0$ and $E[\epsilon|X] \neq 0$

One of the assumptions of OLS is strict exogeneity: $$ E[\epsilon|X]=0 $$ From the law of total expectation, it follows that: $$ E[E[\epsilon|X]] = E[\epsilon] = 0$$ (see the previous link) Can you ...
1
vote
1answer
24 views

Connecting potential outcomes, SUTVA, and regression methods

I'm trying to articulate the differences between identification in regression models and assumptions with the potential outcomes framework. In particular what (if anything) does SUTVA add beyond an ...
0
votes
1answer
33 views

Why strict exogeneity implies that the regressor is uncorrelated with the error term?

One of the assumptions of the classical linear regression model is strict exogeneity, that is: $ E[\epsilon | X ] = 0 $ This should imply that the regressors are uncorrelated with the errors: $E[X^T \...
1
vote
0answers
43 views

For a nonlinear regression task, is either Maximum Likelihood Estimation or Least Squares easier to learn a neural network model with?

I have data (x,y) and I want to create a model f(x) that will best approximate y. Let's ...
0
votes
0answers
9 views

Constrained least squares for standardization of dataset

I have a set of correlation matrices $\mathbf{\Sigma}_{i,j}$ where $i$ is the $i^{th}$ dataset and $j$ is the $j^{th}$ sample of the $i^{th}$ dataset. I am trying to standardize correlation matrix ...
0
votes
0answers
20 views

Variance approximation for large scale least squares problems

The least squares problem I am solving has the objective function $S=\sum_i (y_i-f_i(\textbf{x},\boldsymbol{\beta}))^2$. The variance for this can be approximated as $\text{var}(\beta_j)\approx\frac{S}...
0
votes
0answers
14 views

Y-intercept bias

Have been given a problem with a population model of $Y = Bx + u$ but with a regression model of $Y = a + Bx + u$. Given that the regression model has a Y-intercept, do I assume that a is biased if it ...
-1
votes
0answers
25 views

Predictions with y=ax+b, LMS with a new value of y

I apologize in advance for any mistakes made during this exposition. I am not very proficient in statistics. Recently my professor has given us some lectures about data analysis and these come with a ...
0
votes
0answers
12 views

Should I calculate the real value of net exports using the PPI for a linear regression?

I am using real values for variables such as GDP and Effective exchange rate for my OLS regression I was wondering if I should use the Producer Price index to to find the real value of net exports or ...
1
vote
0answers
9 views

Intuitive explanation behind conditional mean independence?

From what I understand from $E(U|X) = 0$ is that: for every slice of X (i.e. if we fix X, we can get a (normal) distribution of Y values that tend (as the definition of normal distribution) to cluster ...
0
votes
0answers
16 views

Why should covariance of mean of Y and least-squares slope be 0?

I was studying simple linear regression analysis. I was studying the proof of variance of the least squares intercept estimator $\hat \beta_0$ which is in the image. I understand the proof, but can ...
1
vote
2answers
88 views

OLS Loss Function - MLE assumptions?

What are the model assumptions necessary for p(y|x,w) in order for the MLE estimate to provide the OLS loss function? I know that MLE applied to linear regression gives OLS, I'm just not sure what ...
4
votes
2answers
72 views

What does it mean for OLS residuals to be independent from the fitted values?

Let $y \sim N(X\beta, \Sigma)$. The residuals from OLS regression can be written as $$\hat{e} = (I-H)y$$ where $H = X(X^TX)^{-1}X^T$ is the unique projection onto the column space of $X$. The fitted ...
1
vote
2answers
45 views

Intuition About Gradient Descent Convergence

I know that gradient descent takes steps towards a minimum, but I am having trouble coming up with intuitions about when it will converge. For example, on any given convex function is gradient descent ...
0
votes
0answers
28 views

OLS biasedness in AR(1) model [duplicate]

I am trying to show why the OLS estimator in time series models is not conditionally unbiased when using a zero-mean strong AR(1) model. From what I've read so far, this can be done through a Monte ...
3
votes
1answer
73 views

$S_{xx}$ and $S_{xy}$ in linear regression

I am trying to prove that an equivalent way to perform the test for significance of regression in multiple linear regression is to base the test on $R^2$ as follows: to test $H_0:\beta_1 = \beta_2 = ....
18
votes
2answers
1k views

Optimizing OLS with Newton's Method

Can ordinary least squares regression be solved with Newton's method? If so, how many steps would be required to achieve convergence? I know that Newton's method works on twice differentiable ...
3
votes
2answers
68 views

What's wrong with this derivation of the GLS of a linear model?

Question: Find the GLS estimator $\hat{\beta}$ for the linear model $$Y=\beta X+\epsilon$$ where $\mathbb{E}\epsilon=0$ and $\text{Cov}(\epsilon)=\sigma^2\Sigma$, where $\Sigma$ is positive definite. ...
0
votes
2answers
94 views

Why do statisticians often standardize data before performing linear regression?

I am currently reading Regression Analysis by Example by Chatterjee and Hadi and they state "If we are fitting an intercept model as in (3.16), we need to center and scale the variables" (pg....
0
votes
0answers
18 views

Can we use some other norms for solving linear regression?

OLS method is using norm 2 to solve linear regression: $$\mathcal L(w)=\sum_{i=1}^{N}\left(w^{T} x_{i}-y_{i}\right)^{2}=\left\|w^{T} X-y\right\|^{2} \rightarrow \min _{w}$$ Where we try to find the $\...
0
votes
0answers
20 views

Partial F-Test Assumptions

I have a question regarding the underlying assumptions of the partial F-test to compare two regression models: To use the partial F-test, is it sufficient to assume that the assumptions of the normal ...
1
vote
0answers
22 views

Stochastic Gradient Descent - Least Squares [closed]

I am not sure if I implemented the SGD in a proper way since in calculations it gives way to big error even on the training set. Can you help me to figure out where I made a mistake? Here $D$ is the ...
0
votes
0answers
16 views

Coefficient estimates from $(X'X)^{-1}, X'y$ and $\widehat{u'}\widehat{u}$ [duplicate]

How can one calculate the standard scores of the coefficient estimates given $(X'X)^{-1}, X'y$ and $\widehat{u'}\widehat{u}$. Where $(X'X)^{-1}$ is a $n \times n$ matrix, $X'y$ is a $n \times 1$ ...
0
votes
0answers
9 views

Can we derive the Variance of the least squares slope WITHOUT assuming that $X_i$s are fixed or deterministic? [duplicate]

Everywhere in the literature, I have seen that while deriving the variance of the least squares slope estimate $Var(\hat \beta_1) = \dfrac{\sigma ^2}{SS_{xx}}$, we always assume that $X_i$s are fixed ...
1
vote
0answers
28 views

Time Series Chow test & Bai Perron test

I have a question about breakpoints test, that is to select which variables to put in the model ? Example: Y = C(constant) + Ut , Ut ~ N(0,1) ....first model Y = C(constant) + Trend + Ut , Ut ~ N(0,...

1
2 3 4 5
45