Questions tagged [least-squares]

Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.

Filter by
Sorted by
Tagged with
0
votes
0answers
19 views

How does forward stepwise regression select variables

Suppose there are $p = 3$ variables total and suppose the forward stepwise procedure selects the third variable. The forward stepwise procedure will assign it a positive coefficient if and only if $$...
0
votes
0answers
7 views

Are the Generalized Least Square (GLS) and Maximum Likelihood (ML) two different ways of estimation?

I was taught, that OLS and ML are two different ways of estimation. ML gives OLS estimates under met assuptions, but it doesn't change the fact the two approaches differ. If so, how is that possible ...
2
votes
0answers
23 views

Estimating the errors in parameters in the ordinary least square

I am reading the book An Introduction to Error Analysis by John R. Taylor. In Ch8: Least-Squares Fitting, he has derived expressions for parameters $A$ and $B$ in fitting the line $A+Bx$ to the set of ...
1
vote
0answers
8 views

What does it mean to have a model fit via GLS with REML? Aren't GLS and REML two different methods of estimation?

As in the title. I am confused. We often read that a regression model was fit using the OLS, GLS, TLS or ML. But recently I found a text about the analysis of repeated data, where it was modelled ...
-2
votes
1answer
22 views

I am a beginning researcher. I need a textbook or any reading material for step by step research and data analysis [closed]

I want to learn the researching methods and data analysis step by step starting from scratch. starting from the basis statistics to the advance level that includes: Regression, Simple and Multiple. ...
1
vote
1answer
31 views

Why does a Least squares distribution look like a parabola?

For regression analysis, one often uses the least squares method to minimize the quadratic differences between data and a model function f as follows: $$\chi^2=\sum_i \frac{(\text{data}\, _i-f_i(p))^2}...
0
votes
0answers
16 views

How to test normality of errors in multiple linear regression? [duplicate]

In the linear model we make the assumption that the vector of errors follows a multivariate normal distribution: I want to test this assumption for a given data set. How can I do this? I was ...
0
votes
0answers
14 views

Change in the coefficient of determination (R2) when multiplying dependent variables by the independent one in linear regression

I have three independent variables {x_1,x_2,x_3} that I use to fit to a dependent variable y using an OLS regression. The coefficients of determination (R2) of the variables in relation to y are: R2(...
0
votes
0answers
17 views

Modelling Unbalanced Panel Data and Time-Invariant Explanatory variables

I am trying to create a regression model analogous to the following conditions: My dependent variable, $Y$, is unbalanced panel data (Investment flows over varying years for almost all countries) My ...
0
votes
0answers
20 views

Comparing R^2 and Q^2 residuals

I am calculating the Residuals for a PCA algorithm. When I calculate them though the $Q^2$ residuals sometimes are larger than the $R^2$ residuals. My understanding of the $R^2$ and $Q^2$ relationship ...
0
votes
0answers
11 views

Income controlled for different types of education with OLS-regression

This might be a very simple question, but I can't figure it out. I am interested in obtaining the income level controlled for different types of education. Thus, I ran a regression. $$income = b_0 + ...
1
vote
2answers
87 views

Role of data spacing in the x-axis in linear calibrations by least-squares?

What is the role of data spacing along the $x$-axis in linear regression by least-squares? Is it an important criterion to consider in the experimental design of linear calibrations? Linear regression ...
0
votes
1answer
32 views

Any logical meaning behind (1-p value)*coefficient?

Say I have regressions of 3 groups, A, B, and C. I regress some outcome variable Y on an independent variable X. $Y_A = \alpha_A + \beta_A*X_A$ $Y_B = \alpha_B + \beta_B*X_B$ $Y_C = \alpha_C + \beta_C*...
2
votes
1answer
22 views

why in regularisation, we'd assume a dummy basis function?

On p.328 of Introduction to Applied Linear Algebra – Vectors, Matrices, and Least Squares by Stephen Boyd and Lieven Vandenberghe, the authors write that for regularised data fitting, we should ...
1
vote
1answer
21 views

Ordinary least squares estimation explained in “Optimal Design of Experiments” by Goos/Jones, questions about some equationsEq

I am currently trying to become familiar with design of experiments with the book “Optimal design of experiments” by Goos and Jones. In chapter 2, they discuss the use of a Plackett–Burman type design ...
0
votes
0answers
47 views

Understanding a Research Paper's Regression Analysis Equation and Results

I am new to Statistics. I am an undergrad student majoring in Biostatistics in my 2nd Semester and I'm currently interning for a consulting firm for my gap year. I was thrown a research paper to ...
0
votes
0answers
23 views

Consequences of demeaning the independent variable by year or by group/year

I am running a regression on pooled cross section data (five non-consecutive years) as follows, : $$y=B_1NationalEvent + B_2x_2 + u$$ I am trying to evaluate how different groups (A, B and C) respond ...
0
votes
0answers
17 views

Bootstrapping OLS coefficient variance different results than Python Statsmodels

I wanted to test some basic OLS things in python with a sample dataset. One thing I wanted to test was just getting a bootstrap estimate of the variance of the model's coefficients. Here is the setup ...
0
votes
1answer
34 views

Curve fitting - multiple indepedent fits or a single combined fit?

Assume, I have a list of actual, noisy, independent scalar measurements [y1, y2, ..., yn] (think: y1 = [0%, 12%, 52%, 79%, 99%]) for a scalar series ...
1
vote
0answers
18 views

Inference in a system in which (almost) all regressors are endogenous

We conducted a number of tests on individual species over a period of several days. Let $x_{k,s,t}$ denote the result of test $k$ for species $s$ at time $t$. In total there are $S$ species, $K$ tests,...
0
votes
1answer
18 views

Time invariant variable in a panel data model

I have a question about a time invariant variable in a panel data model. As we know, we can estimate the coefficient of a time invariant regressor using pooled ordinary least squares (OLS) or a random ...
0
votes
0answers
16 views

How do you explain the condition under which a simple OLS regression model can be used to yield the causal effect of a treatment? [duplicate]

So, this is a question from an econometric tutorial and I am simply confused (and nervous) about this question. Is an assumption the same as a condition? What kind of example can I give that best ...
3
votes
1answer
45 views

Predictions in a system in which (almost) all regressors are endogenous

Consider the following system where all variables are endogenous. \begin{align*} x_{1} & =\beta_{21}x_{2}+\beta_{31}x_{3}+u_{1}\\ x_{2} & =\beta_{12}x_{1}+\beta_{32}x_{3}+u_{2}\\ x_{3} &...
5
votes
4answers
439 views

Order of variables in R's lm

I dont quite understand the answer given in Order of variables in R lm model in lm() function of R (and generally formulas) why changing the order of variable ...
1
vote
1answer
72 views

Pseudo-inverse matrix for multivariate linear regression

In Andrew Ng's Machine Learning course lecture 4.6 on "Normal Equation", he says that in order to minimize $J(\theta) = \frac{1}{2m}\sum\limits_{i=1}^{m}({h_{\theta}}(x^{(i)}) - y^{(i)})^2$, ...
0
votes
1answer
24 views

OLS Regression > Reverse Causation

I am interested to examine the effect on consumption on marijuana with the implementation of tax. I will be examining it on 51 states in US including DC, over 6 years. The other controlled variables ...
0
votes
1answer
52 views

Econometrics: How to Derive the OLS Estimators of a log-log model

Consider the following two simple linear model specifications: $log(y) = \beta_0 + \beta_1log(x) + u$ $(1)$ $log(y/x) = \alpha_0 + \alpha_1log(x) + v$ $(2)$ where y and x are two random variables for ...
3
votes
2answers
20 views

terms in a simple linear least square model

I'm reading a textbook. In the chapter about least square regression I red that A simple linear least square model can be described as \begin{equation} Y = \alpha + \beta x + e \end{equation} where Y ...
0
votes
1answer
22 views

Proof that the coefficient on a binary predictor in a simple OLS is equal to difference in means of the outcome for that predictor?

In an univariate least squares regression the regression equation is given by: y = a + bx where b is the slope coefficient of ...
0
votes
0answers
24 views

Design synthetic data for regression problem that give access to the true risk value

Consider the following linear regression model: $Y = X \beta + \epsilon$ Let $\hat{\beta}$ be the OLS estimator. It verifies the normal equation i.e $\langle X, X\hat{\beta} - Y\rangle = 0 \iff \hat{\...
1
vote
0answers
48 views

Unbiasedness of estimators of conditional expectation with discrete dependent variable

I'm trying to figure out whether the basic formula for a conditional expectation with discrete conditioning variable (let's call it $X$). The basic argument can go something like the following: a ...
0
votes
0answers
24 views

What is the inverse of a sample mean?

My professor derived the OLS estimates for $\beta$ as following: He started from $\beta = E[(\mathbf{X^TX})]^{-1}*E[\mathbf{X^TY}]$. Since the sample mean is a non-biased estimator of any expected ...
0
votes
1answer
26 views

Linear Models Hat matrix

For OLS in matrix form, we are taught that Hat matrix is $X(X^TX)^-X^T$, and is idempotent etc, i.e. when it multiplies with itself, it will self cancel and thus lead back to the same Hat matrix. I ...
0
votes
0answers
5 views

Pretest posttest design with varying time between tests

I have a pre-test post-test design where several subjects were administered a drug. For each subject, I have $p_0$, a physiological measurement taken directly prior to the drug being taken, and $p_t$, ...
2
votes
1answer
71 views

OLS coefficients of regressions of fitted values and residuals on the original regressors

Let $\gamma$ and $\delta$ denote $K\times 1$ vectors of parameters in models $\hat{Y} = X\gamma+\eta$ and $\hat{\epsilon} = X\delta+\psi$, where $\eta$ and $\psi$ and $n\times 1$ vectors of error ...
0
votes
1answer
36 views

OLS coefficient comparison during Chow test

I'm looking for feedback for a design issue. Say we have a simple OLS model like Y = X1 + X2. Then say we scale all vars by Z, such that the model is Y/Z = X1/Z + X2/Z. Now make a slight change to the ...
0
votes
0answers
5 views

Would the exogeneity assumption be violated with year as a variable?

Let's suppose that I have a model. I have made this model up for simplicity purposes. Death Rate= b0 + b1(gdp_per_capita) + b2(year) + error I know that the exogeneity assumption for a linear ...
3
votes
1answer
55 views

Advantage of within transformation over pooled OLS with dummies?

First, apply the within transformation (fixed effects transformation) on a panel data set. Then, apply pooled OLS with dummies for each cross-sectional unit on the same panel data set. When you ...
1
vote
0answers
13 views

How to numerically solve for a variant of the weighted least squares

I encountered the following problem, and I don't have a good idea on where to start. Suppose that we have the following weighted least square problem: $$ \hat{\beta}_{WLS} = \arg\min_\beta(y - X\beta)^...
1
vote
0answers
48 views

OLS estimator is consistent if the smallest eigenvalue of $X^TX$ goes to infinity as $n\to\infty$

I want to show that if $\lambda_{min}(X^T X)$ (i.e., the smallest eigenvalue of $X^TX$) goes to infinity as $n\to\infty$, then $\hat{\beta}$ is a consistent estimator of $\beta$. My approach is the ...
2
votes
1answer
35 views

OLS variance estimator in linear regression without strict exogeneity

(I don't remember seeing this result stressed enough.) Consider the "benchmark" linear regression model $$\mathbf{Y} = \mathbf{X} \beta + \varepsilon.$$ $$E(\varepsilon) = 0,\, E(\varepsilon ...
0
votes
1answer
34 views

From LEAST SQUARE METHOD to Pearson cofficient and determination coefficient

the following procedure has the scope to describe how, from the Least square method, I obtained the Pearson coefficient and the determination coefficient. At the end of the procedure I have several ...
1
vote
0answers
31 views

Consistency of an estimator [closed]

I have an estimator for the coefficients of the model $$ y=X\beta+\varepsilon $$ with $y_{n\times1}$, $X_{n\times p}$, $\beta_{p\times1}$, $\varepsilon_{n\times1}$. The estimator is in the form $$ \...
0
votes
0answers
23 views

the expected value of Autoregressive model with linear trend

In the stationary time series like the following with intercept and AR(p) model, the expected value is the intercept / (1-AR(1)-AR(2)) = mean ##B0 = alpha(intercept) Then, I include a trend to data ...
0
votes
0answers
8 views

Conceptual difference between STRESS and SSTRESS

I am studying the difference between STRESS and SSTRESS optimization functions in the context of multidimensional scaling. I have encountered this definition of optimization criteria for the SSTRESS: $...
0
votes
0answers
5 views

orthogonal projection in an algorithmic procedure for LARS

To my best understanding, the steps for Least Angle Regression are as follows: ...
2
votes
1answer
37 views

Estimating mean in the presence of serial correlation

Consider the following generating equation: \begin{equation} X_{d+1} = a X_d + b + {\cal E}_d \end{equation} where $a$ and $b$ are constants with $0 <a < 1$ and $b > 0$. Further let ${\...
0
votes
0answers
17 views

RSS of ridge regression in terms of OLS estimator

In the work by Hoerl, Arthur E., and Robert W. Kennard , "Ridge regression: Biased estimation for nonorthogonal problems." the following formula (3.1) is presented: $$ RSS=(Y-XB)'(Y-XB)=(Y-X\...
0
votes
0answers
27 views

How to choose between pooled ols and fixed/random effect

I have unbalanced panel dataset, large N and relatively small T (about 15 timesteps). I know that the ols completely disregards the panel structure of the data. I was wondering if there is a test that ...
0
votes
0answers
58 views

Simple linear regression with skewness, kurtosis and heteroscedasticity

I have several issues with a very simple linear regression. I cannot get Skewness/Kurtosis and Homoscedasticity assumptions to be met, even after removing outliers, adding polynomial terms and using ...

1
2 3 4 5
43