Skip to main content

Questions tagged [least-squares]

Refers to a general estimation technique that selects the parameter value to minimize the squared difference between two quantities, such as the observed value of a variable, and the expected value of that observation conditioned on the parameter value. Gaussian linear models are fit by least squares and least squares is the idea underlying the use of mean-squared-error (MSE) as a way of evaluating an estimator.

Filter by
Sorted by
Tagged with
0 votes
1 answer
44 views

Linear model between data with standard error

I have two set of data from a experiment. I would like to evaluate a linear model between both sets. However, the term corresponding to y variable is composed by two elements: The measure and its ...
handelsarache's user avatar
1 vote
0 answers
51 views

Is Fisher's discriminant analysis equivalent to the Bayes optimal LDA when the no. of classes is greater than two and covariances are all equal?

P.S. While I gave a brief background to make the question complete, informed readers can move to the questions 1 and 2 towards the end of this post, right after 'what are not clear to me are:'. Fisher'...
Mathmath's user avatar
  • 761
0 votes
0 answers
50 views

Diff in Diff: wrong control group or wrong method?

I want to identify the causal effect of renewable energy targets on the environmental policy stringency index (I got it from OECD) for EU countries. My hypothesis is that by setting a renewable energy ...
giulio artemio's user avatar
1 vote
0 answers
34 views

Decision boundary for Cross entropy loss and Least square loss

We can see the source in this paper. My question is that why cross entropy loss has a boundary line in slope but least square loss has horizontal boundary. Can somebody explain?
batuman's user avatar
  • 483
0 votes
0 answers
44 views

Ordered Probit with 20+ categories

Is there any upper limit on categories I can use for the dependent variable in an ordered probit model? In my current model I have at least 20 categories, but I maybe require more (up to 50). Is this ...
JorgenM's user avatar
1 vote
1 answer
80 views

Unbiased and constistency of OLS

For the linear regression $y_t = Bx_t+e_t$ where we have the assumptions: $E(e_t)=0$, $E(e_t^2) = \sigma^2$, $E(e_t e_s)= 0$ for $s\neq t $ ...
WNZ's user avatar
  • 21
1 vote
1 answer
53 views

How can I test if the OLS slope is different from a expected slope? [duplicate]

I am currently testing if an OLS slope is statistically different from a known, expected slope. I think I would need to do a one-sampled t-test but I am not exactly sure how. Specifically, is the ...
Le Wang's user avatar
  • 11
0 votes
0 answers
40 views

Calculus versus matrix representation in OLS

In the Wikipedia article Ordinary Linear Squares there is an example for finding the estimators $\beta_i$ for a linear model of the sort: $$y_i = \beta_0 + x_1\beta_1 + x_2\beta_2 + \ldots$$ In the ...
Minsky's user avatar
  • 135
0 votes
1 answer
54 views

Sampling distribution of ordinary least squares confusion

I was reviewing the derivation for the variance of ordinary least squares estimators and experienced some confusion. $$ \Large Var(\hat{\beta}) = \frac{\sigma^2}{\Sigma^n_{i=1}(X_i-\bar{X})^2} $$ From ...
APerson's user avatar
0 votes
0 answers
11 views

What's the point of using gradient descent for linear regression if you can calculate the coefficients directly using the least squares method? [duplicate]

Gradient descent involves significant computational effort, whereas the method of least squares enables direct and accurate calculation. Does gradient descent offer any advantages over least squares ...
Dawid's user avatar
  • 1
3 votes
0 answers
20 views

Missing assumption in MA estimation

Assume we observe a $MA(1)$ process for which it is known that the mean is zero. Based on a series of length $3$ , we observe $Y_1 = 0, Y_2 = 1$ and $Y_3 = 0.5$. Find the least-squares estimate of $\...
Kilkik's user avatar
  • 445
0 votes
1 answer
63 views

Least squares method for ordinal variables

Does the least squares method for regression assume that the regressors and response are continuous? In particular, I would like to fit a regression line for ranks. Can I use the least squares method ...
Mewbacca's user avatar
1 vote
1 answer
147 views

After selecting variables from lasso regression, is it a good practice to re-run the regression with selected variables?

As the subject suggests, after selecting regressors from lasso regression, is it a good practice to re-run the an ordinary linearly regression with selected variables? I just feel like intuitively, ...
Taylor Fang's user avatar
1 vote
2 answers
149 views

Why don't standard estimation methods consider the distribution of the residual?

Consider a linear model in which a vector of data $\mathbf{d} \in \mathbb{R}^M$ is related to an unknown parameter of interest $\mathbf{x} \in \mathbb{R}^N$ via $$\mathbf{d} = \mathbf{A}\mathbf{x} + \...
Snea Git's user avatar
4 votes
1 answer
142 views

How to prove OLS residual regressions mathematically

Problem Statement So I read in Elements of Statistical Learning, p. 54 that another way of doing OLS for multiple predictors is the following way. Let's assume two predictors $X_1, X_2$ and a target ...
Joe's user avatar
  • 271
0 votes
1 answer
110 views

Slope describe the affect of X on Y in the regression model

I am confused about the rule of the slope in the simple linear regression model. I do understand that the slope indicate the rate of change in the response variable, Y, when the explanatory variable, ...
Alice's user avatar
  • 650
10 votes
2 answers
589 views

Negative R2 on Simple Linear Regression (with intercept)

I am doing a simple Linear Regression (with intercept) which ends up presenting a negative R2, this should not be possible (cf comment 2 at the end) Reproducible examples of the issue: Minimal ...
Jean Lescut's user avatar
0 votes
1 answer
47 views

Difference between SEM AND ols+pca

Can someone explain me the difference between these approaches, if you want i can provide the results, but since they are quite extensive, i could attach on demand. I'm working with the Theory of ...
Jota's user avatar
  • 51
2 votes
0 answers
126 views

Relationship between conditional expectation and regression

I would be grateful if you could help me clear up some confusion regarding conditional expectation and regression. I have seen two formulations of the linear regression framework: $$Y=a+bX+\varepsilon\...
abeeisnotabug's user avatar
0 votes
0 answers
19 views

Instrumental variable identifiability in the presence of unobserved confounders

Long story short, I'm seeing in the literature that linear instrumental variables models are identifiable, even in the presence of unobserved confounders. The unobserved confounding aspect befuddles ...
mortonjt's user avatar
  • 347
2 votes
1 answer
179 views

Why does this multiple linear regression fail to recover the true coefficients?

I am trying to use linear least squares regression to extract the coefficients of a model. Specifically, I am looking at a model with two independent predictor variables $x_1$ and $x_2$, and an output ...
teeeeee's user avatar
  • 169
3 votes
2 answers
44 views

Known coefficient in multiple linear model

Lets say my model is : $y=\beta_0+\beta_1x_1+\beta_2x-2+\beta_3x_3$ Now lets say I know for sure that $\beta_2 = 4$. My teacher said I should create $y’ = y-4 X_2$ and ordinary least squares (ols) ...
WalaWizon's user avatar
  • 103
0 votes
1 answer
245 views

Can I apply PCA to combine correlated variables into one variable? [closed]

I have a dataset with 11 observations and 11 features. I want to use linear regression for estimating the coefficients by using OLS method. I know it is not advisable to use linear regression with ...
Davie Blain's user avatar
2 votes
1 answer
234 views

Are normal errors required for OLS with a large sample size?

It's surprisingly difficult to find a clear answer to this online. Nearly every online source claims that error terms must follow a normal distribution for statistical inference but stops short of ...
Timoffex's user avatar
0 votes
1 answer
67 views

How to reconcile these two matrix equations for obtaining the coefficients for a linear least squares fit?

In ordinary least squares linear regression, given a set of data points $(x_1,y_1),(x_2,y_2),...(x_N,y_N)$, that we want to fit to the function $y=\beta_0 + \beta_1 x$, we would usually write the ...
teeeeee's user avatar
  • 169
0 votes
0 answers
25 views

The introduction of a new variable made my coefficient of interest flip sings : what can I say about correlation?

I am currently studying the relationship between academic freedom (independant variable) and university rankings (dependant variable) using OLS. Each individual is a university, and my variables (...
Muller I. 's user avatar
4 votes
1 answer
1k views

Why feature scaling does not affect prediction output in regression?

I was modelling a linear regression (OLS) and tried using scaling techniques on the predictor variables. I could see the range of the variables change, however the prediction results remain the same. ...
Patrick Priyadharshan's user avatar
1 vote
0 answers
61 views

ARDL OLS assumptions?

can someone direct me to a paper etc. that states the assumptions of ARDL that need to hold for parameters to be valid? ARDL is an OLS based model , does that mean the regular assumptions of Gauss-...
Gus's user avatar
  • 25
0 votes
0 answers
25 views

Can I use log-log OLS with no constant to measrue the relationship between two variables in a non-normal distribution?

I am trying to measure the relationship between the price of a futures-contract and the current price of the underlying asset on the day of purchasing said futures-contract. For this I have a dataset ...
Statauser's user avatar
1 vote
2 answers
176 views

Regression with sample split

I have a question with respect to running multiple linear regressions for the entire sample and different subsamples: I have a dataset that includes a dependent variable y and several explanatory ...
derhard's user avatar
  • 11
1 vote
0 answers
31 views

Regression Coefficients

I am performing regression analysis based on a full sample and two sub samples from the full sample. Is it always the case that the coefficient of the full sample lies between the coefficients of the ...
Osei Kwabena Brefo's user avatar
0 votes
0 answers
57 views

Fitting a Vertical Line to Points on a Plot

I have the following data, which produced the following plot. ...
David Moore's user avatar
1 vote
0 answers
137 views

Least squares more efficient than maximum likelihood?

I have synthetic data which is sampled from a non-central chi distribution (similar to what is obtained experimentally). I am fitting a non-linear model to this data to extract three parameters of ...
user2551700's user avatar
5 votes
2 answers
163 views

Is there sense to be made for two-tailed T-test and OLS of two variables?

Let's say we start off by generating 2 random uniform variables between 1 and 100 ...
Dean MacGregor's user avatar
1 vote
0 answers
26 views

Least Squares Regression with Length and Density of input data

I have a collection of data that I expect to be linear but has a unknown amount of noise to the data. Initially I wanted to use the least squares regression line to determine if the slope, y axis, and ...
Caleb Laws's user avatar
1 vote
0 answers
22 views

Prediction of Multiple Linear Regression With Constant

Let $X$ be a matrix with $n$ rows and $d$ columns. We know that there exists matrices $U, S, V$, with $U$ of dimensions $(n, d)$, $S$ of dimensions $(d, d)$ and $V$ of dimensions $(d, d)$, which form ...
user35083's user avatar
0 votes
1 answer
56 views

Statistical Properties of OLS Estimators (Unbiasedness)

In deriving the unbiasedness of OLS Estimators, $$\hat{\beta_1} = \beta_1 + \frac {\sum_{i=1}^{n} (x_i - \bar{x})u_i}{\sum_{i=1}^n (x_i - \bar{x})^2}$$ My professor changes the above to: $$\hat{\...
rudinable's user avatar
1 vote
0 answers
39 views

Propagation of uncertainty in a derivative of a function [duplicate]

I've performed an ordinary least squares on a data set with one variable. For simplicity, let's say I've fitted a polynomial function $$f(x)=a+bx+cx^2+dx^3.$$ I obtain the best fit and the standard ...
Bert's user avatar
  • 11
2 votes
0 answers
57 views

Understanding diagonal rescaling in multiplicative update rules for NMF

SUMMARY How does the diagonal rescaling fit into the derivation of a multiplicative update rule for non-negative matrix factorization (NMF)? DESCRIPTION The NMF problem aims to find non-negative ...
scho's user avatar
  • 21
2 votes
1 answer
122 views

relation between R-squared and sample size

Here is the question: Suppose $X, Y$ are independent $N(0,1)$ random variables. And take the regression of $Y$ against $X.$ What is the relationship between $R^2$ and sample size approximately? ...
user6703592's user avatar
  • 1,345
1 vote
0 answers
10 views

How do I create a simulated variable to check whether coefficient size arises from technicality?

I am running a regression of y on four variables, two of which are binary variables and two of which are discrete variables which range from 0 to 10. For context, y was data collected from an ...
princesskaguya666's user avatar
1 vote
0 answers
10 views

pooled cross section data

I want to find input demand elasticities using a cost function. Input quantities and input prices are available for individual farmers for 5 food crops and 5 years (2015-2019). But farmers may vary ...
Jevon's user avatar
  • 11
1 vote
1 answer
49 views

Is this model linear function belongs to GLM models?

In GLM, we assume that $\mathbb{E}[Y|X]=\mu(\beta^\top X)$ and $Y|X$ follows exponential family distribution. I am going to assume that the probability of success in the Bernoulli distribution is ...
Amin's user avatar
  • 693
2 votes
1 answer
185 views

What are the minimum conditions needed for the consistency of OLS estimator in the following linear regression model?

Suppose $Y_i=X_i'\beta+\epsilon_i$ with $E(\epsilon_i|X_i)=0$. Consider the usual OLS estimator for $\beta$ using a random sample $\{X_i,Y_i\}_{i=1}^n$: $\widehat{\beta}=(\frac{1}{n}\sum_{i=1}^nX_iX_i'...
ExcitedSnail's user avatar
  • 2,966
0 votes
0 answers
60 views

Classical linear regression with no independent sample

I like to organize my studies always with the weakest hypotheses possible. In this case, I want to understand well what assumptions I should add to be able to study linear regression models in time ...
André Goulart's user avatar
0 votes
1 answer
133 views

Estimating exponent of Zipf distribution using MLE vs fitting linear regression on log-transformed rank and frequency data

I'm having trouble understanding why I get radically different results if I try to find the parameter of a Zipf distribution when I use the methods proposed by Clauset et al. (2009) as opposed to ...
MarcoLin8's user avatar
2 votes
0 answers
71 views

How to determine if my model is robust? Should the coefficients be same?

I want to run robustness tests for my model. For example, by reducing the sample to heavily concentrated groups, running a different regression (probit etc) etc. But, how do I ascertain that my ...
Laiy's user avatar
  • 245
1 vote
0 answers
18 views

Does min-norm least squares solve regular least squares in some basis?

For a data matrix $X$ of dimension $n \times p$ where $p > n$ and corresponding label vector $y$ of dimension $n$, the standard least squares fit, $\hat{\beta} = (X^TX)^{-1}X^Ty$, is ...
Seraf Fej's user avatar
  • 556
1 vote
0 answers
96 views

OLS with $iid$ Cauchy errors still unbiased?

A comment to this question suggests that the OLS estimate of linear model parameters is unbiased, even when the error term is Cauchy. Given that Cauchy distributions lack an expected value, I am ...
Dave's user avatar
  • 65k
2 votes
1 answer
172 views

Why would bootstrap OLS standard errors differ from ML estimate?

Let's say I have a regression dataset (paired x and y) such that the response variable (y) has an unknown distribution (but definitely not Gaussian) and is large enough such that the central limit ...
David Wang's user avatar

1
3 4
5
6 7
58