# Tag Info

## New answers tagged least-squares

1

I agree with Patrick's comments above. I found the following articles useful which highlight that removing independent variables related to multicollinearity will improve the model output and this can be performed using a loop. https://beckmw.wordpress.com/2013/02/05/collinearity-and-stepwise-vif-selection/ Why is multicollinearity not checked in modern ...

0

Here is a great graphical representation of why SST = SSR + SSE.

1

In the scenario you're thinking of, non-negative least squares is still convex in $x$, though it may or may not be convex in something else. Let's consider some possibilities. There are some supplied input variables, and some derived input variables that are (typically nonlinear) functions of the basic variables, including products $a_1a_2$ and other ...

3

i can only answer this question in a confirmatory way. Normally, many ppl in ML have huge datasets with lots of features and rows or only a few features from kaggle with a moderate amount of rows. What is common to most ppl regardless of the datatset is, they dont derive a hypothesis or work out material. They see it as an EDA and want to confirm their ...

5

TNT-NN For the TNT-NN see: Myre, Joe M., et al. "TNT-NN: a fast active set method for solving large non-negative least squares problems." Procedia Computer Science 108 (2017): 755-764. https://doi.org/10.1016/j.procs.2017.05.194 To form the active set, TNT-NN first solves an unconstrained least squares problem. Variables that violate the non-...

0

This is an artifact of the model used for logistic regression, which is model for one of the two probabilities, which add to 1. If your two values are coded, eg, 0 and 1, then linear regression assigns a positive coefficient to any variable, x, that increases in value as y shifts from 0 to 1. So regression assumes that y increases from 0 to 1. The logistic ...

5

I don't think an entire data presentation is needed to give some intuition behind this phenomenon. While we would expect that a logistic regression and OLS model will, on average, produce parameter estimates (slopes or log-odds ratios) that are similar sign, it's entirely possible they will disagree for a given dataset and analysis. The most likely issue is ...

2

Without seeing your code, it is hard to spell out the difference in results. But it sure is possible to get the same results in either package, as - as you correctly point out - all three commands ultimately just run OLS regressions. It is with different degrees of ease, though, reflecting the purpose of the packages. lm is, of course, for all sorts of ...

0

(hi, im noob but "Is this acceptable in a model to have differenced variables and variables stationary and why" (for me, yes, because : ols requires stationarity, just because of mathematics inside of it like "constant mean" if inflation is stationary its ok if another vector (oil) is not stationary, you use diff and when its stationary ...

1

No there is nothing from that output that could have told you about the possibility of model misspecification. The formal way to see if there is misspecification (test if the model is indeed nonlinear) would have been to run a "Ramsay reset test". This mean to estimate the following: $$1: y = a_0 + a_1 x + e$$ calculate: $$\hat{y} = a_0 + a_1 x ... 0 Replying to this since I also got confused from the description in the textbook and it took me a bit to figure out the argument: Suppose x is the current solution (at some iteration of the algorithm) with active set P (i.e., x_i >0 for i \in P and 0 otherwise). Here is what happens afterwards. Let w = C^\top (d-Cx) the current Lagrange dual ... 2 As an overarching framework, indeed MLE is much better. It shows when you really want to use LSE, and when you want to use other estimators, such as LAE, Poisson regression, logistic regression, WLS, GLS, etc. It also leads naturally and seamlessly to Bayesian methods. All quantitative disciplines would be much better off to replace LSE with MLE. These days,... 0 It depends. You would want to know if the variables in the regression are stationary, and if not, take differences until they are. OLS on non-stationary variables will generally be invalid. I say 'generally' because it is actually only the error term that needs to be stationary, but usually the way to address the risk of non-stationary error terms is having ... 0 Answer 1 (simple): The simple answer is that \beta is the sensitivity of the response y with respect to the regressor X (assuming the relationship is true). Its estimator \hat \beta, which is what you quoted, is the result (the slope) of "forcing" (i.e. fitting) a line (consisting of an intercept and slope) to go through all the points, ... 1 Normality is actually quite important. Not in the sense that it must be true, because it never is true, but in the sense that with gross non-normality you should not use OLS, despite asymptotically correct inferences. For example, with grossly outlier-prone processes (substitute "rare, extreme value" for "outlier" to disentangle it from &... 0 I uderstand how it works. If there is a constant value in the array, so formula of centered model will be used. For example centered formula will be used when we have array in x axes like: [ [5, 1], [3, 1], [4, 1] ] As wee see here, there is constant 1 in second column. So Formula is RSS = np.dot(np.transpose(residual), residual) Reg = y - np.mean(y) TSS = ... 2 The assumption of error's normality is only needed for statistical inference if your sample is small (say n<100 to say something). For large samples, one relies on the CLT. In other words. You do nothing. unless your dependent variable is of limited distribution (discrete for example), OLS is fine. 0 You speak about linear regression I suppose. Linear regression can be justified under different set of assumptions, more or less general. ... shouldn't a central limit theorem ensure the normality regardless of how the error terms are distributed as long as the sample is large enough? Asymptotically yes, even if some moment conditions are needed and ... 0 Since you want to incorporate an eye-level variable (inflamation in a particular eye), then either you can average the inflamation variable for both eyes and proceed with a similar model, or you can use a mixed effects / multilevel model with a level-2 outcome (and person-level covariates), and level 1 covariates (eye inflamation). 1 It's just a couple of typos: Yes, it should start from t=1 Apparently, the derivative is with respect to \phi_i (I guessed it from the multiplicand x_{ti}) 2 Define "obvious"! What's obvious to you doesn't need be obvious to someone else. So I can only offer my perspective. First, I hope it's obvious that the slope \beta of the line doesn't change if we shift the data around; only the intercept changes. So, to simplify the formulas, we can, without loss of generality, assume that our data are centred ... 0 Let x = \epsilon = 0. Now check the relevant moments. 0 These are not the usual assumptions for linear regression. \bar{X} is not usually assumed to have a normal distribution. The X_i are assumed fixed. Var[Y_i] is assumed to be \sigma^2. I don't know what u_i means. Also, I don't know what it means to use subscripts i in the formula for the variances without a summation symbol.$$E[\hat{\beta_0}-\...

0

If $Y|X$ is normally distributed, OLS is the maximum likelihood estimator (MLE). If $Y|X$ is Laplace distributed, LAD is the MLE. MLE has some desirable features (as you put it yourself, better estimations); see e.g. Chan "Lecture 8: Properties of Maximum Likelihood Estimation (MLE)" (2015) for details. Therefore, you may well prefer MLE to other ...

0

Two ideas Idea 1: Geometry of Linear Regression Consider the angle between $\theta_1=<y, x1>$, $\theta_2=<y, x2>$ and the angle between $\theta_3=<y, z>$ where z is the projection of y into affine space span by x1,x2 the projection z should be "at most" y itself and "at least" parallels to x2. Since the angle between ...

0

One of the most underrated feature selection tools is a robust suite of exploratory plots. If you haven't already looked at scatter/bar plots for each of your features vs your response, you can look at the plots to get a sense of how your variables interact. For a more algorithmic examination of feature selection you can consider the following principles by ...

0

Yes. See Wikipedia article “proofs involving ordinary least squares”. They are independent and for any independent random variables X and Y, their covariance is EXY-EX EY=0.

Top 50 recent answers are included