New answers tagged

2

Adding more variables will always increase the R squared. Additionally, there is no reason to remove variables which fail to reject the null as I explain here.


1

You might want to look at Frank Harrell's blog post about how to evaluate the value added by a new biomarker. What's most important at first is a test of whether the model with all the predictors including your protein of interest is better than a model with all the other predictors that you are including as controls but omitting your protein. With least-...


0

From An Introduction to Statistical Learning, second edition (James et al, 2021): Given these individual p-values for each variable, why do we need to look at the overall F-statistic? After all, it seems likely that if any one of the p-values for the individual variables is very small, then at least one of the predictors is related to the response. However, ...


0

As an interaction term is the product of the individual predictors, your PC1:PC2 interaction includes all 2-way interactions among the original predictors, including quadratic terms in each of them. First, you should check that you aren't overfitting the data this way. Say that you used cross-validation (CV) to find that 2 principal components provided the ...


0

Let $time_{i}$ as $y_{i}$ and the $\{\mathbf{x}_i\}_{i=1}^{N}$ as your predictors (stops) where $\mathbf{x}_i$ is equal to the row $i$ of your dataset taking the last columns out and $i$ is the row index. Take a deterministic regression problem where you want to fit: $y=f(x;\mathbf{\Theta})$, $\mathbf{\Theta}$ as an arbitrary dimensional parameter. Take $\...


3

You might be able to do something simple with your Likert-scale outcome values that doesn't require a generalized linear model. A prior square-root or logarithmic transformation of those values might bring the residuals into a form that's good enough for a mixed-model linear regression. Those sometimes help when there is substantial skew toward large values ...


1

There are some helpful mathsy explanations, but I thought perhaps this could use an intuitive example. Suppose that you're investigating (perhaps for an insurance company) whether hair colour has an impact on crash risk. You look at the data, and at first pass you see that brunettes are 10% more likely to crash than blondes. But in the same data you see that ...


0

The question is quite old, but if you are still interested in the problem, here is a suggestion that elaborates on the comments by Dave and Frank Harrel. Ordinary linear regression is for continuous unlimited response variables. For responses that are non-negative counts, a special case of "generalized linear models" (GLM) has been specifically ...


0

Numerical funkiness can cause this when you do the math on a computer, but that is a limitation of the computer. You are correct about the math... ...on the development data. When you go do test new points, you might find that including a predictor causes the out-of-sample performance to decrease.


0

In general, if a variable is strongly related to the response, controlling for it reduces the error variance, and thus increases power and precision. That doesn't have anything to do with DAGs—it's basic multiple regression. Controlling for U1 is better than U0 because it will be more strongly correlated with Y, leading again to lower error variance, etc. ...


1

The term you are looking for is overfitting. Wikipedia has a good explanation.


11

I would point out three things: (1) Generally (related to the estimation of causal effects) Usually you want to explain phenomena out there in the world with parsimonious models including variables deduced from some theory. You may just add any variable that comes to your mind to a regression model and end up with an almost perfect fit, but you did not learn ...


22

There is no such thing as a "sweet spot" for the number of variables to control for in order to get an unbiased estimate of the causal effect. Since we are talking about confounding, we must have in mind the estimation of the causal effect of a particular variable. You use a graphic tool called the DAG to map out the causal relationships and then ...


5

Well, it's related to the concept of p-hacking. Given a sufficient number of plausible confounding variables to be added in the study, it's possible to find a combination of them that would yield significant results (you just plug in or out until you get significant results, and you report those). There's a very nice post in FiveThirtyEight about this so you ...


4

As far as I understand, the assumption of Heteroscedasticity and Linearity are violated. As far as I understand, the first causes lower p values and imprecise coefficients, the second causes wrong coefficients? If the "real" relationship isn't linear, then the linear coefficients aren't merely the "wrong coefficients", there aren't any ...


6

A few things to start with. I agree that nature is rarely linear, but I disagree that the approach you've taken is a reasonable attempt to account for this. Polynomial terms do add non-linearity, but they are high bias models (because we can only estimate functions in the set of polynomials). A better approach to adjusting for non-linearity would be use a ...


6

One troubling situation is when the regressor variables are not independent from each other. This can make it look like there is seemingly (causal) relationship that is not truly present. Example: Say we have a model where $y$ is a function of $x$ described by an exponential term $E[y|x] = e^{0.6x}$ and some Gaussian noise with deviation $\sigma = 0.1$ $$y \...


3

On the question: "Linearity assumption violated - can I still draw conclusions from my model?", my answer is a limited yes! You may chose to verify my claim by constructing a simple non-linear model and assuming normal error generating model, project out. Further, fit a linear version of the model. I claim the Least-Square theoretical derived ...


2

Isn't it divided by $\sqrt{(X^TX)^{-1}}$? $\sqrt{(X^TX)^{-1}}$ is not a single number. This is because $X$ is a $n$ by $p$ matrix and $X^T X$ will be a $p$ by $p$ matrix of which you need the $j$-th diagonal element. But I am confused that why $$ \frac{N(0, (X^TX)^{-1})}{\sqrt{(X^TX)^{-1}_{jj}}}\sim N(0,I)?$$ This is not the case. Yes, you do have that ...


2

The first thing I would (always) do is simply plot the data. That‘s probably the fastest and most intuitive way to get a solid understanding of your data and will help you to identify possible relationships between variables. This will also help you to come up with a suitable statistical model if needed. As far as I can tell, Mosaic plots seem to be a good ...


6

You're on the right track, and you can answer your question using some old-fashioned tools. In your current model, the intercept reflects the predicted average productivity when Light, Water and Phenology are all at zero. You can make this a little more intuitive by centring these variables to have a mean of zero. Then, the intercept is just the average ...


4

You are not using OLS, so why do you think $R^2$ should increase when you add more variables? the premise of your question is flawed. You should show what command you used to generate the output. I can see that you probably used robustfit function in MATLAB. This function is not OLS. It uses an iterative algorithm to weight the observations, looking for what ...


1

ORIGINAL ANSWER: All bets are off without knowing what MATLAB is doing here under the heading (robust fit). The results show quite different models: one forced through the origin (intercept zero) and the other omitting a predictor that the first model declares strongly significant. It's implausible -- despite the otherwise appealing figures of merit -- that ...


0

It's hard to answer these questions without at least some domain knowledge about your data. Without getting into the details of your dataset, it is a well established problem in linear regression that additional predictor variables may negatively affect the performance of your model. I suggest that you check the relationship of your predictor variables (Are ...


2

You can build a logistic regression and compute p-values for each covariate using the t-test, i.e., using the parameter estimate divided by its standard error. The significance of this statistic based on the t distribution is given by the p-value, so the effects with the smallest p-values are the most significant.


0

This could be explained by a couple of things. First, note that if all predictors had a coefficient of 0, then the model would return the mean of outcome as its prediction. So, if the coefficients are very or there is a weak relationship between the predictor and the outcome (which is different than being statistically significant), then the model might ...


0

Your two models are functionally equivalent: > attr(terms(~ b * c * (d+e)),"factors") b c d e b:c b:d b:e c:d c:e b:c:d b:c:e b 1 0 0 0 1 1 1 0 0 1 1 c 0 1 0 0 1 0 0 1 1 1 1 d 0 0 1 0 0 1 0 1 0 1 0 e 0 0 0 1 0 0 1 0 1 0 1 > attr(terms(~ b + c + d + e + b * c + b * c * ...


3

Having a Hessian that must be updated every time the guess at parameter estimates $\beta$ changes is a necessary requirement for achieving excellent calibration (i.e., the weighted in iteratively reweighted least squares). Multinomial logistic regression is built for prediction, and I suspect that your stated goal of classification would be better replaced ...


3

The Ridge regressor has a classifier variant: RidgeClassifier. This classifier first converts binary targets to {-1, 1} and then treats the problem as a regression task, optimizing the same objective as above. https://scikit-learn.org/stable/modules/linear_model.html#classification So it's a linear model that has closed-form solution. Logistic regression is ...


3

This could be for a number of reasons. I suspect it is because the logistic regression classifier is trained using Iteratively Re-weighted Least Squares (IRWLS), which requires the repeated solution of a system of linear equations that depends on the current output of the model. For details, see Minka (2003), sections 1 and 2. The ridge classifier, on the ...


1

One visual interpretation of adding more variables to your model, which implies adjusting for them and thus slicing your data, is that you may end up with only the points that are closer to the curve you're fitting. This explains why your $R^2$ is increasing when you add more variables. Indeed, your model fits better the data [that is left], however, this ...


0

Short answer: the reason to "keep" X2 in the model is because the goal of regression analysis is (usually) not to come up with the "best" model of Y in some abstract sense, but to answer specific questions about how different independent variables are related to Y, and the model you ran (with a significant X1 coefficient and a non-...


1

Thanks to @whuber that made me notice that this problem must be equivalent to $n$ separate ordinary least square regressions, where the coefficients of the $k$-th regression are given by the $k$-th row of $A(k,:)$. If this is the case, I should be able to compare the standard OLS estimator with $A = \Sigma({\bf Y},{\bf X}) \Sigma({\bf X})^{-1}$ and obtain ...


2

Let's look at a simple example where you have 1 dependent variable and 1 independent variable. Lets say your independent variable was "male age", and your dependent variable is "height". So your model wants to model how tall a male is given their age. Essentially, what you are trying to do with a linear regression model is plot the mean ...


Top 50 recent answers are included