66

My detailed answer is below, but the general (i.e. real) answer to this kind of question is: 1) experiment, mess around, look at the data, you can't break the computer no matter what you do, so ... experiment; or 2) read the documentation. Here is some R code which replicates the problem identified in this question, more or less: # This program written in ...


50

Yes, the covariance matrix of all the variables--explanatory and response--contains the information needed to find all the coefficients, provided an intercept (constant) term is included in the model. (Although the covariances provide no information about the constant term, it can be found from the means of the data.) Analysis Let the data for the ...


43

At the start of your derivation you multiply out the brackets $\sum_i (x_i - \bar{x})(y_i - \bar{y})$, in the process expanding both $y_i$ and $\bar{y}$. The former depends on the sum variable $i$, whereas the latter doesn't. If you leave $\bar{y}$ as is, the derivation is a lot simpler, because \begin{align} \sum_i (x_i - \bar{x})\bar{y} &= \bar{y}\...


36

Multiple linear regression coefficient and partial correlation are directly linked and have the same significance (p-value). Partial r is just another way of standardizing the coefficient, along with beta coefficient (standardized regression coefficient)$^1$. So, if the dependent variable is $y$ and the independents are $x_1$ and $x_2$ then $$\text{Beta:} \...


35

There are several "flavours" or forms of the bootstrap (e.g. non-parametric, parametric, residual resampling and many more). The bootstrap in the example is called a non-parametric bootstrap, or case resampling (see here, here, here and here for applications in regression). The basic idea is that you treat your sample as population and repeatedly draw new ...


35

You are right. Technically, it is any value. However, when I teach this I usually tell people that you are getting the effect of a one unit change in $X_j$ when all other variables are held at their respective means. I believe this is a common way to explain it that is not specific to me. I usually go on to mention that if you don't have any ...


31

With a complementary-log-log link function, it's not logistic regression -- the term "logistic" implies a logit link. It's still a binomial regression of course. the estimate of time is 0.015. Is it correct to say the odds of mortality per unit time is multiplied by exp(0.015) = 1.015113 (~1.5% increase per unit time) No, because it doesn't model in ...


26

The OLS estimator in the linear regression model is quite rare in having the property that it can be represented in closed form, that is without needing to be expressed as the optimizer of a function. It is, however, an optimizer of a function -- the residual sum of squares function -- and can be computed as such. The MLE in the logistic regression model ...


24

You are right about the interpretation of the betas when there is a single categorical variable with $k$ levels. If there were multiple categorical variables (and there were no interaction term), the intercept ($\hat\beta_0$) is the mean of the group that constitutes the reference level for both (all) categorical variables. Using your example scenario, ...


24

Without seeing your data, this is difficult to answer definitively. One possibility is that your datasets span different ranges of the independent variable. It is well-known that combining data across different groups can sometimes reverse correlations seen in each group individually. This effect is known as Simpson's Paradox.


23

The standard errors of the model coefficients are the square roots of the diagonal entries of the covariance matrix. Consider the following: Design matrix: $\textbf{X = }\begin{bmatrix} 1 & x_{1,1} & \ldots & x_{1,p} \\ 1 & x_{2,1} & \ldots & x_{2,p} \\ \vdots & \vdots & \ddots & \vdots \\ 1 & x_{n,1} & \ldots &...


21

The correlation coefficient is $r$. $R^2$ is the square of $r$, and it is of course always positive, regardless of the sign of $r$. Taking the square root gives that $r= \pm 0.8489$, and since the relationship is negative, you can conclude that $r = -0.8489$.


18

The answer is "not necessarily" — how correlated the variables are dictates how "noisy" the scatter plot is, but not how steep. In fact, the correlation and regression slope are telling you two quite different things. The slope estimates how many units $y$ increases, on average, per one unit increase in $x$. The correlation measures the ...


18

The geometric interpretation of ordinary least squares regression provides the requisite insight. Most of what we need to know can be seen in the case of two regressors $x_1$ and $x_2$ with response $y$. The standardized coefficients, or "betas," arise when all three vectors are standardized to a common length (which we may take to be unity). ...


18

For simple linear regression there is a relationship between slope and correlation: $\hat\beta_1 = r_{x,y}{s_y\over s_x}$ So the relationship of $\hat\beta_1$ and $r_{xy}$ is entirely dependent on the standard deviations of x and y, and, by rescaling variables, can be pretty much any value.


17

If you say your model is ln(y) = b*ln(x) + a it is only part of your model. Your actual model includes an error term: $\ln y_i = b\cdot \ln x_i + a + \varepsilon_i$ and you assume that the error distribution is $\varepsilon_i \sim \mathcal{N}(0,\,\sigma^2)$. Now let's back-transform it: $y_i = \exp(a) \cdot x_i^b \cdot \exp(\varepsilon_i)$ As you see, ...


16

The question, as phrased, is slightly ambiguous. It states that "the coefficients in each model appear to be exactly the same". There are two ways that statement could be interpreted, with respect to: (1) the Estimates of the coefficients, or (2) the tests of the coefficients. Regarding the Estimates of the coefficients, they are being adjusted for the ...


16

"Partial regression coefficients" are the slope coefficients ($\beta_j$s) in a multiple regression model. By "regression coefficients" (i.e., without the "partial") the author means the slope coefficient in a simple (only one variable) regression model. If you have multiple predictor / explanatory variables, and you run both a set of simple regressions, ...


16

Yes, coefficients of dummy variables can be more than one or less than zero. Remember that you can interpret that coefficient as the mean change in your response (dependent) variable when the dummy changes from 0 to 1, holding all other variables constant (i.e. ceteris paribus). The mean height of people in the United States is around 176 cm for males and ...


16

If your data looks something like this then the reason may be more obvious. Your two original regression lines would be almost parallel and look reasonably plausible but combined they produce a different result which is probably not very helpful. The data for this chart came from using the R code exdf <- data.frame( x=c(-64:-59, -52:-47), y=c(-...


16

A simple thought experiment: suppose your predictor was a length, originally expressed in millimetres. If you express it instead in kilometres and fit the model again, you have not really changed anything meaningful about the relationship, but your coefficient will drop by several orders of magnitude. You can also get significant results with very low ...


15

I will mainly focus on your first three questions. The short answers are: (1) you need to compare the effect of the IV on the DV for each time period but (2) only comparing the magnitudes can lead to wrong conclusions, and (3) there are many ways of doing that but no consensus on which one is correct. Below I describe why you cannot simply compare ...


15

The coefficients from the output do have a meaning, although it isn't very intuitive to most people and certainly not to me. That is why people change them to odds ratios. However, the log of the odds ratio is the coefficient; equivalently, the exponentiated coefficients are the odds ratios. The coefficients are most useful for plugging into formulas that ...


15

Elsewhere on this site, explicit solutions to the ordinary least squares regression $$\mathbb{E}(z_i) = A x_i + B y_i + C$$ are available in matrix form as $$(C,A,B)^\prime = (X^\prime X)^{-1} X^\prime z\tag{1}$$ where $X$ is the "model matrix" $$X = \pmatrix{1 & x_1 & y_1 \\ 1 & x_2 & y_2 \\ \vdots & \vdots & \vdots \\ 1 & ...


15

Are the LASSO coefficients interpreted in the same method as logistic regression? Let me rephrase: Are the LASSO coefficients interpreted in the same way as, for example, OLS maximum likelihood coefficients in a logistic regression? LASSO (a penalized estimation method) aims at estimating the same quantities (model coefficients) as, say, OLS maximum ...


15

#Create some example data mydata1 <- subset(iris, Species == "setosa", select = c(Sepal.Length, Sepal.Width)) mydata2 <- subset(iris, Species == "virginica", select = c(Sepal.Length, Sepal.Width)) #add a grouping variable mydata1$g <- "a" mydata2$g <- "b" #combine the datasets mydata <- rbind(mydata1, mydata2) #model without grouping ...


15

If you're trying to generate data from logistic regression's assumed data generating mechanism, your code does not do that. Logistic regression's data generating mechanism looks like $$ \eta = X\beta$$ $$ p = \dfrac{1}{1+e^{-\eta}}$$ $$ y \sim \operatorname{Binomial}(p, n) $$ What it looks like you're trying to do is create a linear regression in the log ...


13

Does your software give you a parameter covariance (or variance-covariance) matrix? If so, the standard errors are the square root of the diagonal of that matrix. You probably want to consult a textbook (or google for university lecture notes) for how to get the $V_\beta$ matrix for linear and generalized linear models.


Only top voted, non community-wiki answers of a minimum length are eligible