Questions tagged [linear-model]
Refers to any model where a random variable is related to one or more random variables by a function that is linear in a finite number of parameters.
368
questions
1
vote
1
answer
127
views
PCA with manually given PC1
I am using R function prcomp to do PCA on my data set. I wonder if i want to force the pc1 direction as given and perform the PCA analysis on the rest, how can i do it.
Thanks.
- 11
1
vote
1
answer
2k
views
Why is Standard Error of Slope small when the data is spread out?
In the book " Introduction to Statistical Learning " , the standard error of the slope term of Linear Regression is given as follows :
The book also says the Slope is more precise when the ...
- 337
1
vote
1
answer
88
views
Example 2.2.2 fromAn Introduction to Generalized Linear Models. Show $Y_{j k}, \bar{Y}_{j}, \hat{\beta}_{j} $ are all independent
This question is related to what I have asked in my previous post:
How to derive the covariance matrix between $\bar{y}$ and $\hat{\beta_c}$ where $\hat{\beta_c}$ is the OLS estimator of a linear ...
- 689
1
vote
1
answer
981
views
What kind of linear mixed model is most appropriate for this data?
I previously wrote about this data here, and was advised that a linear mixed model applied to the raw data would provide "more precision and power" than the originally suggested approach of a ...
1
vote
1
answer
2k
views
Is Ridge more robust than Lasso on feature selection?
My goal is to identify the best n-feature linear model, i.e. pick the model with only n-feature from total N features (n < N) and lowest Mean-Squared-Error (MSE). The experiment is on the Lasso and ...
- 73
0
votes
1
answer
803
views
comparing coefficients from multivariate regression
I have a multivariate linear regression model where the predictors are concentrations of different drugs, of the same units, and the responses are the survival percentages of each different kind of ...
- 65
0
votes
2
answers
2k
views
Meaning of $\hat{\beta}$ of the linear regression model [closed]
In the simple linear regression model, $\hat{\beta}$ is the sum of independent normally distributed random variables.
Is it false because in linear regression there is $\beta$ and not $\hat{\beta}$?
0
votes
0
answers
113
views
Confusion over predict() command in R
I'm trying to use the command predict() to be able to obtain my $x_i\hat{\beta}$ in R
predict(logitmodel,type='response')
As long as I'm not specifying any ...
0
votes
1
answer
1k
views
Is Gradient Boosting Regression Tree able to learn linear models
Assume $Y$ is a linear function of a vector of variables $X$ (plus a noise term). The train data consists of ($X,Y$) such that $X \in ［0,1］$. Assume one use gbdt to learn this linear model. And if ...
- 223
0
votes
1
answer
63
views
How to reconcile these two matrix equations for obtaining the coefficients for a linear least squares fit?
In ordinary least squares linear regression, given a set of data points $(x_1,y_1),(x_2,y_2),...(x_N,y_N)$, that we want to fit to the function $y=\beta_0 + \beta_1 x$, we would usually write the ...
- 169
36
votes
5
answers
3k
views
What if my linear regression data contains several co-mingled linear relationships?
Let's say I am studying how daffodils respond to various soil conditions. I have collected data on the pH of the soil versus the mature height of the daffodil. I'm expecting a linear relationship, ...
- 623
30
votes
3
answers
13k
views
Bayesian lasso vs ordinary lasso
Different implementation software are available for lasso. I know a lot discussed about bayesian approach vs frequentist approach in different forums. My question is very specific to lasso - What are ...
- 3,653
27
votes
1
answer
3k
views
Common statistical tests as linear models
(UPDATE: I dived deeper into this and and posted the results here)
The list of named statistical tests is huge. Many of the common tests rely on inference from simple linear models, e.g. a one-sample ...
20
votes
2
answers
20k
views
If I repeat every sample observation in a linear regression model and rerun the regression how would the result be affected? [duplicate]
Say I have N observations, possibly multiple factors and I repeat each observation twice (or M times) how would a regression on this new set of size NM compare to a regression on just the original ...
- 1,003
17
votes
1
answer
37k
views
Why lasso for feature selection?
Suppose I have a high-dimensional dataset and want to perform feature selection. One way is to train a model capable of identifying the most important features in this dataset and use this to throw ...
- 272
15
votes
2
answers
5k
views
The theory behind the weights argument in R when using lm()
After a year in grad school, my understanding of "weighted least squares" is the following: let $\mathbf{y} \in \mathbb{R}^n$, $\mathbf{X}$ be some $n \times p$ design matrix, $\boldsymbol\beta \in \...
- 5,007
15
votes
3
answers
24k
views
Does cross-validation on simple or multiple linear regression make sense?
Does it make sense to apply train-test split or k-fold cross-validation to a simple linear regression model or multiple linear regression model?
I'm really confused about this because I saw this ...
- 335
15
votes
1
answer
785
views
Restricted maximum likelihood with less than full column rank of $X$
This question deals with restricted maximum likelihood (REML) estimation in a particular version of the linear model, namely:
$$
Y = X(\alpha)\beta + \epsilon, \\
\epsilon\sim N_n(0, \Sigma(\alpha)),...
- 4,541
14
votes
2
answers
7k
views
Difference in reported p-values between lm and aov in R
What explains the differences in p-values in the following aov and lm calls ?
Is the difference only due to different types of ...
- 5,112
13
votes
2
answers
6k
views
Selecting PCA components which separate groups
I frequently used to diagnose my multivariate data using PCA (omics data with hundreds of thousands of variables and dozens or hundreds of samples). The data often come from experiments with several ...
- 7,559
13
votes
2
answers
20k
views
Some of my predictors are on very different scales - do I need to transform them before fitting a linear regression model?
I would like to run linear regression over a multi-dimensional data set. There exist differences among different dimensions in terms of their magnitude of order. For instance, dimension 1 generally ...
- 2,817
13
votes
1
answer
22k
views
should I use na.omit or na.exclude in a linear model (in R)?
I try to understand the difference between using different na.actions (na.omit and na.exclude) to handle missing data in a linear model using R. I used the lm function in R (https://stat.ethz.ch/R-...
- 339
13
votes
2
answers
14k
views
Why take transpose of regressor variable in linear regression
I am stuck trying to understand the basic calculation of ordinary least squares. From Wikipedia:
$$y = \beta X^T + \varepsilon$$
where $X$ is the independent variable, $Y$ is the dependent variable ...
- 6,565
12
votes
1
answer
726
views
Variance term in bias-variance decomposition of linear regression
In 'The Elements of Statistical Learning', the expression for bias-variance decomposition of linear-model is given as
$$Err(x_0)=\sigma_\epsilon^2+E[f(x_0)-E\hat f(x_0)]^2+||h(x_0)||^2\sigma_\epsilon^...
- 1,636
11
votes
2
answers
2k
views
Possible extensions to the default diagnostic plots for lm (in R and in general)?
I started digging a bit into the plot.lm function, this function gives six plots for lm, they are:
a plot of residuals against fitted values
a Scale-Location plot of sqrt(| residuals |) against ...
- 21.6k
11
votes
5
answers
5k
views
Is using deciles to find correlation a statistically valid approach?
I have a sample of 1,449 data points that are not correlated (r-squared 0.006).
When analyzing the data, I discovered that by splitting the independent variable values into positive and negative ...
- 2,913
11
votes
3
answers
3k
views
Is OLS the frequentist approach to linear regression?
In this Wikipedia article, there is this sentence:
This is a frequentist approach
Is 'this' referring to OLS?
Is it really 'a' rather than 'the'? What are some other frequentist approaches? As ...
- 2,434
11
votes
7
answers
3k
views
Why do we use Linear Models when tree based models often work better than linear models?
In Supervised Machine Learning, and specifically on Kaggle, it is usually seen that tree models often outperform linear models. And even in the tree-based models, it is usually XGBoost that ...
- 325
10
votes
2
answers
5k
views
Omitted variable bias vs. Multicollinearity
There's seems to be a bit like catch 22: suppose I am doing linear regression, and I have 2 variables that are highly correlated. If I use both in my model, I will suffer from multicollinearity, but ...
- 3,403
10
votes
2
answers
840
views
Difference between random effetcs and dummy coding of a categorical variable
I'm a bit confused with the definition of random effects and why it couldn't be rephrased in terms of dummy coding of a categorical variable.
Assume the model is linear with one dependent variable $Y$...
- 8,507
10
votes
1
answer
6k
views
What's the underlying algorithm used by R's lm?
I've been asked a question regarding a linear model made with R's lm:
"Did the regression use linear or non-linear iterative least squares?"
I searched a bit and [...
- 313
9
votes
1
answer
3k
views
Using percentiles as predictors - good idea?
I am thinking about a problem which is to predict log(spend) of a customer using linear regression.
I am considering what features to use as input and wondering if it would be OK to use the ...
- 223
9
votes
4
answers
4k
views
Systematic/measurement error on a linear regression
Suppose I have a set of data ${(x_i,y_i)}$ in which the uncertainty in the measurements ${(\Delta x_i,\Delta y_i)}$ (which come from the propagation of systematic errors from the measurement apparatus)...
- 191
9
votes
1
answer
1k
views
When do improper linear models get robustly beautiful?
Are improper linear models used in practice or are they some kind of curiosity described from time to time in scientific journals? If so, in what areas are they used? When would they be useful?
They ...
Tim♦
- 138k
8
votes
1
answer
229
views
Obtaining an estimator for z given an estimator for log z
As per gung's advice in Getting the equation from R's lm when using a product, I am starting a new thread for this question.
I have a model $\widehat{\log z} = a + bx + cy + dxy$ for random ...
- 3,112
8
votes
1
answer
649
views
Are GAM models linear in the parameters?
Consider a GAM model, expressed in mgcv just to fix ideas:
my_model <- gam(y ~ ti(x1)+ti(x2) + ti(x1, x2), method= "REML")
...
- 18k
8
votes
2
answers
6k
views
Estimating linear regression with OLS vs. ML
Assume that I'm going to estimate a linear regression where I assume $u\sim N(0,\sigma^2)$. What is the benefit of OLS against ML estimation? I know that we need to know a distribution of $u$ when we ...
- 5,955
8
votes
1
answer
8k
views
Measure for Separability
I have a (binary) classification problem where after merging single training data points (that can be tracked back to the same source) into aggregates, test accuracy (on single data points again) ...
- 305
8
votes
2
answers
337
views
Intuition for $RSS_2 - RSS_1$ having chi-square distribution in F-test for linear models
In https://en.wikipedia.org/wiki/F-test#Regression_problems, an application of the F-statistic to comparing linear models is given:
Consider two models, 1 and 2, where model 1 is 'nested' within ...
- 223
8
votes
2
answers
8k
views
GLM on unbalanced design
I have a dataset that comprises 200 males and 250 females and I am testing their responses on the relationship between X and Y.
X and Y are continuous and X1 (gender) is categorical.
I am using ...
- 3,255
8
votes
3
answers
2k
views
Residuals in a linear model are independent but sum to zero; isn't it a contradiction? [duplicate]
The sum of the residuals in a linear model equals zero.
The residuals in a linear model are independent.
Isn't it a contradiction?
- 2,235
8
votes
2
answers
700
views
Hacking linear regression
Let's say I perform linear regression on some data that produces the following $R^2$:
$\text{RSS} = 1966815.13$
$\text{TSS} = 2145213.91$
$R^2 = 0.083$
Now let's say I bucket (take the average of ...
- 123
8
votes
1
answer
436
views
Bibliography for linear models
My main question: what bibliography would you recommend for linear models theory?
I'm thinking of acquiring Plane answers to complex questions: the theory of linear models, by Ronald Christensen. Has ...
- 5,692
8
votes
2
answers
3k
views
p-values change after mean centering with interaction terms. How to test for significance?
I assumed the following interaction model:
$$y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_3 x_3 + \beta_4 x_2 x_3$$
And then applied mean centering:
$$y = \beta_0 + \beta_1(x_1 - \bar{x_1}) + \...
- 1,041
7
votes
1
answer
36k
views
ncvTest from R and interpretation
I have done a ncvTest, but I am not really sure how to interpret this properly. I look documentation and examples online but was not able to find anything that clearly explains about ncvTest.
Here ...
- 359
7
votes
2
answers
8k
views
Linear regression with upper and/or lower limits in R?
Is there a way to run a linear regression with upper and/or lower limits
on the coefficients in R?
- 895
7
votes
1
answer
518
views
Estimating a sparse inverse covariance matrix with known sparsity
The inverse of the covariance matrix for a distribution can be a good value for the mass matrix of a Hamiltonian monte carlo distribution.
If the distribution in question is the posterior of a ...
- 4,482
7
votes
1
answer
3k
views
Generalized Least Squares: Estimation of Variance-Covariance matrix
Linear model in matrix form is
$
\mathbf{y}=\mathbf{X}\beta+\epsilon\textrm{ where }\epsilon\sim\mathbb{N}\left(0,\sigma^{2}\mathbf{V}\right).
$
Then $\beta$ can be estimated through generalized ...
- 2,729
7
votes
2
answers
13k
views
Testing a regression coefficient against 1 rather than 0
Brief caveat- I haven't dusted off my stats knowledge since some university courses a few years ago, and I'm struggling with cobwebs.
I have a model where a linear 1 to 1 relationship has been ...
- 173
7
votes
2
answers
26k
views
How to know if "best fit line" really represents known set of data?
I have a known set of data. I have created a "linear best fit line" for that set of data. Is there a way to determine how well my set of data fit that best fit line (some sort of score)?
I'm very ...
- 175