Here is my multiple linear regression code where $Xtrain$ and $Xtest$ are matrices containing respectively the train and test samples in rows and explanatory variables in columns, and $ytrain$ and $ytest$ are the output column vectors containing the response variable values respectively for the train and test samples.
ginv() is the generalized inverse function since normal inverse (function
solve()) would not work here since the number of variables (1000) is larger than the number of samples (30).
$pred$ is the prediction of the response for the test samples using the fitted model.
library(MASS) w = ginv(t(Xtrain) %*% Xtrain) %*% t(Xtrain) %*% ytrain pred = Xtest %*% w
Interestingly, the prediction has 100% accuracy when I compare with the true $ytest$. The responses are correct in all decimal points!
How come this can happen? I know this is impossible with high-dimensional data, and also, when I use
glmnet() (with no regularization) instead of my own code, it gives 22% accuracy. So, do you have any idea about what is wrong with above code? Is there something wrong with
ginv() (this is the first time I use it)? Does
glmnet() function use generalized inverse as well when the sample size is smaller than the variable size?