Methods:
From the machine learning literature, I understand different parameters can show performance of model in machine learning. I would briefly expand my understanding with confusion matrix:
(1) Accuracy: measures the proportion of correct predictions considering the positive and negative inputs.
(2) Specificity: measures the proportion of the true negatives, that is, the ability of the system on predicting the correct values for the cases that are the opposite to the desired one.
ROC curve: Considering many thresholds, it is possible to calculate a set of pairs (sensitivity, 1-specificity) which can be plotted in a curve. This curve will be the ROC curve for the system where the y-axis (ordenades) represents the sensitivity and the x-axis (abscissas) represents the complement of specificity (1 - specificity).
area under the ROC Curve (AUC) - higher the ROC curve's area, better the system.
(3) Sensitivity: measures the proportion of the true positives, that is, the ability of the system on predicting the correct values in the cases presented.
(4) Efficiency: mean of Sensibility and Specificity
(5) Positive Predictive Value: the estimation of how good the system is when making a positive affirmation - POSITIVE HITS / TOTAL POSITIVE PREDICTIONS
(6) Negative Predictive Value: measure indicates the estimation of how good the system is when making a negative affirmation - NEGATIVE HITS / TOTAL NEGATIVE PREDICTIONS
(7) Phi (φ) Coefficient
My data
I have quantitative y variable with many x quantitative variables and fit a lear regression equation. Objective here is how closely x
values predict y
- means that deviation from expectation in both negative and positive direction is not good.
#random population of 200 subjects with 1000 variables
M <- matrix(rep(0,200*100),200,1000)
for (i in 1:200) {
set.seed(i)
M[i,] <- ifelse(runif(1000)<0.5,-1,1)
}
rownames(M) <- 1:200
#random yvars
set.seed(1234)
u <- rnorm(1000)
g <- as.vector(crossprod(t(M),u))
h2 <- 0.5
set.seed(234)
y <- g + rnorm(200,mean=0,sd=sqrt((1-h2)/h2*var(g)))
myd <- data.frame(y=y, M)
Let's say I want to fit ridge
and lasso
to this:
require(glmnet)
# LASSO
fit1=glmnet(M,y, family="gaussian", alpha=1)
# Ridge
fit1=glmnet(M,y, family="gaussian", alpha=0)
Question:
What performance measure listed above (or not listed) are relevant here and how can I calculate them?