For a model like this what performance measures can I calculate and how?

Question

Methods:

From the machine learning literature, I understand different parameters can show performance of model in machine learning. I would briefly expand my understanding with confusion matrix:

(1) Accuracy: measures the proportion of correct predictions considering the positive and negative inputs.

(2) Specificity: measures the proportion of the true negatives, that is, the ability of the system on predicting the correct values for the cases that are the opposite to the desired one.

ROC curve: Considering many thresholds, it is possible to calculate a set of pairs (sensitivity, 1-specificity) which can be plotted in a curve. This curve will be the ROC curve for the system where the y-axis (ordenades) represents the sensitivity and the x-axis (abscissas) represents the complement of specificity (1 - specificity).

area under the ROC Curve (AUC) - higher the ROC curve's area, better the system.

(3) Sensitivity: measures the proportion of the true positives, that is, the ability of the system on predicting the correct values in the cases presented.

(4) Efficiency: mean of Sensibility and Specificity

(5) Positive Predictive Value: the estimation of how good the system is when making a positive affirmation - POSITIVE HITS / TOTAL POSITIVE PREDICTIONS

(6) Negative Predictive Value: measure indicates the estimation of how good the system is when making a negative affirmation - NEGATIVE HITS / TOTAL NEGATIVE PREDICTIONS

(7) Phi (φ) Coefficient

My data

I have quantitative y variable with many x quantitative variables and fit a lear regression equation. Objective here is how closely x values predict y - means that deviation from expectation in both negative and positive direction is not good.

#random population of 200 subjects with 1000 variables 

M <- matrix(rep(0,200*100),200,1000)
for (i in 1:200) {
set.seed(i)
  M[i,] <- ifelse(runif(1000)<0.5,-1,1)
}
rownames(M) <- 1:200

#random yvars 
set.seed(1234)
u <- rnorm(1000)
g <- as.vector(crossprod(t(M),u))
h2 <- 0.5 
set.seed(234)
y <- g + rnorm(200,mean=0,sd=sqrt((1-h2)/h2*var(g)))

myd <- data.frame(y=y, M)

Let's say I want to fit ridge and lasso to this:

 require(glmnet)

   # LASSO 
   fit1=glmnet(M,y, family="gaussian", alpha=1)

   # Ridge   
   fit1=glmnet(M,y, family="gaussian", alpha=0)

Question:

What performance measure listed above (or not listed) are relevant here and how can I calculate them?

Answer 1

Being that you're facing a regression task, the metrics above are not appropriate: They're for assessing classifier performance. The scikit learn metrics documentation provides a handy, quick reference that nests metrics under their appropriate problem types.

In regression, mean squared error (MSE) is the classic measure. You may calculate this by taking the difference of each prediction and actual value, squaring them, and taking the mean of all values. Another measure is the mean absolute error (MAE), which is less prone to outliers. It is calculated by taking the absolute value of the errors, rather than squaring them. Lower is better in both cases, and here's a basic example:

# Suppose that:
#  yTest is your hold-out set for cross-validation
#  Your model was trained using xTrain and yTrain
#  fit1.predictions holds the predictions from using your fitted model on xTest
fit1.meanSquaredError = mean((yTest - fit1.predictions) ^ 2)
fit1.meanAbsoluteError = mean(abs(yTest - fit1.predictions))

Many ML packages calculate this for you, and let you select which to use for cross-validation. From the documentation, for glmnet uses MSE for cross-validation by default, with the option of MAE. (See type.measure on p. 4.)