how to get statistical inference measures for cross validation I am so confused with using cv to evaluate model performance, so the setting is like this: I have 1000 data, and I split that into training and testing set, and then I use that training set to further perform k-fold cv, and I will use the model that gives me lowest error to apply to the test set. Is that correct?
But when performing the cv, how do I get stats inference like p value, and t value etc.....  because there k fold of data, how I can look at stats inference? (maybe take the mean? I do not have any clue!) Or is it even necessary to look at those value and just directly look at RMSE? ( the instructor  and several books mentioned " we have to look at p value and bunch of coeff to make decision).
 A: The priorities in cross-validation lie, not in the model coefficients, from which p-values and t-tests are derived, but on finding optimal parameters for the model based on the average error of the model's predictions during training. I will explain this below, which requires a review and conceptual understanding of the whole purpose of why cross-validation is done in the first place.
When training a machine learning model, we are only given a finite set of observations called in-sample data, which we want to make best use of in preparation for predicting incoming unseen out-of-sample data, which is not given to us.
The purpose of cross-validation is to randomize the in-sample data repeatedly to simulate what the out-of-sample data could look like, on average, in order to find optimal parameters for the model (discussed later). In traditional $k$-fold CV, this means splitting the in-sample data into $k$ parts with the $k$th part being a "make-believe test sample", which we call the validation set, because it simulates what the out-of-sample could be. The randomized repetition of $k$ splits reassures us that what we might think is a good fit, had we only fit the model once, isn't due to selection bias.
Instead of just doing one train-test split on the in-sample data, k-fold CV does this $k$ times, while randomizing the in-sample data, so that there are $k$ fits of the model to help choose the optimal parameters of the optimal, final model. Each of these $k$ fits is usually only evaluated by its MSE, MAE or RMSE of the target variable versus the predicted target variable individually, i.e. the MSE of $y-\hat{y}$, so that after running all of them, there are $k$ different error measurements taken of the training process, for a given set of parameters. Therefore, the answer to your question is that evaluation from CV is not based on model coefficients (from where p-values and t-tests are derived) but on average prediction error. if there is only one parameter to tune in the search for the optimal model, whose values fall in a range from $[0,1]$ which we want to explore in increments of $0.1$, then there will be $k\times 10$ splits in total, $k$ for each different parameter value tried for the example range shown.
Whichever parameter settings has the lowest average prediction error (MSE, MAE or RMSE) across its respective $k$ folds is used to establish the best fitting optimal model from cross-validation.
From this description, it becomes apparent that the model coefficients (for features) during CV training are not the focal criterion. Instead, the focus is on the predictions $\hat y$ for the validation set(s) and how distant they are from corresponding $y$ observations in the target variable, on average, because from the average errors an inference about what the optimal parameters should be can be made (parameters are not to be confused with coefficients). Had the coefficients been the focal criterion of CV, then statistical inference for each CV fold may have been meaningful, and statistical inference measures like p-value and t-tests could be derived from each set of underlying coefficients. Instead, coefficients are given a minor role in cross-validation because the focus is placed on finding the optimal parameterization of the model based on the average prediction error across folds.
A: I assume that you are talking about nested cross validation, in which the inner loop determines hyperparameters that result in the lowest error, and in the outer loop the training set is trained with those hyperparameters and tested on the left-out test set in that fold? 
My guess is that we could take the predicted values from each test set of each outer loop, and input those predicted values with the labels in a ROC analysis in SPSS, since there it gives us not only a AUC but also its p-value ( in the case that you used AUC as your performance measure, that is..).  
