1
$\begingroup$

Can I use cross-validation k-fold for making FIS and optimization it with ANFIS or ANFIS itself has cross-validation?

First my data was divided into two group randomly. One for making model and other for testing best model. Then first group was used for k-fold cross-validation. FIS and ANFIS were performed and best model was selected. Then second group was used for test best model.

Is that method true?

$\endgroup$
1
$\begingroup$

It actually depends on how you do want to measure the final performances.

  • if you want to measure the FIS/ANFIS performances on the test set
    Your method is fine. However the split should be pretty much 70% for training & cross-validation and 30% for testing. You perform then cross-validation on 70% of the data, select the "best results", train again the model with such "best results" and then use such model on the validation set to gather the final performances.
  • if you want to measure the FIS/ANFIS performances directly through cross-validation
    There's no need to split the dataset. You might as well use all of it and gather the best, final performances as the "best results" from the cross-validation itself.

I most certainly will recommend the first approach. The cross-validation is often used to tune a parameter, not to gather the final performances.

$\endgroup$
13
  • $\begingroup$ Thanks, but a question! , After that I'm select the best result, why am I train the best model again with ANFIS? Best result is model that has best correlation coefficient or minimum cross-validation RMSE. With this model I can evaluate a test data for correlation coeff. and RMSE for prediction. is that? $\endgroup$
    – Omid Omidi
    Mar 30 '16 at 22:00
  • $\begingroup$ Indeed. You can evaluate models measuring performances on the test set. So you train and cross-validate the model and then measure the performances on the test set. $\endgroup$
    – AlessioX
    Mar 31 '16 at 6:18
  • 1
    $\begingroup$ Recall that the correlation coefficient is strictly related to the number of samples, so if the two sets have different sizes is absolutely normal to have different correlation coefficients. Also recall that the cvpartition() selects indices in a random fashion, so when testing your code is good practice to avoid taking a single run as granted and you might want to perform several runs $\endgroup$
    – AlessioX
    Apr 3 '16 at 7:48
  • 1
    $\begingroup$ The repeated runs have nothing to do with the correlation coefficient and/or any other performance parameters. It was just something that I wanted to add for the sake of completeness given the fact that cvpartition() acts in a random fashion. It was just an observation to raise some awareness on the matter: in the "lucky case" cvpartition() will select the "lucky" combination of indices that give good performances but in the "unlucky case" cvpartition() might select combination of indices that give bad performances. That's why is unusual to take a single run as granted. $\endgroup$
    – AlessioX
    Apr 3 '16 at 8:06
  • 1
    $\begingroup$ To my knowledge, there are no size-independent parameters. Even if the number of samples N is not explicit in the formula, all of these parameters depend on the mean and/or standard variation and/or variance and all of these parameters are intrinsically related to the number of samples. Some examples of common parameters are: RMSE, MAE, Pearson's-R, coefficient of determination (in case of estimated/imputed values) and all of them depend at least on one of the above (number of samples, mean, variance, standard deviation). I reckon this also partially answers the question on SO $\endgroup$
    – AlessioX
    Apr 3 '16 at 8:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.