I am trying to estimate the performance of a machine learning model on time series data. I saw the example of model evaluation using Monte Carlo Estimates from the book "Data Mining With R Learning With Case Studies", which said:

According to https://en.wikipedia.org/wiki/Monte_Carlo_method Monte Carlo methods vary, but tend to follow a particular pattern: 1.Define a domain of possible inputs. 2.Generate inputs randomly from a probability distribution over the domain. 3.Perform a deterministic computation on the inputs. 4.Aggregate the results.

My understanding is in this example randomly select points means that the probability distribution is the uniform distribution. And then the training and testing sets are decided based on the randomly selected points. Then evaluates the model using these training and testing sets. Finally the expectation of evaluation results represents the expectation performance of the model.

My question now is how to choose the R value that can be sufficiently for evaluating the model?

What I think the normal Monte Carlo methods are like Monte Carlo simulation or Monte Carlo integration that need to make use of many distributions and more complicated calculations. And here I think this example are more likely to apply the concept of Monte Carlo to the way of model evaluation.

  • $\begingroup$ Perhaps you could explain what you understand a "regular Monte Carlo method" to be. Otherwise we won't know what you are trying to compare this passage to. $\endgroup$ – whuber May 25 '16 at 3:44
  • 1
    $\begingroup$ I think I need to learn more about the Monte Carlo methods. I changed my description and question. Thank you. $\endgroup$ – Sabrina Hu May 25 '16 at 19:27
  • $\begingroup$ @SabrinaHu "Monte Carlo" is often used as a general name for using simulation methods that employ generating random numbers (check stats.stackexchange.com/questions/174133/… and stats.stackexchange.com/questions/126904/…). The quote you posted does not say anything about "normal Monte Carlo", it mentions only "Monte Carlo" -- here you could read it simply as "simulation". $\endgroup$ – Tim May 25 '16 at 20:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.