What are some techniques by which we can verify that a current implementation of a machine learning algorithm is correct? Is using the results of a benchmark dataset is enough?
$\begingroup$
$\endgroup$
3
-
3$\begingroup$ Unit testing is the way to go. Make toy data sets for which you know what to expect and verify that you get that results. $\endgroup$– Marc ClaesenCommented Mar 22, 2015 at 13:00
-
$\begingroup$ Thanks a lot for the reply, sometimes that's not a feasible solution, for example if I wanted to validate an LDA topic modeling algorithm? $\endgroup$– MagellaneaCommented Mar 22, 2015 at 14:00
-
1$\begingroup$ @Magellanea it's feasible and natural if you're writing good code $\endgroup$– shadowtalkerCommented Mar 22, 2015 at 15:31
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
This question has several right answers. Here are three:
- Unit test your code. Rather than testing the entire algorithm, test each step separately to make sure it gives the right result. The test cases only need to be as complicated as the individual step. This will enforce good coding habits anyway.
- Use toy data. If you are fitting a SVM, you should be able to find the correct support vector on a linearly separable data set.
- Replicate an existing result. Many, if not most, stats and ML method papers include a demonstration or experiments section. Replicate their demo and make sure your results match.
And of course don't forget the obvious: if the results don't make any sense, re-check your code before you re-check your model.
answered Mar 22, 2015 at 15:31