10
$\begingroup$

Say there is a machine learning algorithm (e.g. classification) that is well known and implemented by the original creators of the algorithm. Yet all you have is the ability to use the algorithm but not see the source code.

Now you want to implement this algorithm and then check if your implementation is correct or not. Since you can't use the same initialization values to the parameters (because you don't have access to the source code of the original implementation) then you can't expect to get exactly the same results. Say that the algorithm gives as outputs probabilities, then your method could give a slightly different result from the original method.

It's true that your implementation could give the correct classification results as with the original implementation but the probabilities could be different.

Given such a scenario, how would you judge that your implementation is correct?

This exact situation happened with a friend whose implementation gave the correct classification results but after all he found a serious bug in his code!

So is there a way to test and validate the implementation of a machine learning algorithm in general? Regardless of type of the algorithm (classification, regression ... etc).

The only problem I can see is not being able to use the same initial values for the parameters. Then maybe there is some kind of a statistical test that can test the results with respect to the initial values of the parameters somehow.

$\endgroup$
4
  • 1
    $\begingroup$ Recall G. Box: "All models are wrong..." :) $\endgroup$
    – Tim
    Commented May 13, 2015 at 12:05
  • $\begingroup$ @Tim can you elaborate on that? $\endgroup$
    – Jack Twain
    Commented May 13, 2015 at 12:12
  • 1
    $\begingroup$ There is no such a thing as a "correct" model, each model is just an approximation of reality, so you are looking for the one that is "least bad" at approximating reality. You cannot expect form an approximation to be "the same" as reality (i.e. "correct") since it simplifies things by definition. $\endgroup$
    – Tim
    Commented May 13, 2015 at 12:17
  • 3
    $\begingroup$ This doesn't seem to be a model issue, but an implementation issue. $\endgroup$ Commented May 13, 2015 at 12:22

3 Answers 3

2
$\begingroup$

There are various metrics for algorithm's performance (precision, recall, f1, etc.). I'd start by searching for a paper by the algorithm's authors where they mention what kind of data they have tested their algorithm on, what are the results (what metric do they mention) of their algorithm on that data. Then I'd search for the same or similar data, run my implementation on it and compare the results with theirs

$\endgroup$
8
  • $\begingroup$ I mentioned in my question that even if you use the same dataset and get similar answers then that is in no way a strong evidence to say that your implementation is correct. I mentioned that my friend had achieved almost identical results although he later discovered a fatal bug/error in his code. $\endgroup$
    – Jack Twain
    Commented May 13, 2015 at 12:11
  • $\begingroup$ Also you can't know if the difference in the results due to different initial parameters values or because of a bug in your code. $\endgroup$
    – Jack Twain
    Commented May 13, 2015 at 12:14
  • $\begingroup$ I think this is fairly good advice, if you want to work with a closed implementation of an existing algorithm you first need to check the sources which includes finding the paper used in the algorithm. If there aren't published peer review papers on the algorithm I would be suspicious about using it in the first place. The next issue I would look at is to find several simple examples, that you can work out easily then check the program against them. A final method is to email the author, there is a human that wrote this code and she should be able to help you out. $\endgroup$ Commented May 13, 2015 at 12:20
  • $\begingroup$ @JonathanLisic that example about my friend was about a method he implemented from a peer-reviewed paper. That advise is the general and commonly used one in academia. But I'm looking for a better way to validate the implementation of a machine learning algorithm. $\endgroup$
    – Jack Twain
    Commented May 13, 2015 at 13:32
  • $\begingroup$ @JonathanLisic just being able to replicate the results of a paper (by having very close results) doesn't mean that your implementation is correct. $\endgroup$
    – Jack Twain
    Commented May 13, 2015 at 13:33
0
$\begingroup$

On the Generalized Testing of Machine Learning Algorithms:

Yes, if there is a known working method, comparing your result to that method over all possible parameters will guarantee that your program is also a known working method. This is usually impossible and always pointless because there already is a known working method.

If there isn't a known working method, then in general no, as a counter point consider this "code" that calculates regression coefficients:

$\hat{\beta} = (X'X)^{-1}X'y + \delta$ where $\delta = 1000$ when $y[1] = \pi$ else 0.

This implementation is right almost all of the time (technically right a.s., but not under IEEE 754), and it is computationally intractable to find it's error.

On the Implementations of Methods:

Standard practice is the same as in all software development small test cases, and constant validation against something known. E.g. if the model under certain parameters has a known closed form solution or is equivalent to another method check that.

Also note that papers's aren't perfect, in most lit reviews I do I usually find a few typo's. Some of these typos actually would cause incorrect results, so always double check your sources.

There is a reason emails are on journal papers (HINT: it's so you can contact the authors). Also note that author's don't generally bite if you are nice to them. Just be courteous, and show that you have done some work. But don't expect them to step through your broken code to find a bug.

If you have the case where an author won't release the code, or won't respond to you, then don't use that method, more than likely the quality of the code was "good enough to publish" but that's about it. There certainly is no shortage of machine learning algorithms out there. It's also worth while checking who has cited the paper in question and see if they have some code.

$\endgroup$
2
  • $\begingroup$ I'm sorry but this is a mis-understanding to the question. So I can only use the method, but I don't have its source code. So there's no way for me to check "all" the initial values of the parameters because I don't know how they are initializing. $\endgroup$
    – Jack Twain
    Commented May 14, 2015 at 9:50
  • $\begingroup$ That was sort of the point. If you have a black box with a few examples and no binary to deconstruct, you are frankly out-of-luck. If this is a 'novel' method and there is a paper you can reproduce the program to the best of your ability, and that's about it. If you have a binary, but can't reproduce the results because of some initial values, you should at least know what object the initial values are (vector, scalar), and check how sensitive the algorithm is to choice of initial values. $\endgroup$ Commented May 14, 2015 at 10:23
0
$\begingroup$

Do you know what "correct" is for your situation? Determine what correct means then test your implementation against your data and see how close you are.

Unless your test coverage is 100% you'll never know if you have a nasty edge case bug in your code. You wouldn't if you had the initial parameters of the algorithm you are modeling unless your test touched the code where the bug lives.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.