1
$\begingroup$

What i am trying to do is to take a machine learning algorithm that is already trained.Run the algorithm several times and collect the output.

Conduct a statistical analysis on the output and try to create an inference regarding the parameters of the algorithm

Is this possible ? Are there known statistical tests which does this or which does something similar to this ?

Adding from the comment

I was not sure whether to add statistical bias or not.I ll try to explain my thought flow when i added that tag. If in fact we can learn something about the algorithm (it could be the parameters used or something else) from the output, then it should be(i am not sure) because the output is biased in some way. My aim to find out if there is some way to learn something about the algorithm used, using the properties of this output.Have you heard of such a thing ?

$\endgroup$
2
  • $\begingroup$ How would a test "determine parameters"? Why is 'bias' in your tags? It doesn't seem to directly relate to the question. $\endgroup$ – Glen_b Feb 12 '14 at 7:23
  • $\begingroup$ Hi Glen, I was not sure whether to add statistical bias or not.I ll try to explain my thought flow when i added that tag. If in fact we can learn something about the algorithm (it could be the parameters used or something else) from the output, then it should be(i am not sure) because the output is biased in some way. My aim to find out if there is some way to learn something about the algorithm used using the properties or this output.Have you heard of such a thing ? $\endgroup$ – user1587457 Feb 12 '14 at 9:13
-1
$\begingroup$

If you have algorithm which otherwise is black-box, but you know which variables were in scoring data-set then you could use this information and output vector to kind of reverse engineer the black-box.

Suppose you have algorithm X, which is proprietary and code is not shown but you see that it produces 0/1 - indicator output vector. Now you could try different algorithms with this output vector and same input variables/features from the scoring data-set. If one of those algorithms could produce same kind of classification results then you might have learned something about the parameters.

$\endgroup$
2
  • $\begingroup$ Hi, thankyou for the reply.Just to make sure that i have understood what you are saying.Is this like configuring different algorithms, and then running them with the same input and then checking each (algorithm,configuration) with the out that the black-box gave us ? Is there some systematic way to approach this ? Have you come across any documents that describes this method ? $\endgroup$ – user1587457 Feb 12 '14 at 9:06
  • $\begingroup$ @user1587457 unfortunately I do not have such document. I would say that you might have to test with several different algorithms and select one which has closest classifying capability (AUC,entropy etc). $\endgroup$ – Analyst Feb 12 '14 at 11:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.