0
$\begingroup$

I have optimized a cost function by constrained minimization using cross validation starting from 200 or so initial points. I have traced the optimization path of each chain, so I have many function evaluations.

I have noticed some things I expected to see, such as certain parameters that I figured would need to be low in order to reduce the value of the function being lower than average at optimal points in the cost function, etc., but I don't know the best way to formally test this. I could comment on them in the discussion section of course, but I am hoping for a more thoughtful analysis.

I get significant results using ANOVA in R with the function evaluations as the outcome against the parameters. But I feel like this might have some problems since some of the parameters are correlated (a subset of them must sum to one) and are mostly not normally or log-normally distributed.

So, after parameter optimization for a machine learning problem, how are the parameters themselves analyzed?

Improve this question
$\endgroup$
8
  • $\begingroup$ Bit wonky. If you run smth like a regression of the parameters against the outcome you implicitly assume that there is a linear relationship between the variables. But if this were true, you wouldn't need to run the nonlinear minimisation in first place. $\endgroup$
    – appletree
    Dec 19, 2017 at 21:27
  • $\begingroup$ @appletree I agree, hence my hesitation to consider significant ANOVA results to be very meaningful. $\endgroup$
    – Jared Becksfort
    Dec 19, 2017 at 21:32
  • $\begingroup$ I guess if you have some prior hypotheses like "feature 10 should have a lower-than-average value at the end of the optimisation" for many features you might consider something like a Wilcoxon rank-sum test: Take the set of features you expect to have large values and compare it to the set of features you expect to have low values, based on their final values at the end of the iterations. $\endgroup$
    – appletree
    Dec 19, 2017 at 21:46
  • $\begingroup$ Interesting suggestion, thanks. I will look into it. $\endgroup$
    – Jared Becksfort
    Dec 19, 2017 at 21:49
  • $\begingroup$ Two things I don't yet get: how did you define (find?) your parameters and your cost function -- both have a huge impact on the outcome. And also, what do you mean by "parameter analysis", what do you want to do? What are you trying to explore in the first place? E.g. function regression? $\endgroup$
    – cherub
    Dec 22, 2017 at 16:59

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.