I have optimized a cost function by constrained minimization using cross validation starting from 200 or so initial points. I have traced the optimization path of each chain, so I have many function evaluations.
I have noticed some things I expected to see, such as certain parameters that I figured would need to be low in order to reduce the value of the function being lower than average at optimal points in the cost function, etc., but I don't know the best way to formally test this. I could comment on them in the discussion section of course, but I am hoping for a more thoughtful analysis.
I get significant results using ANOVA in R with the function evaluations as the outcome against the parameters. But I feel like this might have some problems since some of the parameters are correlated (a subset of them must sum to one) and are mostly not normally or log-normally distributed.
So, after parameter optimization for a machine learning problem, how are the parameters themselves analyzed?