I'm doing some machine learning and get a set of optimum weights in the end. I'd like to verify that these weights are by and large the same no matter how many times I train on the data. I assume that my cost function has a global minimum.
How can I measure the similarity between two set of weights that I got from training two times?
Weight similarity doesn't give much information about which weights perform learning task better. The metric which does is the classification error. So instead of comparing weights it makes sense comparing classification error which should be calculated on a separate validation set to avoid overfitting.
The first part of the question has negative answer. Because you are dealing with multiple weights depending on a starting point the problem you are solving is non-convex which means it has multiple local extremums. In general it is computationally infeasible to estimate ALL extremum of such functions in order to find global one. There methods which can explore space better ( like multiple random restarts etc.) but there no guarantees that they will found global optimum.
Since you've got 2 sets of weights, this scenario could arise in 2 cases. 1. The machine learning algorithm you are using is an iterative or bayesian algo rather than a deterministic one. 2. You are using different sample of training data to train the weights.
In the first case, you anyways have a posterior distribution of weights so you can derive the final weights using any posterior point estimates.
In the second case, you basically want your model to 'generalize' well. so in this case you could use any cross validation or training/testing sample methods.
Instead of answering you question, I've suggested alternatives to what you're doing because, in general, there aren't any ways to compare similarity between weights. You basically try to get a distribution of weights and then choose your final point estimate.
Hope it helps.
Unfortunately, unless you have specific knowledge about the geometry of the weight space for your model, it is not possible to know if a global optimum has been found [Random Restarts in Global Optimization; pp. 17]. Consider a weight space that has a local optimum that is nearly as good as the global optimum, but is geometrically far away from the global optimum. Depending on the starting weights used, the weights you find on any given optimization run may end up in the local optimum and look very dissimilar to the weights that end up in the global optimum. So a measure of similarity is really no use unless you run the optimization enough times that you can distinctly identify the two ending weight clusters. Additionally, if the slope of weight space is very gradual around the global optimum from one approach and very steep from another approach, the ending weights will likely look very dissimilar to each other. The weights from the steep approach will be clustered tightly together taking on very similar values, but their values will also be very similar to bad solutions that they passed through on the steep approach. On the other hand, the weights from the gradual approach will be more widely dispersed and relatively dissimilar to each other. These unfortunate properties make comparison of ending weight values very difficult. Instead I would just use random restarting of your algorithm and ensemble a group of the best weights found.
