4
$\begingroup$

I am attempting to make a Radial Basis Function Network to see if a relationship exists between input/output data that I have been collecting. I have hit a bit of a brick wall with a few issues, and wondered if a subject matter expert could perhaps point me in the right direction?

All of my work is based in Excel 2007. I have 14940 datapoints. There are 32 input dimensions and one "target output" for each datapoint. All input data is normalised to be on a scale of -1 to 1 or 0 to 1 as appropriate. I have written a K-means clustering algorithm to cluster the data into 50 partitions. The centroid of each cluster is set as the centre of each of 50 Gaussian Radial Basis functions. The network architecture is normalised but does not use a regularisation parameter.

I have made a "random" sampling algorithm that picks 13000 training datapoints evenly across the clusters, and "saves" the rest as a validation set.

I have calculated the network weights using the standard matrices methods and then built the final network. I get an RMS error of 1.311, though obviously have no idea how good or bad this is. I thought maybe I could improve on this by adding RBF nodes, so I re-clustered the data with 80 clusters, and run the whole thing again. This time I got an error of 1.307, so barely any better.

I thought that my initial guess of the Basis Function widths (which was near the average distance between the network centroids) was probably not optimal, so I started re-creating the network for many different widths - spanning a whole order of magnitude. I found that the RMS error in the network barely varies at all with varying Basis Function width, which I found a little surprising. I was expecting there to be some variation, pointing to an "optimum value", as per the countless papers and tutorials I have read.

So this is my problem - I dont really know what to do, and wondered if someone with more experience in these things could suggest what I could do next. A few things I am thinking are:

1) Give up, and conclude that there is no pattern in the data and I am trying to get the RBF Network to look for a correlation that doesn't exist. Though this still doesn't explain why I don't see a better correlation when there are 80 centres versus 50 centres.

2) Use more basis functions. I feel like I have a large number of input dimensions and this is confusing or complicating the network. I have tried exhaustive searches on google to try to get estimates of an "initial guess" number of basis functions to try, but keep coming up with nothing. Does anyone have any thoughts or recommendations if my input dimensions are 32 and my training patterns are over 13000? I feel like I might be waaay off the required number of basis functions, and I may need several hundred?

3) Something else? I have checked and rechecked my formulas and macros for "silly" errors (Im an engineer, afterall!) and done a lot of validation by "manual calculation" checks, but cant seem to find anything thats causing it to trip up.

I apologise for the long question, but would greatly appreciate any insights that might help me continue with my model or improve my understanding.

$\endgroup$
  • $\begingroup$ What is the range of the response in the training data? An RMS value of 1.3 is large for, say [1, 10], but quite okay for [1, 100]. $\endgroup$ – Prometheus Mar 9 '15 at 21:45
  • $\begingroup$ Thanks for your response Prometheus. The response is on a scale of -1 to +20. The network is trying to model sporting results, so if the selection was a loss, it target is -1, and if its a "win" its target is whatever-the-odds-were (positive), but is generally up to about 20. This means that there are a many more "-1"'s than there are positive values, because there are significantly more losers, and I am worried that I am giving the network a badly-formulated problem to solve. $\endgroup$ – Paul Mar 10 '15 at 12:38
1
$\begingroup$

I realise that answering (or attempting to answer) my own question may seem a bit silly, but I wanted to post some follow-up/response to my original query to offer some ideas in case any other self-taught RBF-network enthusiasts ever observe something similar. If this post is in contravention to the forum rules, I understand if it needs to be deleted.

After posting my query, I decided to re-run the model with 300 radial basis functions. The resulting error on both the training set and the validation set barely changed, which to me meant that something was fundamentally wrong with the way I was formulating my problem-to-be-solved.

I played around with the "distance" (widths) of the basis functions, but again found that this had very little impact on the RMS error in either the training or the validation set.

In my next "evolution" of the model, I thought that since I now had a large number of radial functions, it wasn't "fair" that every neuron had the same width, and so decided to assign each neuron an individual weight that was related to its distance to its nearest neighbours. Once again I ran the model, and discovered almost no improvement.

Since I had run out of things I could change that might have a benefit - e.g. number of basis functions, widths, etc, I looked at the clustering of the data and thats where I think I have found the cause of the problem - my clustering of the data appears to be complete nonsense!

My input vectors are a mixture of datatypes - some are "continuous scales", some are "binary", some are categorical, etc. I spotted that there were different datatypes when I was writing the clustering code, so I scaled all of the data to be between 0 and 1, or -1 and 1 so that all data was treated fairly.

The trouble is that because I calculate the distance between nodes based on Euclidean distance, certain variables can change a lot but produce little change in distance, which means that the clustering is dominated by certain variables. I recognise that this is baby-school knowledge to a seasoned neural-network professional/enthusiast, but I think that this is the cause of my problem. My clustering looks like its dominated by a few particular variables that are unlikely to be significant to the network, and hence result in the generation of clusters that contain a random mix of lots of data - as long as the "heavy weighted" variable is clustered.

I am now looking into weighting the euclidean distance or using alternative distance measures so that the clustering is more intelligent. The theory is that the RBF network is only as good as the clustering that has been used to set the network centres.

So in summary, its slightly embarrassing but I think if I can partition my data better, it will result in a better network.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

Have a look at this link: http://www.di.unito.it/~botta/didattica/RBF_1.pdf. It has a few pointers at setting the RBF parameters, i.e neuron centers and spread. I usually use all (or some randomly chosen) input data points as clusters centers. In your case, you can first run k-means algorithm for varying number of clusters and pick the k with the best trade-off between error and complexity (k). For spread values, you can either optimize the weights (as suggested in the link) or perform multiple runs.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for the link - I am already familiar/comfortable with most of the content that it presents. I thought that using K-means was "one up" on picking the centres at random. I am finding that most papers deal with setting the spread/centres and very few discuss what a "ballpark" number of centres is if one has X dimensions and N training points. Would you say that if I am finding that varying the centres and widths has no effect on the error, then its a sign that I have way, way fewer basis functions than is required to model the data reasonably? Im sorry if this doesn't make sense! $\endgroup$ – Paul Mar 10 '15 at 15:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.