My task is to develop a system that will take in a series of measurements and return the probability that an object is a type 1, type 2,... type n. I will refer to the system I have to create as a the classifier. I know all the possible types, so I know it must be one and only one of the types.
I have to develop my system based on some data I have been given from a simulation. The physics-based simulation generates both the truth data, what is actually occurring in the simulation, and the simulated measurement data, what the simulated sensors are measuring. Unfortunately, I don't have as much data as I would like, and obtaining more data by running the physics-based simulation is not an option at this time, so I must make do with what I have.
My superior believes that using a Gaussian Mixed Model (GMM) would produce the best classifier, so my solution must use GMM. His proposed solution is to use the Expectation-Maximization Algorithm to develop the classifier, only using the truth data. To measure how likely a given measurement is to be a given type, we measured the Mahalanobis Distance between the mean and covariance given by the GMM and the measurement. The type with the smallest Mahalanobis Distance, we thought would be the most likely type of the object.
Once we went through our initial testing, we could measured the effectiveness of our classifier by plugging in the simulated measurement data and seeing how we did. (We simply looked at which type the classifier decided was most probable).
Our initial performance was dismal, we found that nearly everything was being classified as type 1. The only time any other type was being correctly classified was when the truth and the measurement were nearly identical. After examining the data more closely, the truth data is tightly clustered for all types except type 1, causing the covariance associated with the GMM to be very small. For type 1, the covariance is large, because the particular data set I have shows a lot of variability.
I realize that the "best" solution would be to get more simulation data, but that's not an option at this time, so I'm trying to figure out how else I can improve my classifier performance.
My question is, should I artificially add noise into the training data to give the classifier a "preview" of the measurement data or should I try to find an alternative method of classification?
I'm not a mathematician, I'm a engineer, so if I sound like I don't know what I'm talking about, its because I am in unfamiliar territory. So please let me know if I'm using the wrong terms or if I'm being unclear and I'll do my best to clarify.
The data has a total of three dimensions, one of which is time. I can't seem to find a way to use time dimension effectively because some simulation runs will last 500 seconds and others will last only 200 seconds. I realize that I could increase the number of dimensions by taking the derivative of one of the dimensions with respect to another, or by using a nonlinear equation to create a new dimension based on one or two of the variables, but I'm not sure how helpful that would be.