My task is to develop a system that will take in a series of measurements and return the probability that an object is a type 1, type 2,... type n. I will refer to the system I have to create as a the classifier. I know all the possible types, so I know it must be one and only one of the types.

I have to develop my system based on some data I have been given from a simulation. The physics-based simulation generates both the truth data, what is actually occurring in the simulation, and the simulated measurement data, what the simulated sensors are measuring. Unfortunately, I don't have as much data as I would like, and obtaining more data by running the physics-based simulation is not an option at this time, so I must make do with what I have.

My superior believes that using a Gaussian Mixed Model (GMM) would produce the best classifier, so my solution must use GMM. His proposed solution is to use the Expectation-Maximization Algorithm to develop the classifier, only using the truth data. To measure how likely a given measurement is to be a given type, we measured the Mahalanobis Distance between the mean and covariance given by the GMM and the measurement. The type with the smallest Mahalanobis Distance, we thought would be the most likely type of the object.

Once we went through our initial testing, we could measured the effectiveness of our classifier by plugging in the simulated measurement data and seeing how we did. (We simply looked at which type the classifier decided was most probable).

Our initial performance was dismal, we found that nearly everything was being classified as type 1. The only time any other type was being correctly classified was when the truth and the measurement were nearly identical. After examining the data more closely, the truth data is tightly clustered for all types except type 1, causing the covariance associated with the GMM to be very small. For type 1, the covariance is large, because the particular data set I have shows a lot of variability.

I realize that the "best" solution would be to get more simulation data, but that's not an option at this time, so I'm trying to figure out how else I can improve my classifier performance.

My question is, should I artificially add noise into the training data to give the classifier a "preview" of the measurement data or should I try to find an alternative method of classification?

I'm not a mathematician, I'm a engineer, so if I sound like I don't know what I'm talking about, its because I am in unfamiliar territory. So please let me know if I'm using the wrong terms or if I'm being unclear and I'll do my best to clarify.

The data has a total of three dimensions, one of which is time. I can't seem to find a way to use time dimension effectively because some simulation runs will last 500 seconds and others will last only 200 seconds. I realize that I could increase the number of dimensions by taking the derivative of one of the dimensions with respect to another, or by using a nonlinear equation to create a new dimension based on one or two of the variables, but I'm not sure how helpful that would be.

  • $\begingroup$ There are lots of things missing before we can help you. What sort of data are you simulating? How many dimensions does it have? How is it being generated? How are you trying to classify it? What does it mean to say the statistical distance of one class was shorter? $\endgroup$ – Peter Ellis Jul 20 '13 at 2:35
  • $\begingroup$ Are you really using a classifier, or are you using a predictor? If using a classifier why do you require a forced choice as the output? And are you familiar with proper accuracy scoring rules? $\endgroup$ – Frank Harrell Jul 20 '13 at 17:03
  • $\begingroup$ I tried to clarify things as much as I could by rewriting most of the question above....I'm not familiar with the term, proper accuracy scoring rules, but based on the Wikipedia page, it sounds like its saying that the sum of the probability that the object is type 1, type 2,... type n is 1.0. Is there more to it than that? $\endgroup$ – HardcoreBro Jul 20 '13 at 20:27
  • $\begingroup$ An example of a proper scoring rule is the Brier score, which is the average squared difference between the binary outcome $Y = 0,1$ and the predicted probability that $Y=1$. Like the likelihood function, optimizing the Brier score can result in an optimum predictor. $\endgroup$ – Frank Harrell Jul 20 '13 at 21:38
  • $\begingroup$ Ok, I think I understand, but I'm not seeing how I am supposed to use that information to deal with my situation. My question is "Does adding noise to truth data before using it in the training of an expectation maximization classifier ever make sense?" If it does, how would I be able to determine if this is one of those situations? $\endgroup$ – HardcoreBro Jul 20 '13 at 21:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.