How to implement this classification/labelling problem?

Question

I have a dataset that has 12 explanatory variables for every 1 observation.

I hypothesize that the data is generated by underlying process which undergoes several different phases, and that the 12 explanatory variables would somehow help in identifying clusters.

I want to write a machine learning algorithm that can:

1. At first pass, identify the clusters within the data

2. The data is dynamic in that new (additional) data is generated periodically. After classification (step 1 above), I want to be able to correctly label (i.e. classify) any new data not previously seen into one of the previously identified classes/clusters (or fail gracefully).

I assume that

Y(c) ~ X(c) + error

where:

• Y(c) is a nx1 vector of observations belonging to cluster C
• X(c) is a nx12 vector of explanatory factors that 'belong' to cluster C

Observations in different clusters will differ from each other by having different distribution shapes. That is to say observations WITHIN a cluster will have a different distribution shape compared to observations from another cluster.

I am relatively new to machine learning, and would like some guidance on how to implement such an algorithm (or perhaps one already exists?)

I would be particularly interested in finding out how to 'classify' observations based on determining the shape of the empirical distribution of the observed data.

Last but not the least, I would also appreciate some advice on whether to implement this in R, or Octave (and why).

Answer 1

So..if you dont have the data already into clusters or a set of training data that are already classified ( with their respective labels ) then the first attempt towards solving this problem is an "unsupervised approach".

You could look up.. K-Means clustering algorithms or Expectation Maximization (EM algorithms) or even LDA(latent Dirichlet allocation). These are algorithms that would help you perform clustering based on the features you generate.

Once you get your clusters from these, you get find a way to measure the precision/recall to get an idea of how good/bad the clusters are.

Then you could probably proceed to use this as a training data to develop a model, and classify the "unobserved" data points or "new data" as you call it, you come across using this model.

my two cents!

Answer 2

if the Y() is the class value of the instance you observe that means (if your dataset is useful) other features need to have some classifying structure that separate these different Y() values instances so a cluster algorithm will see that structure on features as well thus you'll have one cluster for each Y() value. that means You have no dist on clusters since you have only one Y() value for each.

However you can get a distribution on data by using a Algorithm that is not defining its own number of clusters and you need to define the cluster number. In that way if you have 4 different Y() value and you set your algo. to give 2 clusters than you have distributions on clusters.

K-means is the one you might use. You can define K as the number < # of possible Y values.

You might use Expectation Maximization a general case of K-means but hard to interpret.