How to cluster multiple datasets

Question

I've got multiple datasets containing measurements collected at different nodes in the real network. Each dataset is associated with one node.

Because some nodes have got similar properties and their datasets follow a similar trend, I would like to group the datasets into clusters. Each cluster should contain all datasets with a similar trend.

As per my understanding, standard clustering techniques usually apply to a single dataset and group data points of a single dataset. How can i apply clustering like k-means to my scenario without tearing datasets apart (i.e. without datapoints from one dataset being added to different Clusters)? I am thinking of finding datasets meta features like statistical features of each data sets and cluster them based on these features. Is there any other way you think more appropriate?

Each dataset in my case has fewer samples in order of few hundreds.

Answer 1

If I understand correctly, you can construct a new "dataset", in which each node (or original dataset) is one observation (point). You need to devise a metric for calculating the "distance" between the nodes. Since you say that some nodes share similar properties and the associated underlying datasets follow a similar trend, you can start from these notions of similarity when devising the metric.

For a more detailed answer you'll need to provide more details about your data, nodes, and what you mean by "similar".

Answer 2

Each cluster should contain all datasets with a similar trend.

Note that, this is your assumption, but can we verify this assumption in data?

Verifying this assumption on data is a very important step. This is because it is possible that data driven results will conflict with your knowledge driven results, i.e, data collected from different nodes will have very similar or even indistinguishable measures. In the extreme case, the nodes do not make any differences, we can just combine all the data into one data set.

Here is some simulation on 1 dimension data with mixture of gaussian model.

• In scenario 1, we have 2 nodes/sensors, and data generated from them are different from each other (in the example, both of them are generating data from normal distribution with different mean and variance.). After verify the data in 2 'clusters' we can fit a mixture of Guassian model on the whole data set.

• In scenario 2, although we have 3 nodes/sensors, the data generated from them are really similar to each other. And it is 'OK' to combine all the data into one big data set to run analysis.

Therefore, it is important to verify the assumption in the data to see how to cluster "multiple data sets"

set.seed(0)
par(mfrow=c(1,2))

# senario 1, data collected from different node are really different
node_1_df = rnorm(1000)
node_2_df = rnorm(1000, mean=5, sd=2)
data = c(node_1_df,node_2_df)
hist(data,50, main='senario 1')
grid()

# senario 2, data collected from different node are similar

node_1_df = rnorm(1000)
node_2_df = rnorm(1000, mean=1.2, sd=1.2)
node_3_df = rnorm(1000, mean=1.3, sd=1.3)
data = c(node_1_df,node_2_df,node_3_df)
hist(data,50, main='senario 2')
grid()