Daily versus monthly averaged values for better cluster analysis

Question

I am using CLARA method to run a cluster analysis of sea surface temperature in the Mediterranean. Temperature daily values are on a regular grid (0.25 degrees grid mesh). Data are stored in monthly files with 30 to 33 columns depending on the month; first two columns are longitude and latitude while the rest stand for the daily SST values, every row representing a lat-lon point.

I have also computed monthly average SST for each point, so there are monthly files with just lon,lat and average SST

>head(datos)
     lon    lat      sst
1 18.875 30.375 291.4571
2 19.125 30.375 291.5510
3 19.375 30.375 291.5977
4 18.375 30.625 291.3013
5 18.625 30.625 291.3635
6 18.875 30.625 291.4942

Please find data sets here (output of dput(datos) is too extensive) for daily values and for monthly average.

Then I applied CLARA method to both data sets for the same month but get a different optimum number of clusters; 4 clusters were found for the daily values while 6 are found for the monthly average.

Cluster map from daily values (clustering the n daily values as a whole)

Cluster map from average monthly values As a meteorologist not expert in statistics I'm not sure the reason for this difference. I "assumed" that average would lead to a more uniform dataset and then, maybe, to a lesser number of clusters. Then, my question is how do I decide which data set is better to run the cluster analysis? daily values monthly aggregated? monthly averaged value?

From a meteorological and oceanographical point of view the monthly averaged map makes more sense to me. Usually, SST to the east of Corsica and Sardinia is higher than in southern France but the example maps are from January and winter SST shows a smooth gradient from north to south so it is possible they are in the same cluster, not usual but possible to some extent.

But as I have monthly files running from 1982 to 2015 I would like to use a robust and objective statistical method to set the optimal number of clusters and then think about their physical significance.

I have run this analysis in R with the following code

library(sp)
library(cluster)
library(ggplot2)

# Path to data
ruta_datos<-"/home/meteo/PROJECTES/VERSUS/TREBALL/"

# List monthly files
files <- list.files(path = ruta_datos, pattern = "SST-mitja")  # For monthly averages
# files <- list.files(path = ruta_datos, pattern = "SST-mitja")  # For daily values

# Run CLARA cluster analysis
for (i in 1:length(files) ) {

datos<-read.csv(paste0(ruta_datos,files[i],sep=""),header=TRUE,na.strings = "NA")

# Subset data to remove longitude and latitude columns, cluster just SST values
x<-datos[3:length(datos)]

# Add lon-lat to the output data cluster
clustdat<-as.data.frame(cbind(datos$lon,datos$lat))
colnames(clustdat)<-c("longitud","latitud")

# Select optimal number of clusters
asw <- numeric(10)
for (k in 4:10) asw[k] <- clara(x, k) $ silinfo $ avg.width
k.best <- which.max(asw)
cat(files[i],"silhouette-optimal number of clusters:", k.best, "\n")

# CLARA cluster analysis
clarasst<-clara(x,k.best,correct.d=TRUE)

# Add CLARA clustering to clustdat
clustdat$cluster<-clarasst$clustering

}

# Visualize map
ggplot(data = clustdat, aes(x = longitud, y = latitud, fill = cluster)) + geom_raster(interpolate = TRUE)

Is this the right way to get the optimal cluster number? Is a monthly SST average as representative as the whole set of daily values? Why are the number of clusters different?

Thanks for your advice

Answer 1

1. Decide whether to scale or standardize the data

   Having some variables with a very different range/scale can often create problems: most of the “results” may be driven by a few large values, more so that we would like. To avoid such issues, one has to consider whether or not to standardize the data by making some of the initial raw attributes have, for example, mean 0 and standard deviation 1.

2. Decide which variables to use for clustering

   The decision about which variables to use for clustering is a critically important decision that will have a big impact on the clustering solution. So we need to think carefully about the variables we will choose for clustering. Good exploratory research that gives us a good sense of what variables may distinguish people or products or assets or regions is critical. Clearly this is a step where a lot of contextual knowledge, creativity, and experimentation/iterations are needed.

3. Define similarity or dissimilarity measures between observations

   Most statistical methods for clustering and segmentation use common mathematical measures of distance. Typical measures are, for example, the Euclidean distance or the Manhattan distance (see help(dist) in R for more examples).

4. Visualize individual attributes and pair-wise distances between the observations

   Plotting a histogram should help you understand the distribution well.

5. Select the clustering method to use and decide how many clusters to have

   The Hierarchial Clustering method, as we do not know for now how many segments there are in our data. Hierarchical clustering is a method that also helps us visualise how the data may be clustering together. It generates a plot called the Dendrogram which is often helpful for visualization. As always, much like Hierarchical Clustering can be performed using various distance metrics, so can Kmeans.

6. Profile and interpret the clusters

   Having decided (for now) how many clusters to use, we would like to get a better understanding of who the customers in those clusters are and interpret the segments.use snake plots to help your self understand the clusters.

7. Assess the robustness of our clusters

   Two basic tests to perform are:

   • How much overlap is there between the clusters found using different approaches? Specifically, for what percentage of our observations the clusters they belong to are the same across different clustering solutions?
   • How similar are the profiles of the segments found? Specifically, how similar are the averages of the profiling attributes of the clusters found using different approaches?

Answer 2

There are a number of interesting pieces to your question.

1) What is the correct number of clusters for this problem?

2) Why are there more clusters in the monthly data than the daily data?

These are quite different questions though, so let me answer them one at a time.

For 1), the way you're doing it is certainly a defensible choice. By optimizing a well-defined criterion, you have a process that someone else, doing the same analysis, would be able to replicate.

However, choosing the number of clusters is NOT a solved problem. There are still many human decisions that go into clustering, many of which GD_N discusses. What is important to emphasize explicitly though, is that your expertise matters here -- statistics cannot give you a definitive answer. If you want to be more confident, I would run several different clustering algorithms. Hierarchical clustering is powerful, but asks you to choose k. There are Bayesian clustering methods, such as the Chinese Restaurant Process, that would choose k automatically. Just because it finds a number doesn't mean it's THE BEST number, but it might make you feel better to see multiple methods agreeing.

For 2), I would guess that the difference has to do with the noise. By choosing only monthly data, you're "smoothing" the daily fluctuations. Imagine if the edges of each cluster in your 6-cluster map were fuzzier -- they would look a lot like your 4-cluster map. By ignoring the "fuzz" though (daily, random, noise) you can see the "true" patterns more clearly. Think about it this way -- do you really think the difference in temperature between two days is meaningful? How about the difference between two months?

Overall, I would say that if the monthly map makes more sense to you, as a subject matter expert, I, as a statistician, would trust it more.

Answer 3

As for the number of clusters used in your analysis, I would not put too much emphasis on the physical interpretation of the number of clusters; choosing the number of clusters in these analysis is more of an art than a science.

The various methods for chosing the number of clusters can be found here:

How to decide on the correct number of clusters?

Now, as for a simple first pass to decide which dimensions are most reliable see

https://datascience.stackexchange.com/questions/1191/using-svd-for-clustering

and the ensuing discussion

For a random matrix, shouldn't a SVD explain nothing at all? What am I doing wrong?