# Daily versus monthly averaged values for better cluster analysis

I am using CLARA method to run a cluster analysis of sea surface temperature in the Mediterranean. Temperature daily values are on a regular grid (0.25 degrees grid mesh). Data are stored in monthly files with 30 to 33 columns depending on the month; first two columns are longitude and latitude while the rest stand for the daily SST values, every row representing a lat-lon point.

I have also computed monthly average SST for each point, so there are monthly files with just lon,lat and average SST

>head(datos)
lon    lat      sst
1 18.875 30.375 291.4571
2 19.125 30.375 291.5510
3 19.375 30.375 291.5977
4 18.375 30.625 291.3013
5 18.625 30.625 291.3635
6 18.875 30.625 291.4942


Please find data sets here (output of dput(datos) is too extensive) for daily values and for monthly average.

Then I applied CLARA method to both data sets for the same month but get a different optimum number of clusters; 4 clusters were found for the daily values while 6 are found for the monthly average.

Cluster map from daily values (clustering the n daily values as a whole)

Cluster map from average monthly values As a meteorologist not expert in statistics I'm not sure the reason for this difference. I "assumed" that average would lead to a more uniform dataset and then, maybe, to a lesser number of clusters. Then, my question is how do I decide which data set is better to run the cluster analysis? daily values monthly aggregated? monthly averaged value?

From a meteorological and oceanographical point of view the monthly averaged map makes more sense to me. Usually, SST to the east of Corsica and Sardinia is higher than in southern France but the example maps are from January and winter SST shows a smooth gradient from north to south so it is possible they are in the same cluster, not usual but possible to some extent.

But as I have monthly files running from 1982 to 2015 I would like to use a robust and objective statistical method to set the optimal number of clusters and then think about their physical significance.

I have run this analysis in R with the following code

library(sp)
library(cluster)
library(ggplot2)

# Path to data
ruta_datos<-"/home/meteo/PROJECTES/VERSUS/TREBALL/"

# List monthly files
files <- list.files(path = ruta_datos, pattern = "SST-mitja")  # For monthly averages
# files <- list.files(path = ruta_datos, pattern = "SST-mitja")  # For daily values

# Run CLARA cluster analysis
for (i in 1:length(files) ) {

# Subset data to remove longitude and latitude columns, cluster just SST values
x<-datos[3:length(datos)]

# Add lon-lat to the output data cluster
clustdat<-as.data.frame(cbind(datos$lon,datos$lat))
colnames(clustdat)<-c("longitud","latitud")

# Select optimal number of clusters
asw <- numeric(10)
for (k in 4:10) asw[k] <- clara(x, k) $silinfo$ avg.width
k.best <- which.max(asw)
cat(files[i],"silhouette-optimal number of clusters:", k.best, "\n")

# CLARA cluster analysis
clarasst<-clara(x,k.best,correct.d=TRUE)

# Add CLARA clustering to clustdat
clustdat$cluster<-clarasst$clustering

}

# Visualize map
ggplot(data = clustdat, aes(x = longitud, y = latitud, fill = cluster)) + geom_raster(interpolate = TRUE)


Is this the right way to get the optimal cluster number? Is a monthly SST average as representative as the whole set of daily values? Why are the number of clusters different?

• What do you want to use the clusters for? – dimpol Nov 18 '16 at 9:54
• Usually for k-mean cluster analysis, sum of the squared error (SSE) is the optimal way for finding the precise number of clusters. – Dark_Knight Nov 18 '16 at 10:04
• @dimpol I want clusters to correlate different and significant areas with precipitations around the Med basin. And also to look for climatic trends at a smaller scale than the whole basin. – pacomet Nov 18 '16 at 10:19
• This question is not suitable for migration to Stack Overflow. Please consult our help center for guidance on how to get more attention for a question. – whuber Nov 21 '16 at 20:22

1. Decide whether to scale or standardize the data

Having some variables with a very different range/scale can often create problems: most of the “results” may be driven by a few large values, more so that we would like. To avoid such issues, one has to consider whether or not to standardize the data by making some of the initial raw attributes have, for example, mean 0 and standard deviation 1.

2. Decide which variables to use for clustering

The decision about which variables to use for clustering is a critically important decision that will have a big impact on the clustering solution. So we need to think carefully about the variables we will choose for clustering. Good exploratory research that gives us a good sense of what variables may distinguish people or products or assets or regions is critical. Clearly this is a step where a lot of contextual knowledge, creativity, and experimentation/iterations are needed.

3. Define similarity or dissimilarity measures between observations

Most statistical methods for clustering and segmentation use common mathematical measures of distance. Typical measures are, for example, the Euclidean distance or the Manhattan distance (see help(dist) in R for more examples).

4. Visualize individual attributes and pair-wise distances between the observations

5. Select the clustering method to use and decide how many clusters to have

The Hierarchial Clustering method, as we do not know for now how many segments there are in our data. Hierarchical clustering is a method that also helps us visualise how the data may be clustering together. It generates a plot called the Dendrogram which is often helpful for visualization. As always, much like Hierarchical Clustering can be performed using various distance metrics, so can Kmeans.

6. Profile and interpret the clusters

Having decided (for now) how many clusters to use, we would like to get a better understanding of who the customers in those clusters are and interpret the segments.use snake plots to help your self understand the clusters.

7. Assess the robustness of our clusters

Two basic tests to perform are:

• How much overlap is there between the clusters found using different approaches? Specifically, for what percentage of our observations the clusters they belong to are the same across different clustering solutions?
• How similar are the profiles of the segments found? Specifically, how similar are the averages of the profiling attributes of the clusters found using different approaches?
• @gd-n Thanks for your extensive answer. I will try hierarchical clustering to explore how the data clusters. Regarding variable selection, I don't use longitude nor latitude, only daily sst so values are relatively uniform across a month. I tried scaling and CLARA results look very similar. – pacomet Nov 24 '16 at 7:24
• @pacomet , hope you got what you wanted. – GD_N Nov 27 '16 at 1:25

There are a number of interesting pieces to your question.

1) What is the correct number of clusters for this problem?

2) Why are there more clusters in the monthly data than the daily data?

These are quite different questions though, so let me answer them one at a time.

For 1), the way you're doing it is certainly a defensible choice. By optimizing a well-defined criterion, you have a process that someone else, doing the same analysis, would be able to replicate.

However, choosing the number of clusters is NOT a solved problem. There are still many human decisions that go into clustering, many of which GD_N discusses. What is important to emphasize explicitly though, is that your expertise matters here -- statistics cannot give you a definitive answer. If you want to be more confident, I would run several different clustering algorithms. Hierarchical clustering is powerful, but asks you to choose k. There are Bayesian clustering methods, such as the Chinese Restaurant Process, that would choose k automatically. Just because it finds a number doesn't mean it's THE BEST number, but it might make you feel better to see multiple methods agreeing.

For 2), I would guess that the difference has to do with the noise. By choosing only monthly data, you're "smoothing" the daily fluctuations. Imagine if the edges of each cluster in your 6-cluster map were fuzzier -- they would look a lot like your 4-cluster map. By ignoring the "fuzz" though (daily, random, noise) you can see the "true" patterns more clearly. Think about it this way -- do you really think the difference in temperature between two days is meaningful? How about the difference between two months?

Overall, I would say that if the monthly map makes more sense to you, as a subject matter expert, I, as a statistician, would trust it more.

• Hi @sophologist and thanks for your advice. A new question comes from your answer. If I use the daily values, 30-31 columns data frame with SST values, what I am actually clustering? SST or days? probably it is better to use only monthly values so I'm sure I'm just clustering SST – pacomet Nov 28 '16 at 8:46
• Daily data absolutely contains some noise but noticeable changes can arise from beginning to end of any month if you have a precipitation event that cools sea surface or noticeable cloudiness that change solar radiation. Monthly average is an indication of general character of SST and can hide some interesting daily patterns. But it is necessary to reach a consensus point between the daily variability and the representativeness of the averages. And SST does not know in which month we are, maybe it should be better to identify "seasonal" patterns. – pacomet Nov 28 '16 at 8:52
• @pacomet If you're looking for a middle ground, which is not a bad idea, then I would try averages over a couple other different windows. -- 3-day, 5-day, 10-day, for example, at the very least to see how stable these clusters are to smoothing. You might find that many windows lead to the same conclusion, while only a few lead to another. – one_observation Nov 29 '16 at 15:04
• @pacomet With respect to what you're clustering -- with days, you're clustering 30-dimensional vectors of SST, and with months, 1-dimensional vectors of SST. – one_observation Nov 29 '16 at 15:08

As for the number of clusters used in your analysis, I would not put too much emphasis on the physical interpretation of the number of clusters; choosing the number of clusters in these analysis is more of an art than a science.

The various methods for chosing the number of clusters can be found here:

How to decide on the correct number of clusters?

Now, as for a simple first pass to decide which dimensions are most reliable see

https://datascience.stackexchange.com/questions/1191/using-svd-for-clustering

and the ensuing discussion

For a random matrix, shouldn't a SVD explain nothing at all? What am I doing wrong?