I would like to cluster 80 days using k-means. Each of the 80 days contains 4 time series (temperature, solar radiation, electricity demand, electricity price) with 288 values each. So all in all I have $4*288=1152$ values for each day and I would like to cluster the days such that similar days with regard to these 4 time series combined will be grouped into the same cluster. For this purpose I use the following code, that I was advised to use (at least for the scaling part) from this answer https://stackoverflow.com/questions/73491673/strange-results-when-scaling-data-using-scikit-learn:

import numpy as np
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.cluster import KMeans

data_Unscaled = pd.read_csv("C:/Users/User1/Desktop/data_Unscaled.csv", sep=";", header=None)

X = data_Unscaled.to_numpy()
X_narrow = np.array([X[:, i*288:(i+1)*288].ravel() for i in range(4)]).T
scaler = StandardScaler()
X_narrow_scaled = scaler.fit_transform(X_narrow)
X_scaled = np.array([X_narrow_scaled[i*288:(i+1)*288, :].T.ravel() for i in range(80)])

kmeans = KMeans(init="random", n_clusters=3, n_init=10, max_iter=300, random_state=42)

print(f"resultClustering Example Day 1: {kmeans.predict((X[20].reshape(1, -1)))}")
print(f"resultClustering Example Day 2: {kmeans.predict((X[30].reshape(1, -1)))}")
print(f"resultClustering Example Day 3: {kmeans.predict((X[40].reshape(1, -1)))}")
print(f"resultClustering Example Day 4: {kmeans.predict((X[23].reshape(1, -1)))}")
print(f"resultClustering Example Day 5: {kmeans.predict((X[10].reshape(1, -1)))}")

In the linked answer it was assumed that my results would suffer from the curse of dimensionality as I have just 80 points in a 1152-dimensional space. Can anyone tell me how can I check if the curse of dimensionality negatively affects the results? I made some test predictions and the are not all grouped into the same cluster which should be correct.

Here is the unscaled input data with shape (80,1152): https://filetransfer.io/data-package/CfbGV9Uk#link

  • $\begingroup$ Unfortunately, the curse of dimensionality prevents us from determining the effects of the curse of dimensionality ;) $\endgroup$ Aug 26, 2022 at 14:21
  • $\begingroup$ @JohnMadden: Thanks John for your comment. But is there no way to determine the quality of the clustering and thus try to infer, whether the curse of dimensionality is active or not in my case? $\endgroup$
    – PeterBe
    Aug 27, 2022 at 9:42
  • $\begingroup$ to be slightly more serious and maybe even useful, the biggest threat posed to a method like Kmeans in high dimension might could be the meaningfulness of Euclidean distance in high dimension; may be worth trying different metrics and seeing how results change. $\endgroup$ Aug 27, 2022 at 15:39
  • $\begingroup$ You have more dimensions than points. This is not only the curse of dimensionality problem but primarily the singularity/multicollinearity problem. The real number of dimensions of your data is 80-1=79 at max. Which is still many. $\endgroup$
    – ttnphns
    Aug 28, 2022 at 1:12
  • 1
    $\begingroup$ @PeterBe Yes! Absolutely do this first. Start with simple things where you can tell if something is dead wrong, and slowly ramp up the complexity as needed/desired, in general with stats :) $\endgroup$ Aug 29, 2022 at 16:43


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.