I know that some of the questions I am asking here have been answered in a general case in the two questions I am referring to in the problem section. Nonetheless, I am asking for a very specific case here. --> The representation of a high-dimensional data set by a low number of data points
Background: I am currently working on a European electricity sector model, which should allow the user to model the development of the sector for the foreseeable future. As the model can grow very big with a lot of input data (and thus be almost infeasible), I am trying to reduce my input data to a minimum. Up until now this problem is solved by choosing snapshots of the data set in a look-on fashion and thus choosing by a sense of proportion and not a scientific method.
Problem: In particular, I am trying to best represent a high-dimensional (numerical) data set, with 144 dimensions and 8760 data points, by a small number of representative data points via the use of a (or better some) data-mining method(s).
My data set actually consists of 144 time series, each describing a feature of the European electricity sector. In other words, I have electrical load, Photovoltaics-infeed, Wind-On- as well as Wind-Offshore-infeed for 36 European countries (hourly --> 8760h). Ergo 4*36 = 144 dimensions.
As I already described, I would love to find a (or better some) best-possible representation(s) of these 8760 hours in order to significantly reduce the number of points in time.
The best-case solution would be, that I would be able to find multiple methods in order to solve the above-mentioned Problem. A next step would be to compare the found methods and their solutions, respectively...
Status-Quo: Unfortunately I am rather new to the data-mining/unsupervised-learning topic. I read a lot over the last few weeks and tried to get acquainted with different methods. Nonetheless, I have serious doubts as to the feasibility of my undertaking.
Having a look at this question 1 and this question 2 made me wonder how much use "classical" methods, meaning partitioning and hierarchical methods can be in my case. Especially regarding the curse of dimensionality.
In order to get around this specific problem I ended up reading a lot about sub-space clustering. As I am pretty sure that all "my" dimensions are relevant and I need a snapshot of all dimensions at any given time, I don't see how this kind of clustering can have viable (wholesome) solutions to the problem.
Somehow, due to my skepticism as to what extend my problem is really feasible with regard to the usage of unsupervised learning I am feeling a little bit like I am at a dead end here and don't really see how this problem can be solved in a meaningful manner. Because of that, I would be very delighted if some of you could help me with my questions and in doing that shed some light into the dark of Clustering high-dimensional data sets.
Questions:
I) Does it make "sense" to use a (plain) partitioning and/or hierarchical method in my specific case? (Sorry, if I cannot infer this from the answers to the other questions. But I really mean in my particular case!)
II) Am I wrong to assume that sub-space clustering could give a lot of clusters in subspaces, without really computing clusters (in all dimensions) that allow me to wholesomely consider the European electricity sector?
III) Apart from the already mentioned methods, which other methods could be feasible in this particular setting? What can I read into?
