Is it logical to name low correlated features as valuable and choosing the low corelated ones for classification? Or it depends on the algorithm used for the purpose? How do I need to interpret a correlation matrix,then?
...It is still unclear to me what exactly your goal is, but I will give this a stab and edit (or delete in the worst case) the answer later on.
You have a number (probably rather high) of features and want to classify. And you wonder about whether you should choose features which are highly correlated with other features or not?
First, consider the following: If the feature is highly correlated with the class labels, it will also be correlated with other features with highly correlate with the class label.
You could also look at the correlation within either the positive or negative class, but then you will miss the correlation with the class labels. It is entirely possible that Feature A correlated with Feature B within the positive class due to common noise and within the negative class but that Feature A correlates with the class labels and Feature B does not.
Long story short, I would not exclude any feature because of correlation (less than 1). Regularized methods like elastic net and lasso will only select correlated features if it helps. Others like Random Forest might use both, which can also increase robustness if they can be independently missing.
Which more details about what you trying to do, this information could also be more specific.