Exclude highly correlated instruments I have 3 financial instruments: A, B and C. And have 3 available correlation pairs:
(A, B) = 0.9
(B, C) = 0.9
(A, C) = 0.5

I need to exclude highly correlated instruments. i.e. only B, because A and C have low correlation (0.5). How I can do it algorithmically on every count of instruments?
 A: In your variance-covariance matrix, just pick the minimum of each column or row. (It doesn't matter which you pick since the matrix is symmetric). You keep the minimum and everything else is thrown out.
A: $r = .5$ is not a low correlation; it's pretty strong for meaningfully distinct variables. $r = .9$ suggests distinctions aren't really very meaningful; maybe this is why you're trying to exclude them. This seems like a problem of determining a threshold where correlations are low enough to indicate meaningful degrees of independence.
If you want to retain more than just the minimally related instruments as per @rocinante's answer (which also wouldn't work if even the minimum correlation is too high for your purposes), you could simply apply a fixed threshold based on conventional interpretations of correlation strength, like $r = .7$, beyond which some would say you're likely to have discriminant validity problems. You could also judge multicollinearity by calculating the variance inflation factors (VIF) for each instrument; this might be more useful than looking at bivariate correlations if you're more concerned with how redundant an instrument is with the whole set of instruments, rather than its redundancy with any one other instrument. Common rules of thumb for judging multicollinearity (that don't always apply) suggest that any VIF $> 5$ is cause for at least minor concern, and a VIF $> 10$ indicates pretty serious multicollinearity, so you could apply fixed thresholds like that to judging excessively strong multiple correlations as well.
If you want to adjust your threshold somewhat to suit different sets of instruments with different degrees of commonality, you could take into account the average correlation (either bivariate or item-total correlations) of the set and reject instruments with correlations (average or maximum bivariate, or item-total) that are too close or too far above that average.
You might also want to look into latent variable modeling, which is a pretty deep subject...but even basic factor analysis produces estimates of commonality / uniqueness for individual instruments, which might give you a useful alternative to the VIF.  Latent variable modeling generally aims to determine whether separate instruments represent common factors, so if you're looking to ensure that each instrument represents a different factor, this could be a useful class of methods for you. Analysts tend to use these methods to seek unidimensionality (redundancy, basically) rather than unrelatedness in a pool of potential instruments though, so if you do look into these methods, bear in mind that your objective is likely to oppose others' typical objectives. You would be using a method like factor analysis in a way that others might describe as "wanting it to fail" (i.e., to indicate maximal multidimensionality and a lack of common factors).
However, I should also warn that if $r = .5$ is a relatively low correlation in your line of work, you're probably likely to find at least one common factor.  If all your correlations are $>.3$, your first ("general") latent factor is probably large, and you're probably looking for instruments with low loadings on that general factor. If this is the case, and you have a reasonably large set of instruments, and you want to eliminate instruments that relate too strongly to subsets of the entire set too, you might want to look into exploratory bifactor analysis too. Again, you'd want to seek items with relatively low loadings on both the general factor and on group factor(s).
