Advice on classifier input correlation I have a lot of data on previous race history and I'm trying to predict a percentage chance of winning the next race using Regression, kNN, and SVM learning algorithms.
Say a race has 5 runners, and each runner has a previous best course time of, say $T_i$ (seconds).
I've also introduced an additional input for RANK of previous best course time of the 5 runners with value 0 to $1 - \frac{T_i-T_{min}}{T_{max}-T_{min}}$
My question is: does introducing both the absolute best course time and rank best course time cause any problems?
I understand that these inputs are likely to correlate but if someone runs a world record time they are more likely to win easily but this will get lost using the rank input only which would assign them a rank of 1.
 A: It depends on the classifier. Some classifiers (such as Naive Bayes) explicitly assume feature independence, so they might behave in unexpected ways. Other classifiers (such as SVM) care about it much less. In image analysis it is routine to throw thousands of highly correlated features at SVM, and SVM seems to perform decently. 
For kNN, adding more features will artificially inflate their importance. Suppose you have two features, best course time and coach experience. They will influence the distance equally. Now you add another feature 'course time multiplied by two'. This is essentially a replica of the first feature, but kNN doesn't know about it. So this feature now will influence the distance computation more significantly. Whether you want this or not will depend on the task. You probably don't want features to influence the distance more just because you thought of more "synonyms" for them.
A compromise might be to perform feature selection first and then use kNN. This way two "synonyms" of the same feature will be retained only if both are important.
A: I think it depends on if the purpose of your model is descriptive (e.g. considering variable importance or hypothesis tests in the regression) or purley predictive. If it is the former, then certainly input features that are strongly correlated will create difficulties in making inference about how variables impact the output and relate to each other. For example, can you say variable 1 is the most important if it shares most of its variance with variable 2? 
Even in regression, multicollinearity will not impact the coefficients, only the standard error (the estimates will still be those that minimize the squared error) so the predictions are ok. 
I tend to consider colinearity between inputs not that big an issue when building a predictive model. The best and only way to know for certain is to build a model with both variables and then with only the one with the strongest relationship to the target variable and see which produces the best predictions on new data.....
