What do you do when your test step has dissapointing results? I'm working with a Random Forest classifier and wanting to wrap my head around the train-validate-test cycle. So, as far as I understand the process of training is as such:
1) Train with training data set
2) Validate: run classifier on separate set of ground truth and fiddle with parameters until desirable results are achieved
3) Test: run classifier on yet another set of ground truth to determine a more realistic measure of accuracy of your classifier.
If I'm off the mark please correct me!
So, here is where my question arises. If after the test phase the error is still higher than desired then what is the next course of action? It seems like if you redid the process with the same datasets but different parameters until the results became desirable, then you would just be creating the same issue that would have been the case if you didn't use a final test phase--which is that you would be choosing the best case scenario rather than the most likely. I get in theory the separation of validation and test phases, but I don't really see how the problem is mitigated if after the test phase the results are still bad. 
 A: Good question! The bottom line is that if you work through data exploration and understanding, train various models, use your validation set (and cross-validation), etc, in a disciplined and principled manner, you won't be rolling the dice on the test run, hoping that you'll get lucky and your model will work well enough.
As you work your way through the exploration and modeling, you should be getting a feel for what you can do. In the simplest case, your training results will be better than your validation results which will be better than your test results, so you know an overly-optimistic upper bound on how well you can do. But you'll also have looked at the problem at hand, and what data can help you versus what data you have, how clean is your data, and what its meaning and provenance is. You'll have looked at variables that are too good to be true, to see if there are leaks from the future. You'll have engineered features based on your understanding of the problem. Etc.
If, for some reason, you have reasonable results right up until the test, and the test results are horrible, you know you have a disaster on your hands and something is horribly wrong: you don't understand the problem, you don't understand your data, your data is bad, you don't have nearly enough data, etc. But you probably went wrong much earlier in the process, just following maximal scores and skipping or glossing over important steps in the process.
In any case, you won't be tweaking your models and iterating around Train, Validate, Test again. You'll be making major changes to your data, your features, your models -- the whole shebang. So it may be safe to go through the overall process again, once. Maybe.
When you're first learning, it's not unusual to get to Test and have problems. As you get burned in various ways, you'll learn what to look for up front, so you don't get to the end and step on a landmine you laid a couple of months before.
