However, how do you correct between tests, if you have several separate hypotheses?
There is a lot of debate going on right now about this. There is almost a consensus now that you shouldn't just go fishing around in your data for significant p-values, as this inflates Type I error (obviously).
However, it is also true that effect sizes are not a product of any previous analyses you have done—they are only products of the data. So why would a test you do 20th be less believable than the one you happened to do 1st because of your a priori thinking?
Obviously, there is a tension here.
Should I still correct for multiple comparisons and how do you generally do that between say 4 t-tests and 6 ANOVAs?
There is no straightforward or agreed upon way of doing this. There are any number of corrections for post hoc comparisons (Bonferroni, HSD, Holm, whatever). A professor of mine had really great advice in saying that: "If there are many ways of doing something, it is because there is no best way of doing it." Or, at least, there is no widely agreed-upon way of doing it.
You could simply divide .05 by 10 (the number of tests you did total, similar to a Bonferroni correction) to be very conservative. However, this might give you a Type II error simply because you were wrong a priori, which we don't want either!
Is doing this many tests fishing and general bad practice?
Collecting data, not finding what you wanted a priori, and then digging around, doing dozens of tests, and then finding something significant is not bad practice. HOWEVER, it becomes bad practice when you communicate the finding to an audience like it was the very first test you did, like you had predicted it the whole time, and hiding any other tests you did beforehand. This becomes a disastrous practice if you are hiding a test you did beforehand that might contradict the significant finding you found.
Should I ignore it completely
No, but be more skeptical than you generally would (I hope you are always skeptical of what one study finds).
could I simply call it exploratory?
Yes. It is OK to include this analysis in a paper. However, be very upfront about how many tests you did beforehand, that you hadn't initially predicted it, and report any analyses that you did in the data that could contradict whatever you found.
Also, do not take one study too seriously. I forgot who said it first—I believe it has been attributed to many people—but the saying goes that, "An ounce of replication is worth a ton of inferential statistics." Instead of getting so in a fuss about how to correct a p-value, why not take the interesting finding you had and replicate it with a well-powered sample?
If I were reviewing a paper and someone says, "We dug around in the data and found X. Here is the study that shows X," I would likely reject.
However, if someone said, "We dug around in the data and found X. Here is a study that shows X. Now here are Studies 2, 3, and 4 that directly and conceptually replicate X." I would say, awesome! Who cares that you didn't predict it at first? You have now done a number of tests replicating it, showing less doubt that it was Type I error.
Overall: Dig around all you want. Get to know your data. But don't find p < .05, go run off and tell people only about that, pretending like you had the idea the whole time. If you find something interesting, think about it, put it in your back pocket, and try to replicate it if you think it is really worth something.
This is a hot-button issue, and I'm sure some may not agree with me. But I think what I've proposed is a reasonable way to approach exploratory analyses when correcting p-values isn't within a test (i.e., I just do HSD on post-hoc comparisons for an ANOVA), but between a number of different, conceptually different statistical tests.
