My statistical training is rooted in mathematical statistics, and taking these methods classes in my M.S. are a bit of a shock at the moment; it is currently difficult for me to be able to understand some of these "applied" methods since I lack experience in the industry.
One of the topics that we've been talking about in my methods classes is the idea of experimental design.
Say, for example, I want to perform an experiment on the effectiveness on an educational program that claims to raise test scores of K-12 students.
In the methods classes, they've taught the following to pursue such a problem: make sure you have a good research question, a good data gathering method, a randomized experiment, homogeneous treatment groups (i.e., one treated with this program, one perhaps not) ideally of equal size, and then run a $t$-test (or some sort of nonparametric hypothesis test), and it's all fine and dandy, right?
I have little faith that this is how it works in reality.
I've learned that, sure, you might have to do some convenience sampling. But other than that, I have no idea how to implement experimental design other than what I've learned from a textbook.
Are there any textbooks, readings, etc. that explore these issues in practice (and ideally, don't gloss over the math - I don't need detailed proofs of everything, but I don't want to be told that everything is "obvious," for example)?