One useful technique is monte carlo testing. If there are two algorithms that do the same thing, implement both, feed them random data, and check that (to within a small tolerance for numerical fuzz) they produce the same answer. I've done this several times before:
I wrote an efficient but hard to implement $O(N\ log\ N)$ implementation of Kendall's Tau B. To test it I wrote a dead-simple 50-line implementation that ran in $O(N^2)$.
I wrote some code to do ridge regression. The best algorithm for doing this depends on whether you're in the $n > p$ or $p > n$ case, so I needed two algorithms anyhow.
In both of these cases I was implementing relatively well-known techniques in the D programming language (for which no implementation existed), so I also checked a few results against R. Nonetheless, the monte carlo testing caught bugs I never would have caught otherwise.
Another good test is asserts. You may not know exactly what the correct results of your computation should be, but that doesn't mean that you can't perform sanity checks at various stages of the computation. In practice if you have a lot of these in your code and they all pass, then the code is usually right.
Edit: A third method is to feed the algorithm data (synthetic or real) where you know at least approximately what the right answer is, even if you don't know exactly, and see by inspection if the answer is reasonable. For example, you may not know exactly what the estimates of your parameters are, but you may know which ones are supposed to be "big" and which ones are supposed to be "small".