This answer was thought-provoking and I think my preferred way of viewing it. In 1-D it's hard to understand why squaring the difference is seen as better. But in multiple dimensions (or even just 2) one can easily see that Euclidean distance (squaring) is preferable to Manhattan distance (sum of absolute value of differences).

@IndieAI Yes! That was the issue.

Hmm...now I'm wondering about the image labels. I have two classes and I labeled them as "1" and "2". But I just read that they should be 0-indexed. Could this be the root of the issue?

Definitely not a silly question. In the training file I specified to use the mean from the imagenet photos. I figured that should be pretty close since it's a fairly similar type of data set. Maybe that's not good enough?

Add iJulia kernel for the iPython/Jupyter notebook ecosystem.

Now I understand your answer better. I needed to visualize the sequences and come to that realization on my own though to really let it sink in. :)