Motivation: I am trying to use Akaike Information Criterion to assess model ranking and over-fitting risk for a set of nonlinear models.
As I understand it, I must compute the maximum likelihood estimator for each model.
If you want an AIC, yes, you would need MLE. But the AIC is not automatically ideal. [One thing you should be careful of is if you're selecting a model on the same data you're using to do inference, you'll have a variety of problems.]
I could assume the residuals are Gaussian* and then the MLE is a least squares one. I am not convinced this is adequate; if my models are nonlinear, does that imply a more complex probability density?
I don't think so. If your model is of the form $y=f(X,\beta)+\varepsilon$, I don't see why nonlinear $f$ would imply anything about $\varepsilon$.
Your actual distribution about $E(Y)$ is almost certainly more complex than a Gaussian/ (When are data exactly Gaussian? I'd have thought almost never.)
And, if my residuals aren't Gaussian and I don't have a good assumption for a probability density, how would I choose one?
If you don't have a good basis for one, choosing a model by looking at the same data you use to fit the model would bias your AICs anyway. [You might be able to get around that by sample-splitting, or cross-validation for example.]
I can also check this by observing them, no?
Your residuals are proxies for your errors, yes, so for example, skewed residuals might indicate non-normality.
If I check these before each analysis for each model, what would I do if they aren't normal?
You might consider a class of models such as generalized nonlinear models (like GLMs but nonlinear in the parameters) and still use AIC.
You might stick with least-squares even though it's not ML, and treat AIC simply as a (monotonic transformation of) a penalized-MSE.
Depending on what you need your models to achieve, you might consider modifying the current model to one with a slightly heavier tail, such as a contaminated-normal model, perhaps still with ML.
You might consider using some different criterion to do both model selection and model fitting.
But in any case your actual errors won't be exactly normal; the question is the degree to which that will impact your inference. (The AIC, for example, may perform reasonably well at trading off fit for model complexity whether or not the fitted model is exactly right.)
though this is just for one set of data, I do not necessarily know that the errors will always be normal for each model against any possible data set.
Note that the marginal distribution of residuals will only approximate the error distribution if the other (more important) assumptions hold up; you should check that the model for the mean and the variance is reasonable first before trying to worry much about normality.