Behavior of $R^2$ in non-linear models I am a bit stumped on the behavior of $R^2$ in non-linear models.
Below is some data and two hyperbolic fits. One in which two parameters are estimated (Model $m_1$), and another in which one parameter is fixed at $100$, and only the other parameter is estimated (Model $m_2$). Model $m_1$ has much smaller residual standard error ($8.01$ on $2$ df) than $m_2$ ($13.7$ on $3$ df) and just by looking at it (see graph below), a better fit than $m_2$ (even though $m_1$ itself could be improved). The residual sums of squares as reported by anova() are $128.35$ for $m_1$ and $565.80$ for $m_2$. With one df difference that yields a p-value for the difference in SS residual of $.12$. 
The difference in residual sums of squares and the eyeball fit is not surprising. 
Yet, the $R^2$ of $m_2$ (the constrained model) is much better ($R^2$ is computed (probably incorrectly) by squaring the correlation of predicted values and observed values of $Y$). $m_1$ has a squared correlation of observed and predicted Y values of $.921$, whereas $m_2$ has $.995$.   
Is $R^2$ useful at all for these models? Can one distinguish these models based on $R^2$ (e.g., an exponential model could be a competing model and we may check $R^2$ of that model). 
I am curious about this, because in my corner of the literature these types of models are favored or discarded based on $R^2$ evidence and I also don't understand how a constraint can improve $R^2$ in this situation.
test <- data.frame(x=c(1,10,50,100),y=c(57.7,28.0,17.8,14.8))

m1 <- nls(y ~ a / (1+b*x),test,start=list(a=200,b=.07))
m2 <- nls(y ~ 100 / (1+b*x),test,start=list(b=.07))
coeffm1 <- coefficients(m1)
coeffm2 <- coefficients(m2)
summary(m1)
summary(m2)

anova(m1,m2)

test$m1pred <- fitted(m1)
test$m2pred <- fitted(m2)

cor(test$y,test$m1pred)^2
cor(test$y,test$m2pred)^2

plot(test$x,test$y,ylim=c(0,60))
curve((y=coeffm1["a"] / (1+coeffm1["b"]*x)),add=T)
curve((y=100 / (1+coeffm2["b"]*x)),add=T,lty="dashed")


Solid line is model m1, dashed line is model m2. Eyeball fit of m1 is better than m2.
 A: I decided to move my comment to an answer and discuss it
To expand on my points a little:
Your thought that the way you're calculating $R^2$ isn't sensible is right.
A high correlation between residuals and arbitrary fitted values doesn't automatically imply a good fit. Indeed, forget nonlinear regression, and consider linear models.
Imagine you have two linear models. For the first model, the Flying Spaghetti Monster comes to you in a dream, touches you with his noodly appendage and tells you the true parameter values:
$y_i = 9 + 2\,x_i + e_i$, (and that the errors are independent and identically distributed $N(0,\sigma^2)$)
The next day, as you're telling a friend over a pint about your experience, a passing homeopathy salesman suggests that instead
$y_i = -1000 + 7.3\times 10^{-14}\,x_i + e_i$
Imagine that on looking at the data, the dream-values appear to be about right (least squares estimates come out very close to them), and that $\sigma^2$ is estimated to be really tiny, so that the residuals from both the dream-values and the LS fit are minuscule.
Further, the correlation is almost 1.
Now, what's the correlation of the data with the fitted values from the salesman-in-the-pub's model?
It should be low, right? The model is completely wrong! Its predictions are no better than the mean.
... in fact, its correlation is exactly the same; almost 1.
If that was what $R^2$ was, it would be useless as a way of comparing models.
This sounds like it might provide a useful object lesson in exactly why they shouldn't be using that definition of $R^2$ to compare those models. 
However, since they have different numbers of parameters, an unadorned residual sum of squares isn't exactly a fair comparison either. 
Even a more correct $R^2$ would not necessarily be a good way of comparing nonlinear models; in the nonlinear realm there's often no good reason to consider a constant-mean model, the 'null' situation.
Indeed, even when comparing two linear models, $R^2$ is not necessarily the best way to do it, for example, for similar reasons that the (to my mind) marginally more sensible comparison of residual sum of squares I mentioned earlier should be avoided with models with different numbers of parameters.

I thought that the squared correlation between observed Y and fitted Y is in fact the standard $R^2$ in OLS. 

Well, yes, it's true that $R^2$ for a simple ordinary-least-squares model, is the square of the correlation between observed and predicted, but the ability to interpret it the way you want to interpret it is conditioned on it being the result of OLS. If you assert parameter values, for example, you don't change the correlation, but you lose its interpretation as a measure of fit at all.

Coding your example seems to confirm this and also shows that the homeopathy salesman has a lower R2. 

The computation in R isn't accurate for the salesman because of round-off error and accumulated numerical error; this is a mathematically exact relationship that we have to take care over doing numerically.
Observe what happens:
x <- runif(100,0,10) 
y <- 9 + 2*x + rnorm(100,0,.005) 
cor(y,x)
cor(y,9+2*x)
cor(y,-1000+ 7.3e-4*x)
cor(y,-1000+ 7.3e-7*x)
cor(y,-1000+ 7.3e-10*x)
cor(y,-1000+ 7.3e-14*x)

