R2 compares the fit of the chosen model with that of a horizontal straight line (the null hypothesis). If the chosen model fits worse than a horizontal line, then R2 is negative. Note that R2 is not always the square of anything, so it can have a negative value without violating any rules of math. R2 is negative only when the chosen model does not follow the trend of the data, so fits worse than a horizontal line.
Here is an example. We fit data to a linear regression model constrained so that the Y intercept must equal 1500.
The model makes no sense at all given these data. It is clearly the wrong model,perhaps chosen by accident.
The fit of the model (a straight line constrained to go through the point 0,1500) is worse than the fit of a horizontal line. Thus the sum-of-squares from the model (SSreg) is larger than the sum-of-squares from the horizontal line (SStot). R2 is computed as 1 - SSreg/SStot. When SSreg is greater than SStot, that equation computes a negative value for R2.
With linear regression with no constraints, R2 must be positive (or zero) and equals the square of the correlation coefficient, r. A negative R2 is only possible with linear regression when either the intercept or the slope are constrained so that the "best-fit" line (given the constraint) fits worse than a horizontal line. With nonlinear regression, the R2 can be negative whenever the best-fit model (given the chosen equation, and its constraints, if any) fits the data worse than a horizontal line.
Bottom line: A negative R2 is not a mathematical impossibility or the sign of a computer bug. It simply means that the chosen model (with its constraints) fits the data really poorly.