Nontrivial nonlinear regression, which transformation to use? I have some data which I think can be fitted using the following function:
$k(x,y)=\beta \cdot e^{-\gamma y} \cdot \mathrm{ln} \left( \frac{x - a}{b} \right) \wedge \beta, \gamma \in \mathbb{R}_0^+$
Where $a$ and $b$ are known values, and $\beta$ and $\gamma$ are the parameters I am trying to find using least squares and I'm pretty sure they're not related to one another. Not looking ahead, I first started calculating the separate partial derivatives and soon saw that the expression for $\gamma$, because it is inside an exponent, was not very easy to find by hand. So I started over by looking at the logarithmic error which made $\gamma$ a linear term.
When I worked everything out, I first found an expression for $\beta$, from which I had hoped to figure out $\gamma$:

EDIT: The last bit has to be $\frac{1}{n}$ times the big summation with the two logs.
But two problems arise already: (1) all of my data for $f_i$ is negative and (2) all of my data for $\mathrm{ln} \left( \frac{x_i - a}{b} \right)$ will also turn out to be negative due to the nature of the original experiment. So even if I could use this expression for $\beta$ to find a potential expression for $\gamma$, I wouldn't be able to calculate either of them using my data.
I read up on log fitting a bit and I noticed that a lot of people just add constants to the negative values to make their sample data positive like so:
$\mathrm{ln}\left( \beta + \lambda_0 \right)$
$\mathrm{ln} \left( \mathrm{ln} \left( \frac{x_i-a}{b} \right) + \lambda_1 \right)$
But I get different results for $\beta$ and $\gamma$ when I use different values of $\lambda_i$, which adds another level of uncertainty.
I do not have a formal education in statistics or mathematics, so I can't really think of any other methods to proceed from here.
 A: Why not try nonlinear least squares? Denote the left-hand side, $f(x,y)$ as $z$. 
$$(\hat{\gamma}, \hat{\beta})  = \text{arg} \min_{(\gamma,\beta)} \sum_{i=1}^N(z_i - \beta e^{-\gamma x_i} \ln (\frac{y_i-a}{b})^\beta)^2.$$
If you cannot work out the solution analytically, which would involve solving 
$$ \frac{\partial \sum_{i=1}^N(z_i - \beta e^{-\gamma x_i} \ln (\frac{y_i-a}{b})^\beta)^2}{\partial \gamma} = 0$$
$$ \frac{\partial \sum_{i=1}^N(z_i - \beta e^{-\gamma x_i} \ln (\frac{y_i-a}{b})^\beta)^2}{\partial \beta} = 0$$
for $\gamma$ and $\beta$, you can estimate these values using a grid search. 
A: I anticipate questions, so please do ask away in the comments.
Let's start out by discussing what ordinary least squares does.
We have some data and want to fit a line. Let's say that we have $x$-values $1$, $2$, and $3$, and $y$-values $4$, $5$, and $7$.
These are points in the plane: $(1,4)$, $(2,5)$, and $(3,7)$.
What least squares says is that the line of best fit is the line that minimizes square loss. "Least" comes from the minimization; "squares" comes from squaring something.
Square loss is just the squares of the true values minus the predictions.
So what we want to minimize in a least squares problem isn't the model $\hat{y}=ax+b$ but this nasty equation, where $y$ is the true $y$-value, $\hat{y}$ is the predicted $y$-value, and $n$ is the number of observations of $y$:
$$L = \overset{n}{\underset{i=1}{\sum}}\big(
y_i - \hat{y}_i\big)^2$$
$y_i - \hat{y}_i$ is by how much the prediction misses the true value; this is the residual. Then we square the residual. Then we add up all of our squared residuals to get a measure of by how much our line misses the data. The lower this number, the tighter the fit.
Since $\hat{y}= ax+b$, we can write the loss function like this:
$$L = \overset{n}{\underset{i=1}{\sum}}\big(
y_i - (ax_i+b)\big)^2$$
For that example with the three points, that is this:
$$L = \big(4-(1a - b)\big)^2 + \big(5-(2a - b)\big)^2 + \big(7-(3a - b)\big)^2
$$
We then find the values of $a$ and $b$ that minimize $L$ and give us the least squares solution. For this type of model, calculus shows that $a$ and $b$ have a convenient formula that gets us out of having to do partial derivatives every time; we just plug the data into the formula and get our values of $a$ and $b$.
You, however, propose a more complex model. There's nothing wrong with that.$^{\dagger}$ You just lose the convenient formula to give $a$ and $b$, but if your data should not be modeled with a line, then that is just the price to pay for getting a model worth using.
You propose that your response variable, $z=f(x,y)$ is described by $f(x,y)=\beta \cdot e^{-\gamma y} \cdot \mathrm{ln} \left( \frac{x - a}{b} \right)$. Stick that into the loss function as your prediction!
$$L = \overset{n}{\underset{i=1}{\sum}}\big(z_i - \hat{z}_i)^2 =\overset{n}{\underset{i=1}{\sum}}\bigg[z_i - \beta e^{-\gamma x_i} \ln \bigg(\dfrac{y_i-a}{b}\bigg)^\beta\bigg]^2$$
I don't want to calculate the partial derivatives by hand, but it can be done. You could use some software like WolframAlpha to help you. The typical way to do this would be to use a computer to find the values of $\beta$ and $\gamma$ that minimize $L$. A grid search is one option. Instead of calculating the derivatives, the grid search picks out points in the $\beta\gamma$-plane and checks which one results in the smallest value of $L$. That point gives you your $\alpha$ and $\beta$ values.
So let's go through the questions in my comment to your original post.
What are you minimizing?
You're minimizing $L$, which is the square loss. Importantly, you do not minimize the model.
Why is this minimization not fundamentally different from ordinary least squares?
Instead of subtracting a prediction found from a linear model like in ordinary least squares, we subtract a prediction found by a different kind of model. However, all of the ideas about using calculus to find the point giving minimal loss are the same.
How is the grid search helping to do the optimization calculus?
The grid search proposes a grid of values as the minimizers of the loss function, then checks what loss results from each. Whichever point in the grid gives the minimal loss is declared the winner.
Note There is a technical point that I have skipped: the $\hat{\text{hat}}$ on variables in dlnB's post. That means an estimate that you have calculated from your data, instead of the true value. In real data analysis, you don't get to know the true value, but if you have good estimates, you can be confident that you're close.
$^{\dagger}$There are cautions throughout statistics about overly complex models. The gist is not to make a model complex just for the sake of being complex. I suspect that you have a scientific reason for wanting the model to take the form that you propose, so please do feel free to use your model.
