I want to let the gradient be a constant, say $3$ and then regress on the offset. Its obvious that one can do GD (or SGD) on something like the L2 loss of this. But this seems such an easy problem that I thought there should be something simpler that uses linear algebra. Is there?
Obviously:
$$ \theta = X^{-1}y$$
doesn't work since $\theta = [3 , c]$ since that doesn't respect the gradient being some constant value.
If I understand your question correctly, you want to fit a function of the form:
$$y = w \cdot x + c$$
where $w$ takes a specified value, and $c$ is the the only free parameter.
Suppose the training set is $\{(x_i, y_i)\}_{i=1}^n$ and we want to minimize the squared error. Then $c$ is the mean of $\{y_i - w \cdot x_i\}$, which follows from the fact that the mean of a set is the point that minimizes the squared distance to each element.
Just to write things out, the problem is:
$$\min_c \sum_{i=1}^n [y_i - (w \cdot x_i + c)]^2$$
Take the derivative of the loss function and set it to zero:
$$-2 \sum_{i=1}^n [y_i - (w \cdot x_i + c)] = 0$$
Solve for $c$:
$$c = \frac{1}{n} \sum_{i=1}^n (y_i - w \cdot x_i)$$
