If I fit the model mgcv::bam(y~s(x), rho=0.8), where y and x are ordered by time, my understanding is that the model can be described in math notation as:
$ y_t = \beta0+f(x_t)+\epsilon_t\\ \epsilon_t = rho * \epsilon_{t-1} + w_t\\ w_t \sim normal(0, \sigma) $
Is this correct? Also is the maximum likelihood estimate of the model equivalent to minimizing the square of $w$?
I tend to think of the model mgcv::bam(y ~ s(x), rho = 0.8) as being equivalent to
$$ y_t = \beta_0 + f(x_t) + \varepsilon_t $$
where $\boldsymbol{\varepsilon} \sim \mathcal{N}(0, \hat{\sigma}^2\boldsymbol{\Lambda})$ and $\boldsymbol{\Lambda})$ is the first order correlation matrix
$$ \boldsymbol{\Lambda} = \begin{pmatrix} 1 & \rho & \rho^2 & \cdots & \rho^{n-1} \\ \rho & 1 & \rho & \cdots & \rho^{n-2} \\ \rho^2 & \rho & 1 & \cdots & \rho^{n-3} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ \rho^{n-1} & \rho^{n-2} & \rho^{n-3} & \cdots & 1 \\ \end{pmatrix} $$
with $\rho = 0.8$.
But this arises from the assumed first order AR process
$$\varepsilon_t = \rho \varepsilon_{t-1} + \eta_t$$
with $\eta_t \sim \mathcal{N}(0, \hat{\sigma}^2)$. (Which is what you wrote but with the notation from my own lecture notes/slides so as to not confuse myself.)
This is essentially a generalised least squares problem, which I understand will seek to minimise the squared Mahalanobis length of the residual vector $\eta_t$ (your $w_t$).
