Nonlinear approximation of a sparse linear transformation

Question

Suppose that we have $y = Ax$ where $x$ is a vector of size $m \times 1$, $A$ is a sparse matrix with size $n \times m$. Suppose that $n \times m$ is very large and we wish to find a non-linear approximation of $y$. In other words, find a set of parameters $\theta \in \mathbb{R}^d$ that

$\min_\theta L(Ax, f_\theta (x))$

where $L$ is an objective such as L2 norm $||Ax - f_\theta(x)||^2_2$.

I think this is a known problem in optimization but I couldn't find a reference on common/prominent approaches for solving this problem. In particular, I'm looking for choosing good non-linear model $f_\theta$ and the number of parameters $d$.

Answer 1

You can use the non-linear steepest descent algorithm. To calculate the step size, take the partial derivative with respect to the parameters and evaluate at the current point (Jacobian).

https://en.wikipedia.org/wiki/Gradient_descent#Solution_of_a_non-linear_system