I have two random variables $X$ and $Y$ and I'm trying to model $\mathbb{E}[Y|X]$.

To this end, I'd like to pick a collection of functions $f_1, f_2 \dots f_n : \mathbb{R} \to \mathbb{R}$ and then fit a model to my data set $(x_i, y_i)_{1 \dots m}$ by OLS:

$$\mathrm{argmin}_{\alpha_1 \dots \alpha_n} \sum_{i=1}^m \left(y_i - \sum_{j=1}^n \alpha_j f_j(x_i) \right)^2$$

I'm looking for a reference that compares different choices of families of $\{ f_i\}$ and answers questions like:

  • Will any linearly independent $f_i$ work?
  • Is an orthogonal collection (Laguerre polynomials, Fourier basis) better and mathematically, why?

I am particularly keen to learn about the numerical stability of the resulting least squares problem. Do some bases $\{ f_i \}$ work fair noticeably worse if the least squares problem is solved by Cholesky or QR factorisation instead of SVD?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.