I will list some of the models/methods so that you can more easily locate information on them (whether here, on Wikipedia and other such sources, via google, in the titles of books and articles and so on) on your own. This is not a complete list but covers a number of the more common approaches. I'll also mention some references and a few online resources at the end.
simple methods for fitting curved relationships within a regression framework - e.g. polynomial regression, though trigonometric and other models may be used.
Polynomial regression example (from here - pink is the fitted values at each $x$):
Trigonometric regression (link above):
transformation of $\mathbf{x}$ and $y$ variables may be used; one common example is the use of Box-Cox family of power transformations. Where $\mathbf{x}$ is transformed alone, this is essentially a form of the previous case. When both sides are transformed, it can deal with non-normality, heteroskedasticity and curved relationships, though balancing up all three at once can sometimes be tricky. It's often the case that making relationships more homoskedastic can improve the normality (though usually not completely). When taking the interpretation back to the original scale, care needs to be taken.
Here's an example of the result of transforming a relationship that's non-linear and strongly heteroskedastic to one where linear regression is more suitable (from here):
nonlinear regression - $y = g(\mathbf{x})+\epsilon$
($\mathbf{x}$ is a vector of regressors) used for a nonlinear functional relationship ($g$) with either an assumption of Gaussian errors ($\epsilon$) or simply a square-error loss function. Widely used in the physical sciences (physics, chemistry and so on), though may be found in a variety of other contexts.
An example (from here black is data, pink is fitted model, $\text{E}(y) = \beta_0 + \beta_1 e^{-\beta_2 x}$):
A second example (from here):
Generalized Linear models (GLM)
Used for both Gaussian and non-Gaussian $Y$ (within the exponential family) for functions where $g[E(Y|\mathbf{X=x})]$ is linear in $\mathbf{x}$ and the variance is a function of the mean. Very widely used.
Example of a fit of a binomial GLM to mortality data ($E(d_x)=E_xq_x$, where $q_x = \frac{Ab^x}{1+Ab^x}$):
There are more general models fitted via maximum likelihood estimation, method of moments (MoM or sometimes MME) or in other ways. In general some model is specified, some loss function is minimized, or some other fitting criterion specified (as in MoM), in order to produce estimates; a variety of optimization or root-finding methods are applied.
Smoothing techniques, including regression splines, smoothing splines (or penalized versions of either), kernel
smoothing/local linear regression/local polynomial regression (including LOESS and other local regression methods), and so on. Also under the umbrella of nonparametric regression, though that includes some other techniques not of direct relevance to your question.
See also Generalized Additive Models.
Example (weighted natural cubic spline fit to log-transformed data):
There's a plethora of information on almost all of these topics here on CV.
The book "(An R and S-PLUS) Companion to Applied Regression" (Fox or more recently "An R Companion to Applied Regression" Fox & Weisberg) has a number of relevant chapters (including coverage of transformaiton and GLMs, (and there are additional online chapters here, especially the first couple of chapters at that link which cover nonlinear regression and nonparametric regression) that you may find useful.
Also see Fox, Applied Regression, Linear Models, and Related Methods.
Some of the models/methods are discussed in Elements of Statistical Learning (Hastie, Tibshirani and Friedman). The 10th printing of the second edition is available online, but its not especially introductory (though parts are pretty straightforward) and its focus is different from what you're likely after. Nevertheless you may find some of its chapters of value.
