An update on this 12 years after the question was asked.

Tests for nonlinear relationships

If the task is just to test for a relationship with a test that is able to discover any kind of (nonlinear) relationship, there are many (>10) tests for this by now. Karch et al. (2024) (I am one of the authors) provides an overview of many such tests. To mention a few prominent ones with corresponding R packages,

• Distance correlation (https://cran.r-project.org/web/packages/energy/index.html)
• Tau Star test (https://cran.r-project.org/src/contrib/Archive/TauStar/)
• Heller-Heller-Gorfine test (https://cran.r-project.org/src/contrib/Archive/HHG/)
• Hilbert-Schmidt Independence Criterion test (https://cran.r-project.org/web/packages/dHSIC/index.html)

All these tests are consistent if the two variables are not independent. Thus, if the two variables are related, their power approaches $1$ as the sample size increases. An important difference compared to nonlinear regression techniques, such as, generalized additive models, as suggested in another answer is thus that those methods are not limited to testing for an influence of $X$ on the mean of $Y$ but can also find all other dependencies such if $X$ influences the variance of $Y$.

This, of course, poses the question of which test to use for a given problem. For finite sample sizes, there are big power differences, and it mostly depends on the form of the relationship (quadratic, exponential,...) which test is best (Karch et al., 2024; de Siqueira Santos et al., 2014). de Siqueira Santos et al., 2014 provides some guidance when there is some knowledge about what relationship to expect. We (Karch et al., 2024) but also others (Simon et al., 2014) concluded that distance correlation tends to perform reasonably well across many relationships.

Test for inverted U

When such specific knowledge about the relationship as in the original question is available, I would not use any of these general tests but instead one tailored to the problem at hand, as it will likely have more power due to being specialized. A peer-reviewed test, for an inverted-U relationship, very much in the spirit of an earlier answer, is https://journals.sagepub.com/doi/full/10.1177/2515245918805755.

An update on this 12 years after the question was asked.

Tests for nonlinear relationships

If the task is just to test for a relationship with a test that is able to discover any kind of (nonlinear) relationship, there are many (>10) tests for this by now. Let's call them independence tests. Karch et al. (2024) (I am one of the authors) provides an overview of independence tests. To mention a few prominent ones with corresponding R packages,

• Distance correlation (https://cran.r-project.org/web/packages/energy/index.html)
• Tau Star test (https://cran.r-project.org/src/contrib/Archive/TauStar/)
• Heller-Heller-Gorfine test (https://cran.r-project.org/src/contrib/Archive/HHG/)
• Hilbert-Schmidt Independence Criterion test (https://cran.r-project.org/web/packages/dHSIC/index.html)

These independence tests are consistent if the two variables are not independent. Thus, if the two variables are related, their power approaches $1$ as the sample size increases. An important difference compared to nonlinear regression techniques, such as, generalized additive models, as suggested in another answer is thus that independence tests are not limited to testing for an influence of $X$ on the mean of $Y$ but can also find all other dependencies such if $X$ influences the variance of $Y$.

This, of course, poses the question of which independence test to use for a given problem. For finite sample sizes, there are big power differences, and it mostly depends on the form of the relationship (quadratic, exponential,...) which test is best (Karch et al., 2024; de Siqueira Santos et al., 2014). de Siqueira Santos et al., 2014 provides some guidance when there is some knowledge about what relationship to expect. (Karch et al., 2024) and (Simon et al., 2014) concluded that distance correlation tends to perform reasonably well across many relationships.

Test for inverted U

When such specific knowledge about the relationship as in the original question is available, I would not use any of these general tests but instead one tailored to the problem at hand, as it will likely have more power due to being specialized. A peer-reviewed test, for an inverted-U relationship, very much in the spirit of an earlier answer, is https://journals.sagepub.com/doi/full/10.1177/2515245918805755.