# "Smoothness" of a statistic for bootstrapping?

I was wondering if anyone could explain what is meant by saying a statistic is not 'smooth'.

For example, in 2.6.2 p. 41 of Davison and Hinkley, they talk about statistics that "depend on the sample in an unsmooth or unstable way such that standard expansion theory does not apply".

It also mentions a function being a differentiable function of sample moments but I'm uncertain if this is what is meant by 'smooth' or not.

If so, can you explain what is meant by that phrase?

$\newcommand{\OLS}{\operatorname{OLS}}$This is essentially a question about mathematical, not statistical terminology, as far as I can tell.

Anyway the point is that the statistics are not a differentiable function of the sample, or are not $n-$times continuously differentiable function of the sample.

In other words there are possibly places where the response of the statistic to changes in the sample is not ideal, or is unappealingly abrupt (hence the terminology 'smooth'), in a way that linear or polynomial functions of the data, for example, could never have.

The Wikipedia page about smooth functions is probably unnecessarily technical at points, but hopefully some of the pictures and extended discussion can give you some intuition for what is meant to be evoked by the term 'smoothness'.

If a certain function is a "differentiable function of sample moments" then it may be a smooth function of the sample moments, depending on what sense "smooth" is being used in that context. I most often see "smooth" used to mean infinitely many times continuously differentiable (e.g. like polynomials or linear functions or sines and cosines), but sometimes the term can be used in a less strict sense, as the Wikipedia page mentions.

In any case you are definitely right that it relates to differentiability -- that is the key idea.

Also it's worth noting that there exist functions which are continuous but not "smooth" -- the idea is that while continuity is in general a nice regularity property, in many cases it still allows a lot of undesirable pathological behavior, while such pathological behavior cannot occur for smooth functions, because they are even nicer still than continuous ones.

Example: Consider, for example, the LASSO estimator with orthonormal covariates:

$$\hat{\beta}_j = S_{N \lambda}(\hat{\beta}_j^{\OLS}) = \hat{\beta}_j^{\OLS} \max\left\{ 0, 1 - \frac{N \lambda}{\left|\hat{\beta}^{\OLS}_j \right|} \right\},$$ where $\hat{\beta}^{OLS} = (X^T X)^{-1}X^Ty = X^T y$.

First we note that $\hat{\beta}_j^{\OLS}$ is linear in the coordinates of $X$ and $y$ since $\hat{\beta}^{\OLS}$ is linear in $X$ and $y$, so (assuming that $X$ or $y$ represents the sample) all of the $\hat{\beta}_j^{\OLS}$ are completely smooth functions and are not the source of the non-smoothness. Instead, any non-smoothness comes from the maximum function $\max$ found in the definition of $\hat{\beta}_j$, as I will try to convince you below.

We use the identity $\max\{x, y \} = \frac{x+y +|x-y|}{2}$ (discussed and proven here) to rewrite the above expression as follows: $$\begin{array}{rcl} \hat{\beta}_j & = & \displaystyle\frac{\hat{\beta}_j^{\OLS}}{2}\left[ -\left( \frac{N \lambda}{\left|\hat{\beta}_j^{\OLS}\right|} - 1 \right) + \left|\frac{N \lambda}{\left|\hat{\beta}_j^{\OLS}\right|}-1\right| \enspace \right] \\ & = & \begin{cases} 0, & \text{when } \displaystyle\frac{N \lambda}{\left|\hat{\beta}^{\OLS}\right|} \ge 1 \\ \hat{\beta}_j^{\OLS}\left(1 - \displaystyle\frac{N \lambda}{\left|\hat{\beta}_j^{\OLS}\right|} \right), & \text{when } \displaystyle\frac{N \lambda}{\left|\hat{\beta}^{\OLS}\right|} \le 1 \end{cases} \end{array}$$

Written in this form, it is obvious that we have at least two possible sources for non-smooth behavior: (1) when $\hat{\beta}_j^{\OLS}=0$, causing a denominator to vanish, (2) and possible cusps at the point(s) where: $$\frac{N \lambda}{\left| \hat{\beta}^{\OLS}_j \right|} = 1 \iff N\lambda = \left| \hat{\beta}^{\OLS}_j \right|,$$ since of course at these points $\hat{\beta}_j$ is the "gluing together" of two different functions $\left(0\text{ and }\hat{\beta}_j^{\OLS}\left(1 - \frac{N \lambda}{\left|\hat{\beta}_j^{\OLS}\right|} \right) \right)$ which, even though they have the same value at the points where $N\lambda = \left| \hat{\beta}^{\OLS}_j \right|$, may not necessarily "play nicely" together in such a way that the left- and right-hand derivatives agree for all $n$. The most basic example of a function for which this fails to happen is $|x|$ at the value $x=0$: it's first left-hand derivative is $-1$ and it's first right-hand derivative is $1$, so it is not smooth at $x=0$. I suspect that an analogous phenomenon likely happens for the function $\hat{\beta}_j$ at those points where $N \lambda = \left| \hat{\beta}^{\OLS}_j \right|$, causing $\hat{\beta}_j$ to not be a smooth function of its inputs.

The function $\hat{\beta}_j$ only needs to be smooth with respect to its input arguments in order to be considered smooth. Presumably its input arguments are the sample itself, or some functions $g$ of the sample. If $\hat{\beta}_j$ is a function of functions $g$ of the sample, then one can by composition $\hat{\beta}_j \circ g$ get a new function $\tilde{\hat{\beta}}_j$ that skips the middleman (i.e. returns the same outputs of interest and is directly a function of the sample). By the chain rule this composed function $\tilde{\hat{\beta}}_j = \hat{\beta}_j \circ g$ is smooth if and only if both functions $\hat{\beta}_j$ and $g$ are smooth.

• Thanks so much for your answer! Any chance someone would give an example of how you would check this for some simple statistics? I found the part about how the statistic 'changes in the sample' quite confusing. I am unsure how a statistic would ever change within a sample?? I'm not sure I'm clear what function would be being differentiated and with respect to what variable??
– Bee
Nov 18, 2016 at 15:35
• @Bee I won't promise anything, but if you type up the passage you're referring to (in particular the definition of the statistic in question) I can try to take a look at it. One common reason for functions not being smooth are isolated singularities, for examples when the function is a fraction and the denominator is zero at a point. Is that the case with you statistic? en.wikipedia.org/wiki/Singularity_(mathematics) Nov 18, 2016 at 15:44
• So for example the LASSO estimator with orthonormal covariates given at wikipedia clearly the max() part of this function could introduce some smoothness issues but what is the variable it needs to be 'smooth' with respect to... thinking of Bols as changing is a strange concept
– Bee
Nov 18, 2016 at 16:28
• just b/c I don't know how to write beta OLS as in the example and make it look proper
– Bee
Nov 18, 2016 at 17:23
• @William +1 Nice answer. A conclusion is somewhat missing. I suppose something along the lines of 'Hence the bootstrap applied to estimate variance of $\hat{\beta}_j$ is invalid because the statistic is a non-smooth function in $X$ and $y$'. Dec 7, 2016 at 13:35