About the convenient plot and the Taylor approximation
The plots seem a little too convenient, where a step size of exactly $\Delta \epsilon=1$, from 1 to 0, magically gets you from $T(P_1)=T(\tilde{P})$ to $T(P_0)=T(P)$ even though you don't know in advance what the $T(P_{\epsilon})$ path looks like.
$$T(P_0)=T(P_1)+\left(\frac{\partial}{\partial \epsilon}T(P_{\epsilon})\Big{|}_{\epsilon =1}\right)\times (0-1)-R_2$$
This remainder term $R_2 = -\frac{1}{2} \frac{\partial^2}{\partial \epsilon^2}T(P_{\epsilon})\Big{|}_{\epsilon = \bar{\epsilon}}$ is a consequence from Taylor's theorem.
It is not the second order term of the Taylor expansion, instead it is the remainder term. Note that for this term $\epsilon \neq 1$, but $\epsilon = \bar{\epsilon}$.
The value $\bar{\epsilon}$ is undetermined but it must be some value between the boundaries (0 and 1).
You can see it alternatively as following: the remainder term $R_2$ of the taylor approximation is limited by the lower and upper limits of the second derivative $T^{\prime\prime}(P_\epsilon)$
$$ \frac{1}{2} \min_{0\leq\epsilon\leq1}T^{\prime\prime}(P_\epsilon) \leq R_2 \leq \frac{1}{2} \max_{0\leq\epsilon\leq1} T^{\prime\prime}(P_\epsilon) $$
and so there is some value $\bar\epsilon$ for the real value $T^{\prime\prime}(P_\bar{\epsilon})$ which is somewhere in between.
$$ \frac{1}{2} \min_{0\leq\epsilon\leq1}T^{\prime\prime}(P_\epsilon) \leq \frac{1}{2} T^{\prime\prime}(P_\hat\epsilon) \leq \frac{1}{2} \max_{0\leq\epsilon\leq1} T^{\prime\prime}(P_\epsilon) $$
The exact path, the thick line remains unknown. We do not magically get it.
But we can know that the remainder term $R_2$, the difference between our linear estimate (the broken thin line thin) and the exact path, is relatively small (an error term that doesn't grow faster than the second derivative, which is for most smooth functions not so large).
About an intuitive view of the robustness due to correction with influence curves
Hoping to see a clear, intuitive explanation of how efficient influence curves are applied to initial probability distribution estimates (perhaps using nonparametric machine learning models) to arrive at an unbiased estimate of a target function.
It is the first time that I read about 1-step estimators, to my intuition it seems like some form of scoring algorithm where the score and fisher information are based the influence functions and an empirical distribution (how a change in the observation changes the parameter estimate can be inverted to how a change in the parameter changes the probability of observations, and related the likelihood function).
Example
The practical example and computation below may provide some intuition:
In this example the target is to estimate for the population distribution function $f(x)$ the integrated squared density:
$$T(f(x)) = E[f(x)] = \int_{-\infty}^{\infty} f(x)^2 dx$$
In the appendix C of the article from Fisher and Kennedy it is stated that in this case the influence function is
$$IF(x,f) = 2(f(x)-T(f))$$
In the code below we first estimate the distribution with a normal distribution in which case the initial estimate is $T(\tilde{f}(x)) = 1/\sqrt{4 \pi \hat{\sigma^2}}$. See in the image below that this estimate with a normal distribution is not a good one if the data is not normal distributed (in this case we generate the data according to a geometric distribution).
So we use the influence functions to correct the biased normal distribution estimate and shift that distribution with a first order approximation to the empirical distribution (a sum of delta functions).
The effect is a reduction of the bias from using the normal distribution as an approximation for the distribution. We get an estimate that is more robust than our estimate with a (potentially biased) parameterized distribution.
In this case the plugin solution $\sum \hat{f}(x)^2$ is actually doing pretty well, and even better than the 1-step estimator. This is because the computation is done with a sample size of $n=100$ in which case the mass distribution $f(x)$ can be well estimated. But for a small sample there will be only a small amount of cases in each bin and we would have $\sum \hat{f}(x)^2 \approx \sum (1/n)^2 = 1/n$ and that's when the approximation with a parametric distribution (and the 1-step estimator for robustness) are useful.
set.seed(1)
trueval <- sum(dgeom(0:300,0.1)^2)
onestep <- function(n = 20, plotting = FALSE) {
x <- rgeom(n,0.1)
# estimating distribution with normal distribution
# using method of moments
mu <- mean(x)
var <- var(x)*n/(n-1)
Test <- 1/sqrt(4*pi*var)
# computing influence functions
#
# for T(f(x)) = integral of f(x)^2 dx
#
IF <- 2*(dnorm(x,mu,var^0.5)-Test)
# making corrections
T1step <- Test + sum(IF)/n
#plotting
if (plotting) {
h<-hist(x, breaks = c(0:200)-0.5, xlim = c(-10,40), freq = FALSE,
main = "geometric data and normal estimate")
xs <- seq(-10,40,0.01)
lines(xs, dnorm(xs,mu,var^0.5))
} else {
h<-hist(x, breaks = c(0:200)-0.5, xlim = c(-10,40), freq = FALSE,
main = "geometric data and normal estimate", plot = FALSE)
}
plugin <- sum(h$density^2)
# return
c(Test,T1step,plugin)
}
onestep(n=1000, plotting=TRUE)
trueval
trials <- replicate(10^3,onestep(100))
sum((trials[1,]-trueval)^2)
sum((trials[2,]-trueval)^2)
sum((trials[3,]-trueval)^2)
h1 <- hist(trials[1,], breaks = seq(0,0.40,0.0025), plot = FALSE)
h2 <- hist(trials[2,], breaks = seq(0,0.40,0.0025), plot= FALSE)
h3 <- hist(trials[3,], breaks = seq(0,0.40,0.0025), plot = FALSE)
plot(h1$mids,h1$density, type = "l", col = "gray", xlim = c(0,0.1),ylim=c(0,100),
xlab = "estimated T", ylab = "density")
lines(h2$mids, h2$density )
lines(h3$mids, h3$density, lty = 3)
lines(rep(trueval,2),c(0,100), lty = 2)
text(trueval,85,"true value", pos=4, srt=-90, cex = 0.7)
legend(0.065,100,c("estimate with normal dist", "1-step improvement", "plugin estimate"),
cex = 0.7, col = c(8,1,1), lty = c(1,1,3))
title("comparing sample distribution of estimates")
In simple words
You could see the 1-step estimator as a sort of blend between two estimators of the population distribution: a parametric estimate $\tilde{f}(x)$, and an empirical estimate $\hat{f}(x)$ (the empirical estimate being a mass distribution with weight 1/n for each data point).
$$\epsilon \tilde{f}(x) + (1-\epsilon) \hat{f}(x)$$
When $\epsilon =1$ you have the parametric estimate which may be biased, but the empirical estimate (when $\epsilon=0$) may be too sparse to correctly describe the true distribution function. Then the 1st order approximation, by using a Taylor approximation, is being used to blend the two together.
The influence functions describe the derivative of the functional as function of $\epsilon$.
