1
$\begingroup$

This question sparks from model interpretation/visualization. To graph the dependency of a function with >2 arguments, one often needs to ignore or average out some arguments.

Problem set

Hastie, Tibshirani & Friedman (Elements of Statistical Learning, 2001) p.370 define a partial dependence of a mapping $f(X)=f(X_S, X_C)$ to a subset of its arguments $X_S$ by averaging the $X_C$ out:

$$\bar{f}_S (X_S)= \mathrm{E}_{X_C} \, f(X_S, X_C)$$

and state explicitly that this function does not describe the effect of $X_S$ on f ignoring the effect of $X_C$ on $f$, but $\tilde{f}_S$ does: $$ \tilde{f}_S (X_S) = \mathrm{E}( f(X_S, X_C) | X_S) $$

Questions

Am I correct in my understanding:

  • The first equation starts with an "existing" function $f(X_S, X_C)$ and then removes the dependency on $X_C$ by averaging over it.
  • The second equation describes the concept of creating an approximating $\tilde{f}_S$, such that for each "value" of $X_S$ the approximation $\tilde{f}_S$ is close to $f(X_S, X_C)$.

What is the mathematical difference between the two equations? The first expectation integrates only over the $X_C$ dimensions, whereas the second one integrates over all dimensions, right?

Is it possible to get an analytical sense of the difference of both?

Is it possible to calculate the second equation in practice for some dataset?

$\endgroup$
1
$\begingroup$

Analytically is a great question. Here's my go at the problem, maybe between the two of us and others we can solve it correctly.

Let's think of two types of yhats. Let's call them: \begin{align} y_{hat}= f(Xs=x_s,Xc=x_c) \end{align}

And,

\begin{align} y*= f(Xs,Xc=x_c) \end{align}

A fundamental difference is, one uses the training set exactly (yhat), while y* manipulates the training set. But analytically, they should be different. Here's my first stab at the problem.

Partial dependence plots use:

\begin{align} E_{Xc}(Xs=x_s,Xc=x_c) \end{align}

\begin{align} f(Xs_j)_{partial dependence} = \sum_{i=1}^n f(Xs=x_{sj},Xc=x_{ci})/N \end{align}

This is done for each Xs=xs. Note, it's a very real (and probably common) possibility that values conceived were never shown in the training set with the partial dependence plots. Hence you manipulate the training set in some way.

Whereas the filtering logic is restricted to using values in the training set. For example,

\begin{align} E(f(Xs=x_s,Xc=x_c)|X_s=x_s) \end{align}

\begin{align} f(Xs_j)_{conditionout}=\sum_{j=1}^ny_{hat}/N_j\ ,where \ X_s=x_{sj} \end{align}

This may not be correct - yet.

$\endgroup$
  • $\begingroup$ Why would $y*$ manipulate the training set? In my understanding, the training set is fixed and is not changed by a model or anything else. Maybe you just meant with "manipulate" what you wrote in the lower paragraph: that the averaging over x_ci yields the average of N predictions from points in the training set. And it is the general case that the average prediction is not in the training set. PS: I guess in the line above partialdependence there is a "f" missing? $\endgroup$ – Lukas Jan 1 at 12:12
  • $\begingroup$ Can $E(f(Xs=xs,Xc=xc) | Xs=xs)$ be seen as a mapping of one specific $xs$ to all the points y_i in the training set with this specific value of $xs$ but different $xc$ (and thus mapping to discrete values y_i, so it is some kind of discrete probability distribution? But after we do the averaging in the last eqn, we would get the same as with partialdependence above? $\endgroup$ – Lukas Jan 1 at 13:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.