# Good machine learning algo for partial derivatives?

Does anyone know of good robust algos to estimate partial derivatives of a regression model? I am talking about a general regression model like this:

$\mathbb{E}(y|x_1, x_2, ... x_n) = f(x_1, x_2, ... x_n)$, where I want estimates of $\frac{\partial f}{\partial x_k}$.

These are of great importance in medicine, economics, social sciences, etc. Now a linear model would give an approximation of these partial derivatives (its parameters) around the mean point.

This approximation can be made valid further around the mean by the inclusion of polynomial terms, but the exponential increase in numbers of parameters as the polynomial degree rises makes this infeasible with a linear model (except perhaps with Lasso?).

For better local estimates over the whole input space, natural candidates would be kernel regression and feedforward neural networks, which have smooth forms (unlike tree-based methods) and from which partial derivatives are easy to recover. But I don't know about the robustness of these estimates, as the main aim of these models is to reduce prediction error, not find the right derivatives...

I know gradient boosting (and really all regression models) give a "partial dependance", but they are not partial derivatives: they are a function $f_k(x_k)$ that gives the mean value of the predicted regression function $\widehat{f}$ for a given value of $x_k$, averaging over the realisations of the other variables $(x_1, x_2 ... x_n)$ in the training dataset.

Any help would be greatly appreciated!

• Have you checked out Gaussian process regression? You can learn partial derivatives directly (e.g. see gaussianprocess.org/gpml/chapters/RW9.pdf). – fabee Jul 25 '14 at 5:59
• Thanks, no I don't know anything about GP, I'm going to dig in! Do you know anything about the robustness of estimated derivatives though? – jubo Jul 25 '14 at 7:42
• So if I get this right, you would get random predictions of partial derivatives, which can then be used to compute local mean and variance? This is great! – jubo Jul 25 '14 at 8:23
• I came across this question and must say I am also very interested in this particular question. I was wondering whether you have already come across relevant research or are doing some yourself? I would be very interested in applying it to my own Bioinformatics research, especially when it has some theoretical justification making it suitable for publication (an issue with neural nets I suppose) – MJW Aug 17 '14 at 17:01
• Hi, I haven't found any solid literature yet (but I haven't looked very hard either). So far potential candidates would seem to be: regularized linear regression (LASSO or ridge, with large basis expansions), kernel regression, neural networks (with some flavors potentially better than other, eg. I suspect extreme learning machines, dropout nets and deep learning should get better derivatives than standard backprop) and Gaussian processes. – jubo Aug 17 '14 at 17:49

I am not sure if this is what you are searching for, but I'll give it a try.

I think you can use Gaussian process regression and also get a nice robustness measure.

Let's say you have $m$ observations $x_1, ..., x_m\in \mathbb R^n$ and a target variable $y_i \in \mathbb R$ for each $x_i$. For a Gaussian process with covariance function $k$, the joint density of a function value $f$ at an new point $x^*$ (the idea is that $y=f+\varepsilon$ where $\varepsilon$ is Gaussian noise) is given by

$$\left(\begin{array}{c} \mathbf{f}\\ f^{*} \end{array}\right)=\mathcal{N}\left(\mathbf{0},\left(\begin{array}{cc} K+\sigma^{2}I & \mathbf{k}^{*}\\ \mathbf{k}^{*} & k\left(x^{*},x^{*}\right) \end{array}\right)\right)$$ where $\mathcal N$ denotes the Gaussian distribution, $\mathbf K$ is the matrix of pairwise covariance function values between the $x_i$ and $\mathbf k^*$ is the vector of covariance function values between $x^*$ and all $x_i$ from the training set.

From that it is easy to compute the posterior mean (i.e. the prediction at $x^*$) and covariance for $f^*$ (i.e. the measure of certainty which I think you mean with robustness). For all the formulae see the book by Rasmussen and Williams.

The interesting fact is that if $k(x_i,x_j)=\mbox{cov}(f_i,f_j)$, then $$\frac{\partial}{\partial x_{i\ell}}k(x_i,x_j)=\mbox{cov}\left(\frac{\partial}{\partial x_{i\ell}} f_i,f_j\right)$$ and $$\frac{\partial^2}{\partial x_{i\ell}\partial x_{jt}}k(x_i,x_j)=\mbox{cov}\left(\frac{\partial}{\partial x_{i\ell}} f_i,\frac{\partial}{\partial x_{jt}}f_j\right)$$ (again, check chapter 9.4 of Rasmussen and Williams). This means, you can just treat the partial derivatives as "unobserved function value $f^*$" and adapt the covariance functions in the above equation to the respective partial derivatives of the original covariance function. Then you simply predict the derivative like you would predict $f^*$. As a measure of certainty, you can look at the posterior covariance which is easily computes as well.

There is a nice Gaussian process toolbox for matlab written by Hannes Nickisch. It can probably help you play around with GPs.

• You're welcome. I hope you can use it. – fabee Jul 25 '14 at 11:22