# How to perform KDE smoothing on a large dataset in python

I have two datasets plotted on two axis like this:

The datasets can be interpreted as a series of counts. The x and y arrays that form the blue line, for example look like

x = [0, 1, 2, ...]
y = [20788583, 4731125, 2534681 ...]


The interpretation is that there are 20.8 million counts at index 0; 4.7 million counts at index 1, etc.

This graph is messy, and I had the bright idea to use a gaussian KDE to smooth out this graph to better display my data. However, I'm struggling with implementing a kernel smoothing in python.

I am attempting to use scipy.stats.gaussian_kde() to smooth the data. But that function seems like it should take a univariate array where each instance of the index is entered separately. For example, my input array is to that function should look like

x_kde = np.concatenate([[i] * y[i] for i in range(len(y))])


Which will look like:

x_kde = [0, 0, 0, ...(20 million more)..., 0, 1, 1, ...(4.7 million)...]


However, as you may have noticed from the counts, this is a very long list, with a length over 100 million. In fact, that list is crushing the memory on my laptop.

The list I have is simply a compressed version of the list that I need to feed into gaussian_kde(), so surely there must be some way to use it as is.

How can I take a python array where the value represents the number of counts at that index and perform a KDE smoothing on it in python?

• The KDE is unlikely to be an appropriate method to smooth these data. Your dataset doesn't even appear to be that large: the plot looks like two "curves," each formed by fewer than 300 points. Why not just apply a standard robust smoother, like Loess? Is there some particular feature of these data that suggests you need a KDE? What would it be? – whuber Jan 27 '18 at 19:37
• @whuber I guess the answer to that is that when you get an MS in Statistics these days, they teach you about kernel density estimates, but not about LOESS smoothing. – kingledion Jan 27 '18 at 20:12

An appropriate method for treating data in this way is gaussian_filter1d from scipy.ndimage.filters.
from scipy.ndimage.filters import gaussian_filter1d

Here, s is the standard deviation for the Gaussian kernel.