Best practices for density estimation of discrete & continuous random variables

Question

I am currently trying to estimate the density of a joint distribution of K single dimensional RVs. I have at my disposal a set of N sample points, each of which represents an outcome of the K RVs.

Some specifics about my problem:

• the RVs are independent

• the RVs need not belong to same family of distributions

• the RVs can either all be discrete or all be continuous, but not both

• I know the upper and lower bounds for each RV; in the discrete case, I know the values that the RV can take on.

Right now, I am estimating the density of each RV separately using the ksdensity function in MATLAB. The independence assumption then allows me to produce a joint density using the product of the individual densities. I am hoping to improve the precision of my estimate this by either using another method (that I can code up in MATLAB) or by playing around with the options in ksdensity (such as the kernel type, support, width of the density window).

I am specifically hoping that people can shed light on:

• What method to use for the discrete case vs. the continuous case. In the discrete case, is it worth specifying the bounds and the values? In the continuous case,

• Whether it ever makes sense to forget the independence assumption and estimate the joint distribution as a joint distribution

• Whether anyone knows about some simple reading material on the issue.

Answer 1

I will only talk about the continuous case.

First of all i dont use MATLAB (rather R or code myself what i need) but i looked at your link :

• the ksdensity function uses normal kernel which is ok (another smooth kernel you could try is the Epanechnikov kernel, two other good choices may be biweight kernel or Silverman kernel but you will have to implement it by yourself).

• but this function uses a very strange choice of width unless you try to estimate normal RV, if not the easiest way to do the job is to use cross validation to chose your width

• if your RV are compactly supported kernel smoothing will generate artifacts near the end points of the support that you may want to correct (your estimated RV may go over the support of the "real" one)

If you are sure that your RV are independant you can estimate each one separatly and do the product (and no you should NOT use copulas since their purpose is to model dependancy)

I'll end with two things you may want to read :

• a very good (but quite theorical) reference is Tsybakov's book "Introduction to non parametric estimation"

• a very easy to read if you can access it reference is "Kernel smoothing" by Chapman & Hall.

Answer 2

You might want to try using a Copula. There are some free versions of this on the web, the most promising of which appears to be Andrew Patton's Copula toolbox (I have not used this, mind you, but it looks about right).