Differentiable method to convert voxel representation into pointclouds? I'm finding a way to convert 3D voxel data into 3D point-clouds.
Since I'd like to do it inside a deep-learning architecture, the conversion has to be differentiable. Is there such a method?
Thanks in advance!
 A: This happens to be in my area of research and I've thought about the problem for a while now. I think there are some fundamental reasons why this might not be possible.


*

*Pointclouds can't model uncertainty. I suppose you might first suggest that less dense parts are regions where our confidence that the region is solid is lower, but this doesn't really seem true -- a flat wall needs less points to model than a detailed object. Given a fixed budget of points, a small object would have a higher density of points than a large object, yet we're not necessarily more certain in one than in the other. I suppose you could also assign a confidence value to every point in the cloud and represent uncertainty that way. But this also doesn't make sense, because then a valid pointcloud model of a car would be to put all the points on a regular grid and set the confidence value of all the points not in the car to 0, and vice versa. This doesn't seem like the right way to do uncertainty in a pointcloud.

*Voxel occupancy aren't really differentiable wrt deformations or transformations. This almost follows simply from the fact that everything is on a fixed grid. Not all hope is lost though -- you can apply any transformation or deformation by resampling the voxel occupancy using trilinear interpolation, allowing the gradients to flow through. The reason this doesn't really work as well as you might expect is that these gradients are usually highly local, since linear interpolation only looks at neighboring voxels. 
So fundamentally, you end up wanting to convert gradients wrt to the position of each point (translational/rotational/deformation gradients) into gradients wrt voxel occupancies. And these are two different things so it never really works out. 
You can always make up some sort of differentiable operation of course: for example you could slide a 2x2x2 window across the voxel occupancy grid, and then for each window, compute the average of 8 occupancies. Use gumbel-softmax to decide whether or not to put a point there with probability equal to the average occupancy. Place the point on the weighted average of the 8 grid-points in your window, using the occupancy scores as weights. However I don't think this will work great, for reasons mentioned above.
