How to deal with a variable-sized real vector of inputs?

Question

I have a collection of objects with properties that I measure. For each object, I obtain a vector of real numbers describing that object. Each object results in a vector having a different length. I also measure, say, the mass of the object, and I now want to relate the vector of things I've measured to the mass.

It's common in my field to extract features from this vector, e.g. take an average or some linear combinations of the values; and then use those extracted features to infer the mass (or whatever) using for example neural networks. It was recently shown, however, that a very complex combination of the elements of the vector result in a much better model of the mass.

There are still residuals in this model, however, even when working on simulated data. Presumably then there is a better way out there to manipulate these variable-length vectors in order to get a better model.

I am wondering if it is possible to do machine learning with real-valued input vectors of all different lengths. I know for text mining there are things like the bag-of-words approach, but it is unclear how such a method would work on real-valued vectors. Is there any research in this area?

Answer 1

It's hard to say without knowing more about the nature of your data. But, here are some possibilities:

1. Transform your vectors to a fixed-length representation, then use any standard method that takes fixed-length vectors as input. You'd have to define the transformation.

2. Use a method that acts on pairwise relations between objects. A simple example would be k nearest neighbors regression/classification, or any other distance-based approach. In this case, you'd define a distance function that takes two variable-length vectors as input and outputs a single value quantifying how distant they are. You could also consider kernel methods (e.g. support vector regression/classification). In this case, you'd define a kernel function that takes two variable-length vectors as input and outputs a single value measuring their 'similarity'. You'd then work with the kernel matrix $K$, where $K_{ij}$ measures the similarity between data points $i$ and $j$. The interpretation is that $K_{ij}$ gives the dot products between data points $i$ and $j$ after implicitly mapping them into a high dimensional feature space. Note that $K$ has to be positive semidefinite.

3. Recurrent neural networks can deal with variable-length sequences. If your vectors represent a sequence (or could be interpreted as one), this could be a good option.

4. The analogy to the bag-of-words approach for real data could be something like a histogram. In this approach, you'd calculate a histogram for each of your data points, using a common set of bins. Then, use the bin counts as the feature vector to pass to downstream learning algorithms.