0
$\begingroup$

I have a bunch of training examples (which are time series segments) that are used to train an algorithm. For any new example that is now presented to the algorithm for classification, i want to the determine how "similar" it is to the training examples (not to a specific one, but all of them) to check if it is "covered" by the training set.

Can you guys point me into some direction how to approach this? My first thought is that I have to come up with a problem specific similarity function, compare it with every training example and then use the mean of all comparisons.

One could also use one of the standard time series similarity measurements, but would those compare the examples with respect to their important features?

Thank you.

$\endgroup$
1
$\begingroup$

Estimation-error-based

After estimating the model, you could use the errors from your model to find similar examples, i.e. examples with the same error. Thinking this one step forward, you could decide from the errors where you need more examples (i.e. examples with large errors) and where you have enough (i.e. many examples with the same small errors). This principle is used in Cook's distance to find influential data points. Comparing the prediction errors on the test set with those on the train set, you find similar examples.

Clustering

If you were not to estimate the model first, and just use the features, I'd say the most straightforward approach is to cluster your data with the features and then see in which cluster your test data points fall into.

$\endgroup$
  • $\begingroup$ the first seems better in terms of the important features question $\endgroup$ – user0 Jul 12 '18 at 18:32
  • $\begingroup$ Thank you. The estimation error based sounds interesting. One problem is (in my opinion) that it quantifies how well certain examples have been learned by the model, referring to an entire class. I am really looking for a way to quantify how "uncommon" a specific example is with respect to the entire training set. $\endgroup$ – sinpalabras Jul 18 '18 at 16:16
  • $\begingroup$ In that case, clustering is probably more reasonable. Then, either allow for clusters of varying sizes and define uncommon examples as examples belonging to small size clusters; or for each cluster define uncommon examples as those examples with the largest distance to the cluster mean. $\endgroup$ – Nic Jul 18 '18 at 16:24
0
$\begingroup$

In relatively high dimension, you would suffer from 'the curse of dimensionality', which would mean that in general your training set will cover very little of the space it lives in.

If I were you, I would compute the kernel (rbf for instance) between the two sample as a measure of similarities. Again, keep in mind that the regular L2 distance does not tell you much about the similarity of the sample in high dimensions.

Hope this helps

$\endgroup$
  • $\begingroup$ Thanks. So lets say I have 9 features per segment and therefore my examples are represented by a 9x1 feature vector. Now I just compute the euclidean distance between a specific examples featrue vector and the feature vectors of the entire training set. This doesnt tell me much about the "uncommonness" of that new example? $\endgroup$ – sinpalabras Jul 18 '18 at 16:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.