Assume that we have 30 features (inputs) each of which can potentially influence the result (output). We try to use the available ("observed") mapping from features to targets to develop a predictive model that uses only some of the features. Each feature can be either included into the input of the model or not. It means that in total we have 2^30 combinations of what arguments to use. I would like to notice that model is fixed, I do not change the model (in a broad sense), I just decide what variables to use.

Now we have considered several thousand combinations and for each of the combinations we have a measure of how good it is (for that, of course, we use the testing set). For example, we know that if we take variable 1, 7, 13, and 27 we get an error (accuracy of the model) that is equal to 123.456. If we use variable 3, 4, 10, 11, 13, 27 and 29. We get an error equal to 23.456.

My question is: How to summarize the knowledge about performance of the model as a function of the used arguments?

In other words I need a "meta-model" that describes performance of my models. I need this because I want to use the obtained knowledge to search over the huge space of potential models more intelligently.

More specifically, I have a mapping from N binary vectors to a real (float) value. Each binary vector indicates if a corresponding variable is used or not and the real output describes the performance of the corresponding model. Now I want to use this "meta-data" to train a "meta-model". But before I start a "meta-training" I would like to decide what model is a good one to train. This model should capture such concepts as "information stored in variables". For example I can imagine that variable 7 has no additional information to the combination of variables 3 and 5. Or, we can have "orthogonal" variables that cover the target space "independently".

  • $\begingroup$ This is a very broad topic. You can get started in learning about it by searching our site for model selection. BTW, there are $2^{30}$ possible combinations of features (without even considering interactions or higher-order relationships), which is much greater than $30^2$. $\endgroup$
    – whuber
    Nov 17 '14 at 17:37
  • $\begingroup$ It was a typo with 30^2 and I think that the questions is much more specific than the model-selection question. I am not asking here about hot to use training and testing sets, how to validate a model, what measure to use. My question is more specific. I would like to know what is a good model to summarize knowledge about performance of another model. In other words, I am searching for a meta-model. $\endgroup$
    – Roman
    Nov 18 '14 at 7:54
  • 3
    $\begingroup$ After the edit, this is actually a really interesting question. I suspect you can just start with linear regression but I'd be curious to know if there's a "right" answer. $\endgroup$ Nov 18 '14 at 8:30
  • 1
    $\begingroup$ Once you're at this point with your $2^{30}$ scores, there's an optimization problem. If this is too large a space to search exhaustively, you need a search heuristic. For example, add/subtract variables one at a time, always choosing the optimal next one to add/subtract. $\endgroup$
    – awcc
    Nov 20 '14 at 8:40
  • $\begingroup$ The more interesting question is whether scoring all $2^{30}$ models by this one training set is reasonable (probably not if it's small), and what else to do if it isn't. Also, and maybe this is what you're getting at, you can look to some meta model that blends the various choices depending on how well they do. This seems pretty interesting. $\endgroup$
    – awcc
    Nov 20 '14 at 8:43

In case you are interested in a linear model, here's direction that hasn't been mentioned in the comments / other answers:

Let $x \in R^{30}$ be the feature vector, and you want to learn a linear mapping $y = w\cdot x$ that maps $x$ to a value (that can be transformed into a probability using logistic regression, from which you can do anything).

I assume that there's a cost involved in obtaining many features / using all of them for prediction (otherwise, why not use all of them).

What you can do is to regularize the learning process with $L_1$ norm. It is known that $L_1$ regularization results in sparser models (i.e. a weight vector $w$ with lots of zeros in it), which is what you want.

Specifically, you would minimize the following: $$ \textrm{RegularizedLoss}(w,\lambda) = \textrm{Loss}(w) + \lambda \cdot \|w\|_1,$$ where $\textrm{Loss}(w)$ is the logistic loss. There are efficient ways to do this (for example, vowpal wabbit can handle $L_1$ regularization).

The specific value of $\lambda$ controls the tradeoff between accuracy and sparsity, i.e. how much loss in accuracy are you willing to take in return for a model that uses less features.



Whenever I approach problem of optimization function that I cannot define, but I can calculate, I use some heuristic method.

As your searched space is big (but not BIG big, 30 features is not a lot), I would go with some genetic algorithm. This could be quite trivial, but representation of your problem may just be binary vector, so standard masked crossover and bit flipping mutation may have quite good results.

As for your question specifically: you can summarize this knowledge in form of GA itself. You do not need human-readable representation of your knowledge, so actual knowledge will be encoded in specimen representation and evaluation function.

Disclaimer: This is my intuition, close to guessing (but with some experience in heuristic optimization). I cannot show you any papers that would back this idea, but I'm pretty sure you'll find something on that.

  • 1
    $\begingroup$ it is clear that an alternative to an exhaustive search is not-exhaustive search with some optimization technique (for example genetic optimization). However, my question was about what is an appropriate to summarize our knowledge about importance of arguments (how informative they are). $\endgroup$
    – Roman
    Nov 20 '14 at 11:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.