I am familiar with the basics of how to present a problem to a machine learning algorithm using binary encodings. I am also familiar with, but still learning about, feature selection/extraction and construction. However, I am wondering if there is a term or keyword for the way in which you deicde to encode your problem for an algorithm, and how someone can improve their ability to present this problem to an algorithm. I believe this to be related to, but distinct from feature selection and construction. I am looking for a general answer, and a topic/keyword to continue learning afterwards, but I have provided a few examples below to illustrate where I have found difficulties:

For example, a common problem is how to solve a maze in under 100 steps by representing each step (forward, left, right, backwards) using binary values 00, 01, 10, 11. When presenting this to a genetic algorithm you would then use two bits for every step for 200 total bits. But I'm not sure I could easily present this same problem to a neural network.

For a neural network, days of the year could be represented by using 365 individual nodes, of which only one would be activated and the rest would have a value of 0. When combined with other features, perhaps one or two, they seem to be dwarfed by the number of other dimensions created using 365 input nodes to represent a year. We could reduce the dimensions by using th 52 weeks of the year, but at the cost of losing information about individual days of the week. Surely, there must be a better way to think of this problem.

*Also if there are other methods than binary encodings, please indulge.


Truth be told, what I am asking is so basic it seems to be glossed over as trivial in many machine learning tutorials. They may give an example, such as the maze above, and quickly explain how to encode this to the algorithm at hand, but I have seen very little material which focuses solely on the many different ways a problem can be encoded and presented to an algorithm... or else I am not looking in the right places which is why I asked for some keywords. This lack of focus is surprising to me considering how important it is to ask the machine the proper question and in the correct form, which in my opinion, supercedes even the subject of feature selection which would come after the encoding patterns have been figured out; technically, the features are selected first and then they are encoded second. Surely there are more ways than one to think of a given problem.

  • $\begingroup$ Please read the ADDENDUM, it's the source of the reason for my question. $\endgroup$ – user58446 Apr 10 '15 at 22:33
  • $\begingroup$ Yes. you're spot on in your observation I think. On the flip side though, machine learning can be considered a sub-field of AI. In which case, you don't want to have to have a human engineer features for the learning algorithm, the algorithm should deal with it. In this sense, your question might be somewhat outside the scope of machine learning. Not to mention the best way to engineer features an is extremely domain specific question. Features for finding a path through a maze is pretty different than differentiating dogs from cats in image data. Also, see my edit in the answer below. $\endgroup$ – bill_e Apr 10 '15 at 22:52

This is more from a linear model point of view, you can use these encodings for other machine learning algorithms. Some may be more appropriate than others. For a search term or keyword you could look into "categorical variable encoding", or "contrast coding". There are many possibilities. Here is a resource from UCLA that provides a nice overview for some of these.

Another popular encoding that can't be used in regression (because it makes $X^T X$ not invertible) but is commonly used is what you described for neural networks. It's called one-hot encoding.

Further down the rabbit hole, hierarchical linear models avoid the problem of having large numbers of independent variables, and so large degrees of freedom, (like the 365 NN nodes like you mention) by what amounts to partially pooling the regressions. The software packages STAN, JAGS, and BUGS are primarily for these types of models.

Also, consider making features (call them $f_{i}$) that are products, etc. of other features. E.g. define $f' = f_1 \times f_2$, or maybe $f' = \frac{f_1 \times f_2}{log(f_3)}$. This can be very effective in some cases. Maybe google "feature engineering".

This isn't a complete list, I'm sure there is more, but hopefully this can get you started.

  • $\begingroup$ Thank you for your answer, I am looking into your suggestions. Please see the addendum I added in case it may offer more insight to the question I am asking. $\endgroup$ – user58446 Apr 10 '15 at 22:36
  • $\begingroup$ Is that not simply 'feature extraction'? I think I need more creativity when it comes to developing the encoding of the features once they have been selected... is this different from 'feature construction' and 'feature engineering'? $\endgroup$ – user58446 Apr 10 '15 at 22:55
  • $\begingroup$ No.. with the term feature extraction I think more of like, say, the average intensity of the pixels in some 20x20 patch of an image. I think I gave you a lot of options for creativity. You have to design features that make sense for a specific problem. How you make/encode features is generally very domain specific. The best performing algorithms always have excellently designed or encoded features that make sense for the problem at hand. How you do that is really the creative part I think. $\endgroup$ – bill_e Apr 10 '15 at 23:03
  • $\begingroup$ One last thing: Have you ever seen any instructional material which deals with just this task? Sure I could gather a bunch of problems from a bunch of different sources and see how they encoded them... but I am looking for something more concise and plentiful of examples. I could also think of features on my own and figure out how to encode them, but I was hoping to see how they might be encoded beyond just what I can come up with on my own. $\endgroup$ – user58446 Apr 10 '15 at 23:18
  • $\begingroup$ Nope I don't think you will find that. @seanv507 gave a good answer (science!). Looking at how people treated real world problems is the best way to learn. As a bonus you'll learn a lot more than just feature preprocessing too. Maybe if you want to focus on algorithms look at solutions to kaggle competitions. $\endgroup$ – bill_e Apr 11 '15 at 5:33

As mentioned by Andre5 the term is called 'feature engineering', and is mostly outside machine learning. I personally find the term misleading since it is all that is known about the problem (namely science)! Nevertheless one can come up with general guidelines by considering how similarity is represented In the algorithm

Consider a Neural network, the fundamental operation is a dot product (cosine similarity).. So you need to ensure that what inputs seem similar to you are similar by dot product. So your example of movement doesn't seem like a good representation.. A better one would be left/right as one dimension and fwd/back as the other.

Similarly your day of the year is bad because every day is orthogonal (not similar) to any other day. Instead you want your input to represent dimensions along which days are similar (and relevant to your problem) eg is it a shopping day/work day/hot cold day etc.

Hierarchical representations where you add the the higher level category (eg week) can be helpful together with l2 regularisation (weight decay).. The idea being that you learn a simple model based on the higher level feature when you have a few instances per week, but as you get more examples you develop a more detailed model of daily effects

  • $\begingroup$ I had to give @Andre5 the Answer Mark because he put up and kept up with all my additional comments to his answer as well as provided additional keywords, but as soon as I get enough reputation I will come back and +1 your answer. Thank you very much. $\endgroup$ – user58446 Apr 12 '15 at 0:32
  • $\begingroup$ Not sure why it matters how the inputs are presented to the algorithm as long as they are consistent, in terms of forward/backward/left/right. Also, I would think that the day of the year may be relevant for certain types of data that span many decades. However, the increase in dimensionality may not be worth it. Instead, I would have thought using a 52-week period, or perhaps a Monday-Friday relationship may be useful for finding correlations in dates or days. $\endgroup$ – user58446 Apr 12 '15 at 2:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.