How To Better Represent A Problem To A Machine Learning Algorithm

Question

I am familiar with the basics of how to present a problem to a machine learning algorithm using binary encodings. I am also familiar with, but still learning about, feature selection/extraction and construction. However, I am wondering if there is a term or keyword for the way in which you deicde to encode your problem for an algorithm, and how someone can improve their ability to present this problem to an algorithm. I believe this to be related to, but distinct from feature selection and construction. I am looking for a general answer, and a topic/keyword to continue learning afterwards, but I have provided a few examples below to illustrate where I have found difficulties:

For example, a common problem is how to solve a maze in under 100 steps by representing each step (forward, left, right, backwards) using binary values 00, 01, 10, 11. When presenting this to a genetic algorithm you would then use two bits for every step for 200 total bits. But I'm not sure I could easily present this same problem to a neural network.

For a neural network, days of the year could be represented by using 365 individual nodes, of which only one would be activated and the rest would have a value of 0. When combined with other features, perhaps one or two, they seem to be dwarfed by the number of other dimensions created using 365 input nodes to represent a year. We could reduce the dimensions by using th 52 weeks of the year, but at the cost of losing information about individual days of the week. Surely, there must be a better way to think of this problem.

*Also if there are other methods than binary encodings, please indulge.

ADDEDUM:

Truth be told, what I am asking is so basic it seems to be glossed over as trivial in many machine learning tutorials. They may give an example, such as the maze above, and quickly explain how to encode this to the algorithm at hand, but I have seen very little material which focuses solely on the many different ways a problem can be encoded and presented to an algorithm... or else I am not looking in the right places which is why I asked for some keywords. This lack of focus is surprising to me considering how important it is to ask the machine the proper question and in the correct form, which in my opinion, supercedes even the subject of feature selection which would come after the encoding patterns have been figured out; technically, the features are selected first and then they are encoded second. Surely there are more ways than one to think of a given problem.

Answer 1

This is more from a linear model point of view, you can use these encodings for other machine learning algorithms. Some may be more appropriate than others. For a search term or keyword you could look into "categorical variable encoding", or "contrast coding". There are many possibilities. Here is a resource from UCLA that provides a nice overview for some of these.

Another popular encoding that can't be used in regression (because it makes $X^T X$ not invertible) but is commonly used is what you described for neural networks. It's called one-hot encoding.

Further down the rabbit hole, hierarchical linear models avoid the problem of having large numbers of independent variables, and so large degrees of freedom, (like the 365 NN nodes like you mention) by what amounts to partially pooling the regressions. The software packages STAN, JAGS, and BUGS are primarily for these types of models.

Also, consider making features (call them $f_{i}$) that are products, etc. of other features. E.g. define $f' = f_1 \times f_2$, or maybe $f' = \frac{f_1 \times f_2}{log(f_3)}$. This can be very effective in some cases. Maybe google "feature engineering".

This isn't a complete list, I'm sure there is more, but hopefully this can get you started.

Answer 2

As mentioned by Andre5 the term is called 'feature engineering', and is mostly outside machine learning. I personally find the term misleading since it is all that is known about the problem (namely science)! Nevertheless one can come up with general guidelines by considering how similarity is represented In the algorithm

Consider a Neural network, the fundamental operation is a dot product (cosine similarity).. So you need to ensure that what inputs seem similar to you are similar by dot product. So your example of movement doesn't seem like a good representation.. A better one would be left/right as one dimension and fwd/back as the other.

Similarly your day of the year is bad because every day is orthogonal (not similar) to any other day. Instead you want your input to represent dimensions along which days are similar (and relevant to your problem) eg is it a shopping day/work day/hot cold day etc.

Hierarchical representations where you add the the higher level category (eg week) can be helpful together with l2 regularisation (weight decay).. The idea being that you learn a simple model based on the higher level feature when you have a few instances per week, but as you get more examples you develop a more detailed model of daily effects