What exactly is a representation in the context of machine learning? Is it the parameters that a model has learned? Is it the data? Is it a mathematical description of the model? Is it something else? Or is it a loosely defined term without a precise meaning?
Clearly representations are important. Ian Goodfellow mentions (Goodfellow, Bengio, and Courville 2016, p3):
The choice of representation has an enormous effect on the performance of machine learning algorithms.
In the context of neural networks, Chollet says that layers extract representations. In Chollet's words (Chollet 2018, 28):
The core building block of neural networks is the layer, a data-processing module that you can think of as a filter for data. Some data goes in, and it comes out in a more useful form. Specifically, layers extract representations out of the data fed into them--hopefully, representations that are more meaningful for the problem at hand. Most of deep learning consists of chaining together simple layers that will implement a form of progressive data distillation. A deep-learning model is like a sieve for data processing, made of a succession of increasingly refined data filters--the layers.
That makes me think that representations are the form that the training/test data takes as it is progressively transformed. e.g. words could initially be represented as dense or sparse (one-hot encoded) vectors. And then their representation changes one or more times as they are fed into a model.
Mitchell says that we need to choose a representation for the target function. He describes representations as follows (Mitchell 1997, 8):
Now that we have specified the ideal target function $V$, we must choose a representation that the learning program will use to describe the function $\hat V$ that it will learn.
This makes me think that the 'representation' could be described as the architecture of the model, or maybe a mathematical description of the model. With this definition, we don't know the true representation (equation) of the target function (if we did we would have nothing to learn). So it is our task to decide what equation we want to use to best approximate the target function.
So which is it? And does anyone have any good sources on the topic?
References
Chollet, F. 2018. Deep Learning with Python. Manning Publications Company. https://books.google.com/books?id=Yo3CAQAACAAJ.
Goodfellow, I., Y. Bengio, and A. Courville. 2016. Deep Learning. Adaptive Computation and Machine Learning Series. MIT Press. https://books.google.com/books?id=Np9SDQAAQBAJ.
Mitchell, T. M. 1997. Machine Learning. McGraw-Hill International Editions - Computer Science Series. McGraw-Hill Education. https://books.google.com/books?id=xOGAngEACAAJ.
