1
$\begingroup$

What exactly is a representation in the context of machine learning? Is it the parameters that a model has learned? Is it the data? Is it a mathematical description of the model? Is it something else? Or is it a loosely defined term without a precise meaning?

Clearly representations are important. Ian Goodfellow mentions (Goodfellow, Bengio, and Courville 2016, p3):

The choice of representation has an enormous effect on the performance of machine learning algorithms.

In the context of neural networks, Chollet says that layers extract representations. In Chollet's words (Chollet 2018, 28):

The core building block of neural networks is the layer, a data-processing module that you can think of as a filter for data. Some data goes in, and it comes out in a more useful form. Specifically, layers extract representations out of the data fed into them--hopefully, representations that are more meaningful for the problem at hand. Most of deep learning consists of chaining together simple layers that will implement a form of progressive data distillation. A deep-learning model is like a sieve for data processing, made of a succession of increasingly refined data filters--the layers.

That makes me think that representations are the form that the training/test data takes as it is progressively transformed. e.g. words could initially be represented as dense or sparse (one-hot encoded) vectors. And then their representation changes one or more times as they are fed into a model.

Mitchell says that we need to choose a representation for the target function. He describes representations as follows (Mitchell 1997, 8):

Now that we have specified the ideal target function $V$, we must choose a representation that the learning program will use to describe the function $\hat V$ that it will learn.

This makes me think that the 'representation' could be described as the architecture of the model, or maybe a mathematical description of the model. With this definition, we don't know the true representation (equation) of the target function (if we did we would have nothing to learn). So it is our task to decide what equation we want to use to best approximate the target function.

So which is it? And does anyone have any good sources on the topic?

References

Chollet, F. 2018. Deep Learning with Python. Manning Publications Company. https://books.google.com/books?id=Yo3CAQAACAAJ.

Goodfellow, I., Y. Bengio, and A. Courville. 2016. Deep Learning. Adaptive Computation and Machine Learning Series. MIT Press. https://books.google.com/books?id=Np9SDQAAQBAJ.

Mitchell, T. M. 1997. Machine Learning. McGraw-Hill International Editions - Computer Science Series. McGraw-Hill Education. https://books.google.com/books?id=xOGAngEACAAJ.

$\endgroup$
0
$\begingroup$

Representation is a word and the exact meaning will significantly depend on what is said before or after (the context) than the word itself.

In the 3 examples of usage drawn from the classic literature one can see the same word being used to refer to different concrete things:

  • representation as the way the registers are encoded in order to be used by an algorithm
  • representation as a pattern or regularity in the data
  • representation as a definition of a family of functions

From those few usages, we can extrapolate that an representation is an abstraction, a model, a way to refer or describe something else, the ojects (or its properties) which will be concretely defined, instantiated or manipulated in a different point/place/instant.

Finally, Representation is not a specific term of Machine Learning field terminology or even an mathematic jargon. Representation is basically an word find in general epistemology and/or linguistic. Thus, the best reference for it is a good dictionary.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.