Currently I am reading "Deep Learning" by Ian Goodfellow, Yoshua Bengio, and Aaron Courville. I'm on Chapter 2 which is the Linear Algebra section where they go over the linear algebra that pertains to the book. I understand most of what is being taught but not at a deep level. And when I get to some of the latter parts of the chapter like "2.9 The Moore-Penrose Pseudoinverse" and specifically "2.12 Example: Principal Components Analysis", I don't really understand them that well at all.

Would it be okay if I go onto Chapter 3 and beyond before I understand these concepts comfortably, or will I be fine with having basic knowledge of them and the symbols they use?

  • 3
    $\begingroup$ No, you should learn all linear algebra you can get your hands on if you want to understand machine learning in any depth. $\endgroup$ Aug 5 '18 at 7:23

This is a question that often pops up when reading mathematical literature.

The initial chapters, of this book or any other math book, lay out tools that you will be using in later chapters, so strictly speaking, you will not understand the rest of the book without understanding these foundational chapters.

Realistically speaking, don't worry if you don't understand something. Continue reading until the topic actually appears and is applied. Then, and only then, re-read the earlier section, and try to make sense of it in the light of the later application. By then, you will have seen a lot of other material and may be able to understand it much better against this background.

In addition, it is often very good to look at other sources at this point, when you actually need to understand the application of something. Different authors have different ways of explaining stuff. Looking at things from different angles can be very helpful.

It has been said that good mathematical writing is the kind where you can mentally replace every formula by "foo" and still understand the gist. Read the formulas when you need to understand something in depth and detail.

Regarding the two specific topics you mention:

  • The Moore-Penrose pseudoinverse is fundamental when you want to create an actual estimation algorithm. If you are mainly interested in applying algorithms someone else has developed and implemented, then you need to understand that algorithm, but much less so the gory details. I have never needed to understand the Moore-Penrose pseudoinverse. We only have very few threads here on it, too.

  • PCA is much more useful to someone actually applying a tool. Conversely, someone building a tool will likely not use it very much. It's really good to understand this and related ways of reducing dimensionality or compressing information. If you come across a situation where PCA can be helpful in preprocessing, there will not be a big sign pointing this out, so you need to develop your own intuition and understand that this method exists. Happily enough, we have an astronomically upvoted mother-of-all-canonical-threads on PCA, along with an entire tag. Go through that thread, then re-read Goodfellow et al. on PCA. Enlightenment is almost sure to follow.

  • 3
    $\begingroup$ (+1) Another benefit of skimming details until you need them is by then you'll actually have a reason to understand, and this motivation facilitates the learning process. $\endgroup$
    – dsaxton
    Aug 4 '18 at 23:38
  • 1
    $\begingroup$ I agree with both Kolassa's answer and Saxton's comment, but I would like to add another point. Partly, the answer depends on the question- what are your intentions ? If your intent is to write algorithms or improve upon current code, at some point you will need to master the mathematics. If your intent is to apply the techniques, then viewing things like PCA or inverses as black boxes that accomplish a specific purpose is reasonable. For example, I truly think that teaching the details of 'back-propagation' is at best useless for most users of a neural net. $\endgroup$
    – meh
    Aug 8 '18 at 17:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.