I am reading the following section of the book Deep Learning.

enter image description here

Can you provide an intuitive explanation of the above section? I don't quite understand the statement "When this number is large, matrix inversion is particularly sensitive to error in the input" and "This sensitivity is an intrinsic property of the matrix itself, not the result of rounding error during matrix inversion. Poorly conditioned matrices amplify pre-existing errors when we multiply by the true matrix inverse"


I think you're confused with the usage of the word input. Naturally, in deep learning context we mean a vector $x$ by input. However, in this passage it is the matrix $\textbf A$ that is referred to as input.

Think of the matrix $\textbf A$ not as a constant predetermined matrix, but as of a parameter that is estimated. Maybe you estimate $\textbf A$ from training data etc. So, in a way it is a random value itself, random matrix it is. Hopefully, your estimation routine is consistent so that you can improve your precision by increasing the sample training data.

Now, what the passage states is that if the matrix property called "condition number" is very large then inverting the matrix $\textbf A^{-1}$ is very sensitive to the input $\textbf A$. Any random variations (noise) in estimated $\hat{\textbf A}$ will result in widely different outcome of matrix inversion routine $\hat{\textbf A}^{-1}$. Condition number tells you how much the input noise is amplified in the output of the inversion routine


In mathematics, a condition number is a number representative of the change of an output proportionate to a change in the input of a function. For example, if a small change in the input results in a small change in the output, the function produces a small condition number and is said to be well-conditioned. Alternatively, if a small change in the input results in a large change in the output, the function produces a large condition number and is defined as ill-conditioned.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.