What is the difference between parameter estimation which includes system identification and learning in machine learning perspective?
Let say the model is y= Ax. x is the input and y is the output. In estimation, I have seen parameters to be estimated, maybe in this case it is A and the samples are also estimated (unsure)
What is learning then?
Basicaly same, but flavor of terms is a little different - by estimation people usually mean that you specify underlying distributions and then estimate their parameters. Learning may be distribution free - just optimizing some target function and it applies in situations with complex structured data when it not reasonable/possible to build a distributional model.
Estimators are rules for estimating a value based on observed data. We have estimators like GMM or MLE which estimate certain values, whether parametrically or not. These rules must satisfy certain properties like consistency, unbiasedness, efficiency, etc.
Learning is more general, and includes methods with or without estimators. K-nearest neighbors uses distance functions and not statistical estimation to classify or regress. Fine tuning the hyperparameter $k$ is also practically a matter of improving the error metric rather than any statistical estimation per se.
