For time series analysis: "Forecasting Principles and Practices" by Hyndman and Athanasopoulos is absolutely excellent and is roughly on the same order of mathematical complexity as ISLR (i.e. enough, but not too much). It has the additional bonus of being available for free online, and having many code examples. It has one weak point: It doesn't ...
If you’re interested in Bayesian Inference then there’s a wonderful book (goes into GLMs quite a lot) called Statistical Rethinking by Richard McElreath. The second edition is just out and there’s lecture series on YouTube. The most recent series (called Winter 2019 IIRC) follows the second edition.
There are a couple of good review papers on the topic of deep learning for forecasting:
Neural forecasting: Introduction and literature
Recurrent Neural Networks for Time Series Forecasting: Current Status
and Future Directions
And a very good presentation by the amazon team
A word of warning though: I am a very big fan of LSTM based forecasting ...
The "classical" methods comprise much more than ARIMA and GARCH (which address different questions, and at least ARIMA is not very useful for forecasting), e.g., decomposition, Exponential Smoothing etc. I recommend this very good free online textbook by Athanasopoulos & Hyndman.
I agree that there is very little in terms of textbooks on HMMs ...
Another two books similar to Nocedal & Wright are
Numerical Optimization, by Bonnans et al.
Optimization Theory and Methods, by Sun & Yuan
But since you're looking for optimization methods applicable in data science and machine learning, keep in mind that the sheer size of models in this field usually requires stochastic versions of algorithms ...
Aside Nocedal & Wright (2006), as books of a similar level, I have found:
"Iterative methods in Optimisation" by Kelley (1999) and
"Optimization" by Lange (2013).
Both books are equally easy to follow too with N&W and cover standard numerical optimisation topics (KKT Theory, Newton's Method, Quasi-Newton, etc.) nicely. Kelley's ...
Haven't read this new edition, but the first edition is a classic, so this one, available starting September 2020, will be a great reference for sure.
I second the recommendation of "Statistical Rethinking" by Mooks, that's a great one.
(Not really an answer, simply an elaboration on the comment. Apologies in advance.
It is a very interesting question. Hopefully the comment will be complementary to the answer(s) when they come along.)
It seems that, in order to make meaningful statistical statements, one needs densities/likelihood functions. Therefore a dominating measure necessarily shows ...
Yes, the book by Eugene Demidenko, "Mixed Models: Theory and Applications with R" is a good one for a mathematician.
You will see from the "why I wrote this book" that he is coming from a mathematical perspective:
Amazon has quite of lot of preview pages:
For survival analysis, Kleinbaum (2013) - Survival Analysis -- A self-learning text is straightforward with R examples. It's even freely available on Springer now due to COVID-related university lockdowns: https://link.springer.com/book/10.1007%2F978-1-4419-6646-9.
I think Frank Harrell's Regression Modelling Strategies is also freely available now for the ...
For GLMs I recommend Faraway's Extending the Linear Model with R. I would also recommend Frank Harrell's Regression Modeling Strategies, which provides a nice in depth explanation of regression as a whole and various extensions including survival modeling. Both textbooks include code in R.
Entropy pops up everywhere in statistical inference and machine learning. For instance:
Finding a parameter which maximizes the likelihood of the data is equivalent to finding a parameter which minimizes the KL divergence.
The Principle of Maximum Entropy is applied to find statistical models that have the greatest entropy for a set of given constraints (a ...
 Shows consistency of the gradient boosting algorithm and gives some background on related work.
 Extends gradient boosting to the case of nesterov momentum and gives bounds on the optimization rate (how fast does the excess empirical risk go to zero).
 Biau, G., Cadre, B. (2017). Optimization by gradient boosting arXiv https://arxiv.org/...
This is the easiest book I have ever seen Head First Statistics (A Brain Friendly Guide)
This one is also a very good book with a lot of example. Schaum's Outline of Probability and Statistics: 897 Solved Problems
The combination of differential equations (e.g. ODE of SIR models) and HMM are often used in epidemiology. The hidden states are models as ODEs and the observation process are modeled as HMM. One example is pomp. The model is trained on existing data and produces forecasting on the future. Another goal of this kind of model is to understand epidemiology ...
Cauchy distribution is often used in finance to model asset returns. Also noteworthy are Johnson’s Bounded and Unbounded distributions due to their flexibility (I’ve applied them in modeling asset prices, electricity generation and hydrology).
Some common probability distributions; From here
Uniform distribution (discrete) - You rolled 1 die and the probability of falling any of 1, 2, 3, 4, 5 and 6 is equal.
Uniform distribution (continuous) - You sprayed some very fine powder towards an wall. For a small area on wall, the chances of falling dust on a spot on the wall is uniform.