This is a good question, I was in this position myself a while back and it feels extremely daunting.
Knowing the theory is very important in using machine learning practically. Without the theory machine learning just becomes black box you can send data into to get an answer. Great if your plotting a set of meaningless x's and y's from an example you don't care about; but terrible if you actually want to make use of the result. A simple example of this would be regression, higher order regression (eg. aX^2+bX+c) will almost always give you a better fit to your data. You can try this with any program for plotting data and finding the line of best fit (excel is what I have in mind). As you increase the order of the polynomial your error, or R squared values approaches 1 (1 being perfect fit). Seems like a no brainer, whak the order up to 100 and get your perfect line of best fit. However if you understand the way regression works you know that as the number of features in the regression (X^2,X^1,X^0) increases you need more data to fit these features. This is especially the case when you're considering extrapolation over interpolation. Understanding the way the model works allows you to use the right type of model for your data, and get useful answers; answers that you understand the scope and limitations of.
So we know why theory matters. But outside of the simple example about how do you know which model to use. It's a tough question.
A good rule of thumb I read many years ago on this site is you want at least 15 times as many data points as you have features, this is to deal with overfitting - but this is not always possible in my experience.
Another good rule is pick a model that feels like the problem. Got a problem with periodic data (regular ups and downs) maybe think about what models you know that can use a sin wave.
A hard rule is don't confuse regression and classification problems, it makes sense to round 1.5 up to 2, it doesn't make sense to round Red up to Blue.
A model with fewer features is simpler. Simpler models tend to optimize more quickly.
This is the most important rule: if you don't really understand the way the model works write down the kind of result you expect (y increases wrt x^2, houses with more rooms sell for more money, etc.) then if your model disagrees either find a really, really good reason why your intuition was wrong or accept that you don't know what has happened and this model isn't for you until you've done more reading.