I've often used and heard this way of speaking. I'd guess that the sequence mentioning the outcome or response before the predictors follows from conventions in writing, using words or using notation or mixing the two, all the way up to
$Y = X\beta$
setting aside the equally interesting (or uninteresting!) question of what we call different kinds of variables.
But it seems equally valid mathematically and statistically to mention the predictors first, just as many mathematicians write mappings or functions with arguments first.
What often perhaps drives the sequence we use in statistical discussions is that scientifically or practically we usually have a clear idea of what we are trying to predict -- it is mortality, or income, or wheat yield, or votes in an election, or whatever -- while the pool of potential or actual predictors may not be so clear. Even if it is clear, it makes sense to mention the important things first. What are you trying to do? Predict whatever. How are you going to do it? Use some or all of these variables.
I don't have a story for "on" rather than any other word that would fit. I don't hear "regressed against" or "regressed with". There may be no logic here, just memes passed on along in textbooks, teaching and discussions.
In general, watch out. Consider a related issue, the meaning of "versus". I was brought up to say "plot $y$ [vertical axis variable] against (or versus) $x$ [horizontal axis variable]" and the reverse sounds singularly odd to me. Nevertheless people with considerable experience and expertise have it the other way round. Sometimes, this kind of difference might be traced to charismatic and idiosyncratic teachers who you have imitated ever since you sat at their feet.