A random forest composed of not CART trees, but gradient boosted trees has the minimization of exponential errors as one of its priciples. (Feature Selection with Ensembles, Artificial Variables, and Redundancy Elimination by Tuv, Borisov, Runger, Torkkola (2009)). You should be able to weight your "output" vector (matrix multiply) and the boosted tree will account for that weight in its "training". (adaboost tree tutorial).
I have been giving some thought to the domain in terms of statistical design of experiment (DOE) and you can account for a euclidean distance and higher order interactions with the (very) weak learner of a CART by "manually" expanding your domain.
If you had univariate data in a single column (trivial case) titled "x" then you could make "x^2" or "x^3" and construct an augmented domain matrix as [$ x$ $ x^2$ $ x^3$ ...]. If you have two variables composing your domain as "$ x_1$" and "$ x_2$" then you could consider interactions as well as powers as composing your augmented domain matrix as [$ x_1$ $ x_1^2$ $ x_1^3$ ...$ x_2$ $ x_2^2$ $ x_2^3$ ... $ x_1 x_2$ $ x_1^2 x_2$ ... $ x_1 x_2^2$ ...]. If you were motivated you could also consider contrasts ($ x_1-x_2$), their powers, and their interaction with variables, variable powers, variable interactions, other contrasts and their powers.
Though this might substantially increase the number of columns, one of the great strengths of random forests is the ability to handle very high dimensional data. This is not a "no go" computationally or analytically.
This gives you a way to account for interactions and distances in the data without having to abandon your current tool.
It isn't exactly what you were looking for but satisfies some of your initial question and gives you some directions you can consider exploring.
Best wishes.