When using the average, you are saying two things:
- Outliers are not a huge problem (otherwise you would use the median or at least filter out some outliers before taking the average)
- Every prediction has the same weight (otherwise you would factor in weights)
You shouldn't expect there to be huge outliers since you can make the sample size big enough for them to matter less in the average and since you would expect a minimum of stability from the predictions of the individual trees.
There is no reason to think some trees should have more predictive weights asweight than others, nor a way to determine such weights.
You cannot really use mode since the predictions are on a continuous scale. For example, if you had the predictions 80 80 100 101 99 98 97 102 103 104 96, mode would predict as 80. That cannot be what you want. If all values have distinct decimals, mode wouldn't know how to decide.
Other averages than the arithmetic mean exist, like the geometric mean and the harmonic mean. They are designed to pull the average down if there are some low values in the series of data. That's not what you want here either.