I want to estimate a Random Forest in some statistical language (SAS). This can only be estimated by setting the target variable as nominal instead of ordinal. Isn't it possible to estimate a Random Forest with a ordinal target variable in general? If it is possible, does estimating the target variable as ordinal instead of nominal have any prediction gains?
You can certainly use random forest for regression with an ordinal target variable, as forests algorithms do not use metric information in the data (so should give the same results for say, target variables $Y$ or $\log Y$), just tell the algorithm (if necessary) your target is continuous (and code it as that.) See for instance this or this list of post on this site. Or, as pointed out in a comment, there is a dedicted R package.
If it is possible, does estimating the target variable as ordinal instead of nominal have any prediction gains?
If your target variable is truly ordinal, then modeling as only nominal would be a huge loss of information, don't do it!
As hinted at in the comment by @Frank Harrell below, there is the possibiliy that you would be better off with dedicated methods for ordinal regression, see Best algorithms for ordinal classification.
If your question is about how doing this in sas or some other software, that is off-topic here. Ask on a sas forum.