I got the residual mean deviance over 600, which seems a lot?

This is exactly what should happen. By dividing your Prob_1 outcome variable by $100$, you change the units (such as going between centimeters and meters).

When you do divide by $100$, you get that the error is $0.06002\space m^2$. When you do not divide by that $100$, you get that the error is $600.2\space cm^2$.

$$0.06002\space m^2 = 600.2\space cm^2$$

You might not be working in meters and centimeters, but these error values all have units, and you are getting the same answer whether you divide by $100$ or not, just in different units.

What are considered to be good values of MSE, when can I say the tree is predicting correctly?`

See this for why that requires a context.

The % Var explained= 0.69, which is quiet low, and when changeing the mtry, the value is some times even with a - sign (for example -3.22).

Again, whether or not a particular measure of performance is any good requires a context, and it might be that your value of $0.69$ is pretty good! For instance, I have seen papers in top journals with values a tenth as high as that.

Regarding the alues below zero, in a nonlinear regression like a random forest, the notion of "proportion of variance explained" is a bit dubious, as I explain here. However, you can regard that value as being a comparison of the mean squared error of your model to that of a baseline model that you must beat. If your value is less than zero, your model is doing a worse job of predicting than that "must beat" model is doing. If you get a result that your model performance can range from a solid value of $0.69$ to a totally unacceptable value less than zero, it would seem that your predictions are unstable.

I got the residual mean deviance over 600, which seems a lot?

This is exactly what should happen. By dividing your Prob_1 outcome variable by $100$, you change the units (such as going between centimeters and meters).

When you do divide by $100$, you get that the error is $0.06002\space m^2$. When you do not divide by that $100$, you get that the error is $600.2\space cm^2$.

$$0.06002\space m^2 = 600.2\space cm^2$$

You might not be working in meters and centimeters, but these error values all have units, and you are getting the same answer whether you divide by $100$ or not, just in different units.

What are considered to be good values of MSE, when can I say the tree is predicting correctly?`

See this for why that requires a context.

The % Var explained= 0.69, which is quiet low, and when changeing the mtry, the value is some times even with a - sign (for example -3.22).

Again, whether or not a particular measure of performance is any good requires a context, and it might be that your value of $0.69$ is pretty good! For instance, I have seen papers in top journals with values a tenth as high as that.

Regarding the values below zero, in a nonlinear regression like a random forest, the notion of "proportion of variance explained" is a bit dubious, as I explain here. However, you can regard that value as being a comparison of the mean squared error of your model to that of a baseline model that you must beat. If your value is less than zero, your model is doing a worse job of predicting than that "must beat" model is doing. If you get a result that your model performance can range from a solid value of $0.69$ to a totally unacceptable value less than zero, it would seem that your predictions are unstable.

This is exactly what should happen. By dividing your Prob_1 outcome variable by $100$, you change the units (such as going between centimeters and meters).

When you do divide by $100$, you get that the error is $0.06002\space m^2$. When you do not divide by that $100$, you get that the error is $600.2\space cm^2$.

$$0.06002\space m^2 = 600.2\space cm^2$$

You might not be working in meters and centimeters, but these error values all have units, and you are getting the same answer whether you divide by $100$ or not, just in different units.

I got the residual mean deviance over 600, which seems a lot?

This is exactly what should happen. By dividing your Prob_1 outcome variable by $100$, you change the units (such as going between centimeters and meters).

When you do divide by $100$, you get that the error is $0.06002\space m^2$. When you do not divide by that $100$, you get that the error is $600.2\space cm^2$.

$$0.06002\space m^2 = 600.2\space cm^2$$

You might not be working in meters and centimeters, but these error values all have units, and you are getting the same answer whether you divide by $100$ or not, just in different units.

What are considered to be good values of MSE, when can I say the tree is predicting correctly?`

See this for why that requires a context.

The % Var explained= 0.69, which is quiet low, and when changeing the mtry, the value is some times even with a - sign (for example -3.22).

Again, whether or not a particular measure of performance is any good requires a context, and it might be that your value of $0.69$ is pretty good! For instance, I have seen papers in top journals with values a tenth as high as that.

Regarding the alues below zero, in a nonlinear regression like a random forest, the notion of "proportion of variance explained" is a bit dubious, as I explain here. However, you can regard that value as being a comparison of the mean squared error of your model to that of a baseline model that you must beat. If your value is less than zero, your model is doing a worse job of predicting than that "must beat" model is doing. If you get a result that your model performance can range from a solid value of $0.69$ to a totally unacceptable value less than zero, it would seem that your predictions are unstable.