# How to identify that relationship between two variables are monotonic or not?

As we know that the Spearman correlation is suitable measure under the assumption of monotonic relationship.

I would like to know that is there any standard method to figure out mathematical rather graphically (by assessing scatter diagrams)?

I am interested to identify variables that have non-monotonic relationship.

Your understanding of Spearman's rank-correlation measure seems wrong. There is no monotonicity assumption in applying it. In fact it was designed exactly for the purpose of measuring how monotonic the relationship is: if it is 1 (resp. -1), then a higher x value means a higher (resp. lower) y value.

The closer the value is at 0, the more you can say "the relationship (if any) is non-monotonic".

Now if you want to distinguish a weak non-monotonic relationship from a strong non-monotonic relationship, we need to get a little bit creative: One option is to compare the square of Spearman's rank correlation with the R-squared from a rank-regression with non-linear parts such as squares or splines: $$\text{rank}(y_i) = \alpha + \beta_1 \text{rank}(x_i) + \beta_2 \text{rank}(x_i)^2 + \varepsilon_i$$ A low squared Spearman's rank correlation but high R-squared from such regression indicates a strong non-monotonic relationship.

A small example for illustration:

x <- seq(0, pi, by = 0.01)
y <- sin(x)
plot(y ~ x)

cor(rank(y), rank(x))^2 # almost 0
summary(lm(rank(y) ~ poly(rank(x), 2))) # Multiple R-squared:  0.9375

# More safe (works in almost any case):
require(splines)
summary(lm(rank(y) ~ ns(rank(x), df = 6))) # Multiple R-squared:  0.9986


So here, we would speak of a strong non-linear relationship. A quick look at the scatter plot: 