You might want to read Guerrero, V.M. (1993) Time-series analysis supported by power transformations. Journal of Forecasting, 12, 37–48.
The above paper is the only paper that I know of have developed automatic technique to determine box cox transformation parameter lambda and importantly this procedure is model independent. This is done by minimizing coeficient of variation of time series. This is implemented in the forecast
package in R software. See example below:
library('forecast')
lambda <- BoxCox.lambda(AirPassengers,lower=0)
There are better ways to test stationarities. Box cox transformation is used for stabilizing variance not to check stationarity. You might want to use specific procedures such as Ljung-Box test, augmented Dickey-Fuller test, ACF, PACF and others to check stationarity. See this website for a nice summary of methods and how to apply in $R$.