A few clarifications first.
When you say that the dummy is 1 for exactly half the observations, that is clearly a "dummy" example, since it is unlikely that the regime shift happened exactly half-way through your sample. In a realistic scenario, you'd have to build the possibility of a break in the intercept and/or slope at an unknown data inside your sample.
If however you are certain that the break date occurs at the time that you think it does, then no further adjustment needs to be made, and you can follow the two-step Engle-Granger procedure to test for cointegration. This is highly inadvisable though, unless this is homework.
The issue of the critical values of the ADF is well-known and would affect the test of cointegration even in the absence of regime shifts. The Engle-Yoo critical values should be used in all cases.
What does cointegration mean in the context of the equation that you have set up? Recall that in the standard case, Engle-Granger residual-based tests of cointegration have no cointegration as their null, and the alternative is cointegration, that is, the existence of a cointegrating vector (a linear combination of the I(1) variables that is I(0)).
In this context, the null remains that the variables are not cointegrated, but the alternative now is existence of a cointegrating vector that potentially changes at the estimated break date, that is, there is a structural change in the cointegrating relationship at the data of the structural break. Note that there is a regime shift both under the null and the alternative hypothesis.
The appropriate reference here is the paper by Gregory and Hansen, which discusses residual based tests of cointegration in the presence of regime shifts.
Standard cointegration model
Consider the simple cointegration model
$$
Y_{1t} = \beta_0 + \mathbf{Y}_{2t}'\boldsymbol{\beta} + \varepsilon_t
$$
where $\mathbf{Y}_{2t}$ is an I(1) $m$-vector, and $\varepsilon_t$ is I(0). In the simplest case, $m=1$. The standard tests of cointegration estimate the above relationship by OLS and conduct tests of unit roots (where the null is that there is a unit root in the process) on the estimated residuals, $\hat{\varepsilon}_t$.
Regime-shift cointegration model
Now consider the model that you are interested in
$$
Y_{1t} = \beta_0 + \mathbf{Y}_{2t}'\boldsymbol{\beta} +\boldsymbol{1}_{\left[t \geq \tau\right]}\left(\beta_0^\dagger + \mathbf{Y}_{2t}'\boldsymbol{\beta}^\dagger\right)+ \varepsilon_t
$$
Note that the breakpoint is fixed at $\tau$, but is unknown. Just like before, $\varepsilon_t$ is I(0).
Testing in the regime-shift cointegration model
Following exactly the procedure in the non-regime shift case, we estimate this model by OLS, and compute the residuals $\hat{\varepsilon}_t$. Then we form tests of unit roots in the errors, considered at the fixed break date $\tau$ -- the $Z_\alpha(\tau)$, $Z_t(\tau)$, and the $ADF(\tau)$ tests. The construction of these test statistics is given on pages 104-105 of Gregory and Hansen -- there are no differences to the standard case with no regime shift.
The next step, however, is new, in that, the statistic for the sample is computed as the minimum of the statistics computed over all the possible break dates in the sample (trimmed to exclude the edges) in order to date the break. So,
$$
\begin{align}
Z_t &= \inf_\tau Z_t(\tau) \\
Z_{\alpha} &= \inf_\tau Z_{\alpha}(\tau) \\
ADF &= \inf_\tau ADF(\tau) \\
\end{align}
$$
This additional step adds an order of complexity, and critical values are no longer available in closed form and are difficult even to simulate.There is a great deal of theory on these tests, see for example, this comprehensive review by Pierre Perron.
In case you are interested in implemeting these tests, Bruce Hansen makes all the codes available on his website accompanying the paper, including newly added R codes. Note also that the Gregory and Hansen paper has a fully worked application of these tests.
