ARMA model vs ARIMA model

Suppose I use the same p and q, are the two models below theoretically the same?

1. ARMA(p,q) on 1st order differenced y
2. ARIMA(p,1,q)

I am aware of this post. But I would like to know are they conceptually the same model besides the minor differences in numerical optimization?

Also, if I run these two models, can I compare the two models say through RMSE based on the predicted values or through AIC/BIC of the model? Are they comparable as the y in the two models are not the same?

Conceptually, no; if we assume that "the future is much like the present, only longer" in the case of ARMA we assume the future refers to the observed sequence data and in the case of ARIMA at the successive changes between the observed sequence data. Big conceptual difference. To that effect, notice that the "error term" (the "$$\epsilon$$") has a much different meaning depending on the domain!
You are correct to suspect that because the response variable $$y$$ is different comparing AIC/BIC values is improper. That is because the likelihood used for these calculations targets different distributions.
Note that we can define an ARIMA(p,1,q) model as follows: $$\phi(L)\Delta y_t = c + \theta(L)\epsilon_t,$$ where the solutions of $$\phi(z) = 0$$ and of $$\theta(z)=0$$ lie outside the unit circle. As a consequence, $$y_t$$ admits a stationary and invertible $$ARMA(p,q)$$ representation after differencing once. To see this better, suppose that you have a series $$y_t$$ and you take the first difference. You would get a new series $$x_t \equiv \Delta y_t$$. At this point, considering an $$ARIMA(p,1,q)$$ model for $$y_t$$ or an $$ARMA(p,q)$$ for $$x_t$$ is exactly the same.