Q1: Yes, using a basis expansion such as a B-spline basis is acceptable in a linear mixed model. The model is really a linear fit to the response over a set of derived variables rather than the actual data, in much the same way as you created two new variables from iv, its square and cube, and fitted a model for those instead of just iv itself.

Q2: That said, these are two very different model. The cubic polynomial is a global model, the best fitting cubic polynomial over the range of the data. The model using the B-spline basis is a piecewise polynomial of some degree $k$, say $k = 3$ for cubic splines. The individual pieces are fitted over portions of the data, the boundaries between portions are interior knots. Some constraints on the basis functions force the joins at the knots to be smooth. The key point however is that the B-spline model fits polynomials over local sections of the covariate iv whereas the polynomial model fits a global cub polynomial model over the entire range of iv.

In both cases, the fitted curve should be smooth. What you may be seeing is the effect of irregular sampling over the range of iv. The fitted values will be for the observed values of iv and then a line is drawn to join up these fitted values. If there are varying gaps in the (ordered) iv data, this can show up as low smoothness. If so, and what I tend to do by default regardless, is to predict from the model for $N$ new iv points spaced regularly over the range of iv. You can make $N$ as large as you need to achieve a sufficiently smooth representation of the fitted model.

Q1: Yes, using a basis expansion such as a B-spline basis is acceptable in a linear mixed model. The model is really a linear fit to the response over a set of derived variables rather than the actual data, in much the same way as you created two new variables from iv, its square and cube, and fitted a model for those instead of just iv itself.

Q2: That said, these are two very different models. The cubic polynomial is a global model, the best fitting cubic polynomial over the range of the data. The model using the B-spline basis is a piecewise polynomial of some degree $k$, say $k = 3$ for cubic splines. The individual pieces are fitted over portions of the data, the boundaries between portions are interior knots. Some constraints on the basis functions force the joins at the knots to be smooth. The key point however is that the B-spline model fits polynomials over local sections of the covariate iv whereas the polynomial model fits a global cub polynomial model over the entire range of iv.

In both cases, the fitted curve should be smooth. What you may be seeing is the effect of irregular sampling over the range of iv. The fitted values will be for the observed values of iv and then a line is drawn to join up these fitted values. If there are varying gaps in the (ordered) iv data, this can show up as low smoothness. If so, and what I tend to do by default regardless, is to predict from the model for $N$ new iv points spaced regularly over the range of iv. You can make $N$ as large as you need to achieve a sufficiently smooth representation of the fitted model.