Rather than scaling the coefficient, I think you should try to explain to whom it applies. An instrumental variable does not identify an average treatment effect for everyone. Instead, you get a Local Average Treatment Effect (LATE), which is the average treatment effect for those whose treatment status is affected by the instrument. In your setting, this is a very small group and it is possible that this group has a very high test score without treatment and a much larger treatment effect than other people. So the coefficient might not be implausibly large when interpreted correctly. Even if you will not be able to generalise the effect size, perhaps it will be possible to argue (based on your understanding of the context) that the effect should be negative (and large?) if other groups were exposed to treatment. Of course, there is also the possibility that your instrument is invalid.
Regarding the problem that your regression predicts a negative test score for the treated: unlike OLS, where the intercept will give you the mean outcome for the untreated group, the intercept from an IV regression is not generally a relevant control mean against which to compare the estimated treatment effect.
To see why, let $Y_i$, $X_i$, and $Z_i$ denote the observed outcome, treatment and instrument. Further, let $Y_{1i}$ and $Y_{0i}$ denote the potential outcomes for $i$ with and without treatment and define the treatment effect for $i$ as $\beta_i = Y_{1i}-Y_{0i}$. Similarly, let $X_{1i}$ and $X_{0i}$ be potential treatment status for $i$ with the instrument switched on and off, and $\theta_i = X_{1i}-X_{0i}$ the first-stage effect for $i$.
$$ Y_i = Y_{0i} + \beta_i X_i$$
$$ X_i = X_{0i} + \theta_i Z_i$$
For a binary instrument and treatment, there are four types of individuals: always takers ($A_i$; $X_{0i} = 1$, $\theta_i = 0$), never takers ($N_i$; $X_{0i} = 0$, $\theta_i = 0$), compliers ($C_i$; $X_{0i} = 0$, $\theta_i = 1$), and defiers ($D_i$; $X_{0i} = 1$, $\theta_i = -1$). For a randomly assigned instrument that effects the outcome only through the treatment and assuming no defiers (i.e., $\theta_i$ is either $0$ or $1$), it can be shown that $\hat{\beta}_{IV}$ will identify
$$ E[\beta_i|C_i=1], $$
which is a LATE; the average treatment effect on the compliers. In a simple linear regression, we would get the intercept as
$$ \hat{\alpha}_{OLS} = \bar{Y} - \hat{\beta}_{OLS}\bar{X} $$
With 2SLS, we instead have
$$ \hat{\alpha}_{IV} = \bar{Y} - \hat{\beta}_{IV}(\hat{\mu}_{OLS} + \hat{\theta}_{OLS} \bar{Z} ), $$
where $\hat{\mu}_{OLS}$ is the intercept from the first-stage regression. If we define $p_a = \text{Pr}(A_i = 1)$, $p_n = \text{Pr}(N_i = 1)$, $p_c = \text{Pr}(C_i = 1)$, and $p_z = \text{Pr}(Z_i = 1)$ we can write a theoretical version of this equation as
$$ \underbrace{E[Y_i]}_{\bar{Y}} - \underbrace{E[\beta_i|C_i=1]}_{\hat{\beta}_{IV}}(\underbrace{p_a}_{\hat{\mu}_{OLS}} + \underbrace{p_c}_{\hat{\theta}_{OLS}} \times \underbrace{p_z}_{\bar{Z}} ), $$
To have a straightforward interpretation in relation to $\hat{\beta}_{IV}$, we should like the intercept to identify
$$ E[Y_{0i}|C_i=1] $$
Substituting for $E[Y_i]$, we can rewrite this in terms of potenital outcomes to see that we instead have
$$ E[Y_{0i}|A_i=1]p_a + E[Y_{0i}|C_i=1]p_c + E[Y_{0i}|N_i=1]p_n + (E[\beta_i|A_i=1] - E[\beta_i|C_i=1])p_a $$
In your case, this should be close to the average potential outcome without treatment for the never-takers, becasue your sample is predominantly made up of this group.
Regarding you approach to dealing with non-linearities (which I have seen discussed here [section 19.3.3], for future reference), we should not expect this to make any difference when the instrument is binary because, by defintion, the relationship between treatment and instrument must be linear.
