If you do that, then your "test" data is no longer entirely test data --- it is now partly training data. Indeed, the entire distinction between these two classes is that the training data is used to formulate hypotheses and models, and the test data is then used to make inferences in relation to those hypotheses under the model. For that reason, it is a good idea to take care to formulate a sufficiently general model at the training stage, noting that you might want to include higher-order effects that allow for more general functional forms than are exhibited in the training data.
In any case, if you use your "test" data to adjust the model, but then still treat it as test data, you are effectively using the data twice, first as training data, then as test data. The danger here is that the model choice might relate to the hypotheses of interest, in which case this method induces confirmation bias in your analysis, whereby the tests are biased in favour of acceptance of the hypotheses. If your modelling changes are unrelated to the hypotheses of interest then you might get away with it without imposing bias (or at least, without imposing too much bias), but it is very difficult to be sure. If you decide to do this, I would suggest that you at least conduct a sensitivity analysis that compares the conclusions from your preferred model with the conclusions from the original model. That way you can check to see if the change in the model affected any of the conclusions in relation to the hypotheses of interest (or if it just fit better in other respects).