I'm trying to estimate parameters of ARMA. Ljung-Box statistic reveals no serial correlation in residuals. But one coefficient is statistically insignificant. When I remove the variable corresponding to this coefficient I get a model that is worse than the former one(AIC and SC choose the first model, Ljung-Box statistic reveal serial correlation in residuals for the second(reduced) model). Should I choose the first model with statistically insignificant coefficient? And one more question: if I use assumption about t-distribution of errors then how to check whether a coefficient equals zero(i.e. which statistic should I use and what distribution does it have?)
The fact that one of the ARMA coefficients is not significant does not necessarily mean that the coefficient doesn't play any role in the model. It is the entire polynomial of the model what determines the cycles that are captured by the model. Changing or removing one coefficient will in general change the dynamics of the underlying cycles that characterize the ARMA model and, hence, relevant cycles may be lost. See for example my answer to this post where a non-significant coefficient was actually found to be relevant.
I didn't look at the data but I think I can safely say that you should choose the model with the lowest AIC and no autocorrelation in the residuals, rather than the model with significant coefficients but autocorrelated residuals.
As regards your second question, you may take as a rough guide a $\sim 95\%$ confidence interval by taking two times the standard error of the parameter estimates and see if zero lies within it, but as said before, this may not give you enough conclusive information.