Your data seem to exhibit heteroskedasticity (non-constant variance) and your model does not account for this through either explicitly modeling the variance or weighting of observations. That is a problem. I would consult your favorite statistics book on weighted least squares and variance-stabilizing transformations.
I don't have much to offer in terms of methods, I think the ones presented here (esp inverse variance weighted approaches) are good ones. What I can add is a small simulation study to prove that under the assumption of Gaussian errors in the regression, this process has good enough coverage
I imagine the 95% confidence intervals come from some assumptions on normality of data. Otherwise, please state how you got these CI. This implies you believe the mean of each slope (viewed as a RV) is $m_i$ with some variance $\sigma_i$
In this case you can average the slopes as you did and get the new variance of the averaged estimator (assuming ...