A few suggestions/clarifications before directly addressing your questions.
First, a significant $\chi^2$ doesn't indicate poor fit; it indicates that you have rejected the null of perfect fit (i.e., $\sum$ = $S$). Kline (2015) aside, most SEM specialists (e.g., Brown, 2006; Finch & French, 2015; Hu & Bentler, 1995; Little, 2013; MacCallum & Austin, 2000; West et al. 2012) do not seem to recommend that you put a great deal of stock in this test, if only (amongst other reasons) because many find the null hypothesis a pretty ridiculous aspiration.
Secondly, be careful not to get sucked into specifying correlated error terms willy-nilly, just on account of the mod indexes. Many reviewers are savvy enough to see this for what it is: post-hoc fit chasing. If you have good a priori reason for specifying these, great, but if not, you might reconsider (and as you will see, your fit might not be as bad as you originally feared).
Now to your questions:
- No, not necessarily. For one, there are a plurality of recommended cutoffs for fit indexes (see Little, 2013, for some alternatives). Also, it's not as though your model fits horribly by all standards; sure, the RMSEA/SRMR aren't great, but your CFI looks quite strong. Finally (and most importantly), you should interpret your model fit indexes in context of your average standardized loading values for your model. McNeish et al. (2018) have a nice (and recent) review paper on the "reliability paradox", in which the Hu and Bentler (1999) cutoffs penalize models with more reliable indicators, and are too liberal for models with less reliable indicators. As the Hu and Bentler (1999) cutoffs were normed on average standardized loadings of ~.75, if your typical loading value is north of that, there's perhaps an argument to be made that your model fit is much better than it appears at first.
- CFI is a relative index of model fit, whereas RMSEA and SRMSR are absolute indexes of model fit. The former appraises the improvement in fit of the current model relative to a deliberately poor model (most often, a "null" model which specifies that all observed variables are uncorrelated with one another), whereas the latter appraises the appraises the fit of the current model against a perfect fitting model. With a great CFI and questionable RMSEA/SRMR (though be mindful of the reliability paradox, described above), I would wager that the null model in your case is fitting especially poorly, making your model like great by comparison, yet in an absolute sense it still might have a ways to go before it approaches "perfect" fitting.
- As my initial cautioning betrays, I actually wouldn't recommend that you refine the model much further, especially if you've already played around with mod indexes. What you might consider is abandoning a confirmatory specification of your model, and instead consider a framework like exploratory structural equation modelling (ESEM, Asparouhov & Muthén, 2009), where you can let the data guide your measurement model more explicitly, but still can model structural relations among latent (exploratory) variables.
Asparouhov, T., & Muthén, B. (2009). Exploratory structural equation modeling. Structural Equation Modeling, 16(3), 397-438.
Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York, NY: Guilford Press.
Finch, W. H., & French, B. F. (2015). Latent variable modeling with R. New York, NY: Routledge.
Hu, L., & Bentler, P. M. (1995). Evaluating model fit. In R. H. Hoyle (Ed.), Structural equation modeling: Concepts, issues, and applications (pp. 76-99). Thousand Oaks, CA: Sage.
Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1-55.
Kline, R. B. (2015). Principles and practice of structural equation modeling. New York, NY: Guilford Press.
Little, T. D. (2013). Longitudinal structural equation modelling. New York, NY: Guilford Press.
MacCallum, R. C., & Austin, J. T. (2000). Applications of structural equation modeling in psychological research. Annual Review of Psychology, 51(1), 201-226.
McNeish, D., An, J., & Hancock, G. R. (2018). The thorny relation between measurement quality and fit index cutoffs in latent variable models. Journal of Personality Assessment, 100(1), 43-52.
West. S. G., Taylor, A. B., & Wu, W. (2012). Model fit and model selection in structural equation modeling. In R. H. Hoyle (Ed.), Handbook of structural equation modeling (pp. 209-231). New York, NY: Guilford Press.