It's not just the RMSEA--other conventional indexes of model fit (CFI, TLI, SRMR) all point to this being a poor fitting model (see Hu & Bentler, 1999, for a description of various model fit indexes).
Before moving to adjusting your measurement model, I think you need to change your estimator. You're using a typical maximum likelihood estimator (the lavaan default), with categorical (dichotomous) observed indicators; that's going to violate the assumptions of the ML estimator, and likely hurt your fit in a nontrivial way. You can learn more about the suite of estimators (normal and robust variants) that lavaan can use here. I would suggest you consider you consider one like DWLS, that is pretty categorical-friendly (see Rhemtulla et al., 2012, for a discussion of estimators for continuous vs. categorial data).
Failing that, unless you have a specific competitor model in mind, I'd say you will be squarely in exploratory data mode. And in that case, you might consider using exploratory factor analysis (with an estimator for categorial observations), so that the data can guide you to a more appropriate (and complex) factor solution. One factors, as you say, is about as simple as you can get, so it's likely a more complicated solution is needed (as more factors = better fit).
References
Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1-55.
Rhemtulla, M., Brosseau-Liard, P. É., & Savalei, V. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17, 354-373.