This sounds like you want to use the [McNemar][1] test which is intended for paired nominal (boolean) data.

There seems to be a Python implementation in the [statsmodel][2] library.
See also [this][3] question.


  [1]: https://en.wikipedia.org/wiki/McNemar%27s_test
  [2]: https://www.statsmodels.org/devel/generated/statsmodels.sandbox.stats.runs.mcnemar.html
  [3]: https://stackoverflow.com/questions/41539164/mcnemars-test-in-python-and-comparison-of-classification-machine-learning-model