Say if an A/B test proves to be insignificant but the test group shows that its doing better than the control group, what would be some of the reasons that the test is insignificant?
There are two kinds of significance: statistical significance and practical significance. Your question is asking about statistical significance (or lack of statistical significance).
In general, a lack of statistical significance says that with a given confidence level, the data we have and the statistical test we are performing cannot say that the effect we're testing is something that is unlikely to be due to some quirk of the sample of data that we have rather than something true about the overall population. (I tried to put it into laymen's terms, based on https://en.wikipedia.org/wiki/Statistical_significance, and hope I mostly succeeded.)
It's true that group A may have done better than B this time around. But if you chose A's and B's repeatedly, would it be unusual for A to turn out to not be better than B? Is it just this A and B, or would you expect the same kind of difference going forward? (Which is what you want to make your decision on: if you do A going forward can you confidently expect to do better than if you did B going forward instead?)
This cuts both ways. You can have statistically significant results -- you can be very certain there is a difference -- but the difference is so small that it's not practically significant. For example, your weight loss program could lose an average of 0.005 more ounces than your competitor's.
In your example, you can see a difference -- or perhaps you only think you see a difference, depending on how complex A and B and your test are -- but the numbers say it may be illusory.
It sounds like you're describing the difference between effect size and p-values.
Effect size is
a quantitative measure of the strength of a phenomenon
which is, "the test group show[ing] that its doing better than the control group".
So your statistical test may be "insignificant" because it is not significant from a statistical standpoint.