Why don't I have metric measurement invariance even with very small differences between factor loadings across groups?

Background: I'm running a multi-group model in lavaan. It has two groups: One clinical sample (n = 160) and one healthy control sample (n = 248) that I chose to be comparable regarding age, gender and SES. I'm looking at their scores in several intelligence tests, which form three latent intelligence domains/factors. To check for measurement invariance, I first ran my model without any constraints and got good fit. Then I constrained the factor loadings of all subtests to be equal across the two groups. I was not surprised when the constrained model turned out to have significantly worse fit - it makes sense to me that intelligence subtests in a clinical group could have different loadings on intelligence factors compared to a healthy group.

The thing that puzzled me: I wanted to find out which of my indicator variables' loadings differed so much between groups that I didn't get metric invariance. Some of the loadings were very similar between the groups (differences in standardized factor loadings of around .02 in the cases of lns or dsf, for example), which is why I expected only some of them to make the difference. But even after setting all of them free between the groups (with the group.partial argument in lavaan), except one that differed at .02 between the groups, the constrained model was still significantly different according to a chi-square difference test.

My question: Is the reason for the nonexistent metric invariance really that the small difference in factor loadings (.02) between the two groups is significant? Or am I missing some obvious reasons for the metric measurement variance between my two groups? Reasons I thought of were the different group sizes, or the different variances of the indicator variables between the groups, but my stat knowledge is too tiny to explain why they (or something else) would lead to this. Here's the relevant output for the multigroup model with no constraints:

Group 1:
Latent Variables:
Estimate  Std.Err  z-value  P(>|z|)   Std.lv  Std.all
PS =~
cd               12.620    0.994   12.693    0.000   12.620    0.890
ss                4.789    0.540    8.865    0.000    4.789    0.634
WM =~
lns               3.686    0.258   14.302    0.000    3.686    0.860
dsf               1.295    0.118   10.932    0.000    1.295    0.662
dsb               1.731    0.169   10.240    0.000    1.731    0.709
Gf =~
mr                4.696    0.208   22.621    0.000    4.696    1.000

Group 2:
Latent Variables:
Estimate  Std.Err  z-value  P(>|z|)   Std.lv  Std.all
PS =~
cd               11.031    0.918   12.017    0.000   11.031    0.862
ss                6.477    0.487   13.296    0.000    6.477    0.917
WM =~
lns               3.923    0.255   15.370    0.000    3.923    0.844
dsf               1.218    0.171    7.125    0.000    1.218    0.654
dsb               1.472    0.146   10.066    0.000    1.472    0.817
Gf =~
mr                6.018    0.277   21.718    0.000    6.018    1.000


Note: I know my model isn't ideal (only one indicator for Gf, sample sizes are really small for multigroup models, etc...). I'm mainly asking this question to get a better understanding of how measurement invariance testing works.

Note II: I'm aware that Chi square difference tests can be biased in larger samples, but I don't think this is the case here as my n isn't really large (?)