3
$\begingroup$

I am running a k-means clustering process in R and I'm comparing cluster partitions of different number of clusters: k = from 1 to 17.

Using the elbow-method, I have a minimum (of within-cluster SS) at k=5, but this value looks like an "outlier" to me, it is lower than subsequent values. Is there any explanation for this nonmonotonic behaviour?

My code in R is

kmeans.wss.k <- function(test, i){
  km <- kmeans(test, i, iter.max=20)
  return (km$tot.withinss)
}

for(i in 1:17){
print(i)
print(kmeans.wss.k(test,i))
}

# the data set includes some hundred thousand rows, so I can't post it

elbow-method

$\endgroup$
  • $\begingroup$ @ttnphns just added my R code. can't post the data set since it is far too large. Does this help? $\endgroup$ – rmuc8 Jun 30 '15 at 14:18
4
$\begingroup$

The kmeans algorithm is not deterministic, since the first step involves a random selection of the cluster centers. It could be that this particular run of the algorithm (k=5) got initialized near a bad local minimum. You may want to set nstart to a value larger than one, so that kmeans will try multiple initializations and return the best clustering, which will decrease the chances of getting stuck in a bad solution.

See the R documentation here: http://stat.ethz.ch/R-manual/R-patched/library/stats/html/kmeans.html

$\endgroup$
  • $\begingroup$ I had not heard of nstart. Thank you so much. I cannot believe I did this manually by running it with different set.seed values. $\endgroup$ – rmuc8 Jun 30 '15 at 15:11
2
$\begingroup$

I generated 5 quite well-separated random spherical normal clusters:

cluster_       v1       v2       v3       v4       v5       v6       v7       v8       v9      v10

       1  -.35889  -.80415  -.95355   .09492   .09878  -.57061   .04876   .32444   .43613  -.75848
       1  -.15922  -.33836  -.85863   .20295  -.02157  -.05508   .22093  -.20209   .12078  -.78535
       1  -.04343  -.57511  -.48917  -.11231  -.72780  -.26187   .28009   .12036   .74661 -1.28238
       1  -.63787  -.37796  -.85504   .19031  -.42068   .10327   .10764   .42621   .37841  -.31551
       1  -.26072  -.27328  -.23779  -.72944  -.00055   .02082  -.56056   .44621   .18630  -.44929
       1  -.69277  -.73115  -.98069  -.52607  -.19524   .93742   .22414   .02783  -.04566   .04284
       1  -.86744  -.24549  -.94066   .43189  -.31535  -.16167  -.23409   .37400   .29167  -.27242
       1  -.61638  -.77716 -1.01088   .41879   .25247   .06539   .93231   .17475   .06597  -.83623
       1  -.39131  -.71233  -.66181  -.11544  -.60630  -.24238   .74176  -.22127   .29012  -.48975
       1  -.70326  -.37406 -1.01975   .08320   .25287  -.54553   .37006   .56295  -.08655  -.42384
       1  -.58532  -.78090  -.52363   .34909  -.13363  -.03828  -.54699   .58049   .40722 -1.02372
       1  -.37289  -.30283  -.96715  -.25034  -.44174  -.14580   .11320   .40625  1.14946  -.71669
       1  -.07412 -1.06823  -.50983   .02912   .29590  -.57374   .21784   .96741   .39346  -.55247
       1  -.64557  -.28752 -1.06065   .91234   .33351   .11040  -.18679   .41887  1.31325  -.60888
       1  -.57587  -.18204  -.87064 -1.02938  -.32472  -.21606   .38594   .24866   .78516  -.47058
       1  -.18757  -.45480  -.85457   .28213   .09075   .01695   .09179  -.15911   .05814  -.64389
       1  -.48608  -.46857  -.94948  -.18636  -.45284  -.13831  -.04654   .20109   .23887  -.53225
       1  -.74734  -.08176  -.40716  -.08287   .12918   .15853   .74587   .27689   .54414  -.13492
       1  -.49310  -.66883  -.96552   .12861   .17556   .30066   .44258  -.05136   .20040  -.94728
       1  -.29509  -.04972  -.96719   .15146  -.15333  -.14490  -.22875   .21457   .49441  -.50094
       1  -.44203  -.65198 -1.31387   .14698  -.50402   .14900  -.22044   .63406   .66311  -.18015
       1   .36509  -.33656  -.44303   .29778   .19249  -.17983  -.20325  -.11391   .61558  -.71136
       1  -.58545  -.71832  -.58915  -.01282   .19542   .44213   .80000   .25524   .50858  -.15348
       1  -.61502  -.23686 -1.40380   .32189  -.28770   .16869   .06525  -.10767   .65105   .10628
       1  -.85519  -.56026  -.87019  -.15699  -.11682  -.26702   .42833   .61952   .54453  -.30528
       1  -.27144  -.47066  -.50293  -.04654   .20285  -.78603  -.05650   .04141   .43587  -.38720
       1  -.66833  -.85999 -1.39358  -.17907  -.35803   .26559  -.15205  -.06139   .07303  -.21759
       1  -.92891  -.83885  -.87182  -.57503   .34250  -.09976  -.49820   .15678   .56266  -.55564
       1  -.24621  -.57793  -.36508   .10734   .02193  -.03344   .42350   .44285   .15249  -.28804
       1  -.72920  -.46724 -1.04991  -.09380  -.59355  -.16053  -.35367  -.12637   .88971  -.36361
       1  -.95095  -.59221  -.75699  -.35541  -.45564  -.67606   .27492   .01158   .60059  -.36376
       1  -.42749  -.19381  -.11913   .66490   .53474  -.20897   .08913   .73044   .57282  -.71333
       1  -.35771  -.06046  -.61185  -.12414  -.33521  -.18176   .27938   .51311   .09970  -.78810
       2   .19264  -.39227  -.45144  -.34946  -.50718   .37317  -.68262 -1.21150  -.27218   .26079
       2  -.26762 -1.14943  -.24108  -.61609  -.53647   .54013   .37995  -.14258  -.92335   .20464
       2   .71585  -.57530  -.42672  -.33682  -.17193   .02049  -.11441 -1.01213  -.07658  -.01067
       2  -.18252  -.40495  -.72124  -.10062  -.86607   .93464   .37502  -.98797  -.28578   .25459
       2  -.27979  -.19103  -.36350  -.06228   .13197   .60294  -.38538  -.61523  -.70839   .49169
       2   .28744  -.05596  -.28775   .08955  -.44324   .26238   .68400  -.76522 -1.35614   .10266
       2  -.29865  -.34482   .19768   .05101  -.71033   .63751   .60821  -.87805  -.09203  -.32935
       2   .33705  -.29259  -.09520  -.20307  -.22723   .46284  -.15910  -.35052  -.79023   .15426
       2  -.17243  -.03626  -.08962   .04984  -.28323   .36873   .15831 -1.02778  -.43664   .04473
       2   .52032   .16039   .05790   .05482  -.58955   .57874   .15957  -.65148  -.60509  -.04410
       2  -.49298  -.14708  -.02757  -.29101  -.13378   .16864   .17596  -.90204  -.62725   .00444
       2   .34508  -.52929   .16544  -.23151  -.35914  -.39765   .06071  -.72253  -.26575  -.01780
       2   .18039  -.68884  -.17237  -.11827  -.76440   .88507   .31510  -.85731  -.55658  -.08526
       2  -.06160   .14602  -.10263 -1.22261  -.35885  -.08512  -.70837  -.49577  -.41909  -.21043
       2   .61226  -.48731   .17086   .01094  -.01556   .89763  -.09912  -.95948 -1.24026   .18679
       2   .27781  -.11220   .19625  -.80179 -1.05123   .34191  -.14444  -.92017  -.72394  -.38116
       2  -.13057  -.25444  -.29116  -.46638  -.45466  -.23273  -.35136  -.58928 -1.19391  -.21628
       2   .53544  -.07734   .18806  -.29863  -.35207   .16474  -.39829 -1.04920  -.04360   .27012
       2   .09011   .14510  -.13585  -.29632 -1.40790  -.54687  -.35850 -1.09756   .01308   .27056
       2  -.27727  -.35648   .18564  -.20406  -.62323   .42556   .04233  -.89799  -.95699   .07104
       2   .02676  -.00119  -.31353  -.47823  -.74560   .37957  -.37345  -.55019  -.72160   .78464
       2   .42959  -.37294  -.61238   .27562  -.57481   .36761  -.07886  -.19893  -.51787   .08761
       2   .84452 -1.02129  -.20080  -.14087  -.45586  -.14549  -.51673  -.95325  -.33927   .32113
       2   .45584  -.97896  -.29851   .26187  -.00994   .23504   .39993  -.47371 -1.43147   .22279
       2   .45992   .25763  -.50511  -.24222  -.47895  -.56162  -.09741  -.39730  -.77720   .34671
       2   .67029  -.80599  -.21568  -.20057  -.57457   .46333   .42951 -1.46737  -.87203   .14155
       2   .28548   .14793   .20718  -.62599  -.81032   .50961   .14414  -.87512 -1.33294  -.29246
       2   .02684  -.74806   .03674   .35342  -.02703   .47246   .00974  -.96049  -.61683  -.15511
       2   .06397  -.67032   .20367  -.71854 -1.07909   .13888  -.15452  -.93031  -.84782   .05661
       2   .08814  -.41217  -.16592  -.78108  -.53232   .38670  -.08234 -1.20331  -.70393   .09167
       2   .47916   .57869   .04185   .05530  -.11371   .06468  -.10352  -.58962  -.47385  -.23243
       2   .30212  -.30889   .02052  -.00621   .06990   .56321  -.41854  -.63496  -.72044   .01441
       2   .28807   .14702  -.11912  -.71629  -.37887   .05496   .08769  -.93546  -.65293  -.30587
       2  -.00337  -.23993   .15920  -.39460  -.17969   .39506  -.26512  -.58096  -.53665  -.08089
       2  -.24791   .35754  -.10902   .14933  -.70239   .45544  -.23488  -.36102  -.53181   .06614
       2  -.06024  -.18187   .14727  -.37599   .15671   .49404  -.09129  -.44988  -.08485   .31208
       2   .20165  -.20496  -.16680  -.27965  -.49118   .50500  -.13241  -.55039   .02795  -.52744
       2   .65964  -.36595  -.71629  -.52986  -.38274   .19508  -.00661  -.17767  -.16829   .44395
       2   .39324  -.06167  -.15007   .26749  -.12583   .21957   .24441 -1.12913  -.34984   .12255
       2   .34302  -.74048  -.23877  -.40451  -.28469   .59656  -.09161  -.98368  -.26619   .48246
       2  -.36048  -.66483  -.06230  -.61534  -.03873   .24258   .20101 -1.02728  -.51428   .92295
       2  -.38574  -.25897  -.06990  -.02799  -.08942   .08306   .33846  -.44243  -.67854  -.29403
       2  -.36852  -.44039  -.27738  -.42919  -.13750   .31308   .14585  -.56002  -.40077   .15878
       2   .21363  -.09843   .35279  -.28396  -.39179  -.38450  -.19170 -1.10894  -.79801   .09346
       2   .26392   .02051  -.03783  -.37110  -.57680   .39785  -.58526 -1.15637  -.86041   .48684
       3  -.24395   .16123  -.52569  -.54756  -.46173   .78313  -.32469   .86162   .45817   .18950
       3  -.45149   .27338   .33748  -.23151  -.40367  -.01772  -.28303   .32204  -.46198   .21000
       3   .09345   .71764   .17051  -.00560  -.90011   .16755  -.24581   .74744  -.31104  -.19755
       3   .00171   .78304  -.53418   .24089  -.04693   .76982  -.09617   .95082  -.72464   .00413
       3   .14384   .67336   .00242  -.01307   .17794   .38635 -1.37637  1.09934   .43910  -.59356
       3   .45663   .46030  -.71005  -.42481   .27294   .25727   .38373  1.27638   .03208  -.38087
       3  -.35099   .57253   .37563  -.46467  -.15921   .08879 -1.06144   .69125  -.17132  -.42077
       3  -.50472   .69089  -.06146  -.20484  -.67984   .49757  -.56882   .80419  -.75014  -.31072
       3  -.17699  1.50970   .12051  -.59455  -.00184   .50617   .26104   .63712  -.63130  -.31656
       3   .19155   .54132   .12707  -.42273   .08332   .14384  -.59585   .29190  -.47406  -.10795
       3   .59290   .17106   .08535  -.01423  -.54075   .94399  -.60894   .43943  -.38056   .14853
       3  -.17509   .79728  -.06453  -.95079  -.23326   .51459  -.47951   .63232  -.59717  -.09292
       3  -.02898   .42072  -.35931  -.68248  -.30044   .33594  -.67137   .19582  -.33122   .05719
       3   .06747   .89734   .80501   .10828  -.34636   .63065  -.28847   .71106  -.17164  -.47439
       3   .22291   .30021   .42112   .05373  -.46780   .39958  -.14513   .19257  -.60642  -.51323
       3  -.67178   .12969  -.35625  -.06941  -.90086  -.09086  -.70433   .08350  -.21895  -.03402
       3   .06226  1.30616   .06839  -.02109   .25937   .37931  -.01058   .64054  -.04348  -.43756
       3   .34702  1.32103  -.13883  -.55438  -.12072   .11828  -.70851   .50123   .00989  -.39402
       3  -.17608   .38942   .09153   .23233  -.30488   .69906  -.07760  1.15005  -.60196  -.72459
       3  -.03706   .20870  -.20816   .14613   .02804   .00570   .48920   .47652  -.44141  -.56459
       3   .11554   .72674  -.28102   .61072  -.54430   .74248  -.65831  1.08916  -.52031  -.46012
       3  -.08616   .76170   .60656  -.20260  -.61624   .95084  -.61945  -.53094  -.48601  -.12913
       3  -.23377   .28812  -.00465  -.63814  -.54961   .60212  -.73909   .57522  -.09417  -.83123
       3   .04647   .67864   .24538  -.45371  -.75121   .46089  -.12164   .24679  -.96037  -.51968
       3   .43037  1.19863   .03038   .01944  -.53914   .57373  -.46845  -.37273   .01472  -.35485
       3  -.05359   .58196   .14807  -.16891  -.73624   .49092   .17588   .94871   .67527  -.51166
       3  -.37767   .59255   .10369 -1.34807  -.34045   .17858  -.11296   .82858  -.07735  -.37090
       3   .06322   .08763   .16581  -.70023  -.07563   .45287  -.00975   .63799  -.28614  -.33280
       3  -.51456   .70515   .77411  -.33698  -.30368   .36648  -.31280   .07921  -.25082  -.32052
       3  1.14969   .93388   .02509  -.12116  -.07666   .54490  -.09371   .97084  -.30443   .08455
       3   .08126   .42957   .18818  -.57343  -.43837   .52381  -.14258   .81012  -.18564  -.39681
       3   .00904   .35280   .34457  -.32577  -.20415  1.25479  -.44810   .85328  -.34538  -.56894
       3   .57401   .98226   .24527  -.50222   .09948   .73297  -.41718  1.05607  -.78782  -.91479
       3   .14213   .83693  -.14598   .11372   .11413   .99507  -.66617   .47991  -.51509  -.75660
       3  -.21072   .56046  -.12287  -.09227  -.07915   .65564  -.68789  1.04063  -.82856  -.25795
       3   .26696   .46613   .37524  -.30427  -.23610   .84144  -.09927  1.10323  -.29640  -.32335
       3  -.35833   .50586   .15598   .27042  -.29747   .84245   .10441   .82107  -.18041  -.32057
       3   .38234   .29171  -.07923  -.19716   .05640   .34179  -.69348   .69792   .11555   .00594
       3   .39440  1.15490  -.50808  -.38042   .24665   .41070  -.13404   .86015  -.67955   .10854
       3   .01013   .05924  -.21301   .25495  -.58790   .72584  -.12632   .82050  -.26645  -.20871
       4   .12481   .64539  -.03975   .64673  -.40431   .13063  -.52849  -.07650  -.11618  1.24660
       4  -.01452   .00504  -.13760   .66609  1.09446  -.58416  -.53981   .63190  -.96223  1.02409
       4   .91232   .48534  -.29704   .18219   .36045  -.79304   .22392   .42187  -.01970   .47275
       4   .49115   .46320  -.69861   .69819   .18075   .00179   .49793   .23133  -.56602   .67727
       4   .43851   .75341  -.36828   .22170   .55148  -.39889   .17800  -.01949  -.27295   .17505
       4   .43239   .08964   .15651   .60433   .58125  -.51438  -.07613   .15524  -.36421   .79300
       4   .34400   .25761  -.77976   .16498   .59756  -.34697  -.14232   .04085  -.38930   .48834
       4  -.04106   .79610 -1.11204   .44975   .78673  -.06170  -.35702   .36731  -.54020   .49166
       4  -.00527   .50502  -.07809  -.18947   .97660  -.60149  -.18532   .15491   .30075   .31105
       4   .18186   .55742  -.56654   .47627   .60055  -.69449   .27283   .23622  -.34590   .53413
       4  -.50832   .28511  -.04698   .17425   .76304  -.20335  -.03809  -.01743  -.03948   .98610
       4   .78833  -.03687  -.62573   .28408   .39957  -.34038   .13137   .29892  -.08960   .52031
       4   .95380   .61980  -.52965   .72138   .92326  -.77683  -.34070  -.15212   .14155   .73830
       4   .35160   .28312  -.83647   .09864   .83778  -.88948  -.49741   .09432   .34290   .77631
       4   .27600   .92787  -.08412  -.20978   .38395   .05653  -.61637   .11731   .33753  -.08716
       4   .58476   .62382  -.11519   .50597   .60866  -.47202  -.26130   .21178  -.38129   .77234
       4  -.07336  -.40171  -.54984   .17316   .69194  -.32094  -.80006   .54560  -.48985   .46294
       4   .25167   .19768  -.56227   .29683   .22327   .10291  -.49781   .57931   .03583   .29051
       4   .03777  -.06726  -.28299   .30957   .61156   .24576  -.56313   .70846  -.37586   .46174
       4  -.03174   .48932   .03035   .23212   .78673   .22815  -.41416   .00390  -.02809   .25024
       4   .47553   .15852  -.52593   .28778   .51391   .03696  -.40845   .06569  -.13843   .78394
       4   .66051  -.24317  -.38019   .36846   .63298  -.58473   .06140   .20684   .10806   .90685
       4   .78153   .24951  -.04040  1.19602   .19623  -.65166   .22046   .00106   .38641   .83804
       4   .43238   .61964  -.38654   .03870   .56794  -.76404   .49067   .22001  -.54198   .40200
       4   .81075   .61281  -.60007   .81533   .65307  -.55407   .08571  -.54866  -.51458   .71056
       4  -.32060  -.22371  -.59960   .33757   .76813  -.35013   .27732   .19311  -.52868   .27842
       4   .43247   .38941   .05445  -.22539   .45748  -.91286  -.26040   .03905  -.29523   .56303
       4   .60153   .16959 -1.09957   .82787   .38469  -.41677  -.55122   .29787  -.01731   .85011
       4   .09208  -.04104  -.16329  -.32714   .68905  -.16825  -.32004  -.06744  -.34247   .42238
       4   .53446  -.02837  -.41285  -.05115   .93666   .09275   .28974  -.43819  -.02035   .78239
       4   .52346   .16028  -.30639   .80242   .64151  -.67335  -.27878   .03999  -.17939   .52376
       4  -.07272   .86453  -.04191   .51539  1.30292  -.17256   .25074   .13524  -.89523  -.12336
       4   .26381   .22033  -.48637   .00789   .07699  -.20436   .30045  -.55430   .28213   .56632
       4  -.38775   .18806  -.35541   .16925  1.53684   .20027  -.32236   .29665   .36345  1.10464
       4  -.56245   .73046   .07529  -.01363  1.00474  -.58000  -.43443   .49613   .23503   .49014
       4   .19397   .60054  -.80352   .06133   .38149  -.01459  -.46508  -.22733  -.02982   .80440
       4  -.19880   .54340  -.59345   .37518   .89532  -.57358  -.35951   .04535   .60423   .62265
       4  -.11259   .20349  -.61756   .71470   .47077  -.64705   .29557  -.33549   .46786  1.02565
       5   .94221  -.76616   .93816   .20651  -.22434  -.64455   .21913   .08471  1.03554   .12535
       5   .68932  -.03953  1.31733   .46692   .39142   .55476  -.14973   .06212   .77552   .00833
       5  -.78798  -.43622   .70848   .64921  -.12265  -.61095   .16492   .64428   .87688   .66544
       5   .10115  -.16724  1.05262   .97713   .66306  -.74898   .12452  -.23221   .60067   .09814
       5   .23656  -.37081   .50543  -.30092   .16232  -.40432   .05915  -.08133  -.04027  -.20363
       5  -.29437  -.23215  1.13539   .00675   .47423  -.22462   .29859  -.38251   .22398   .11379
       5   .20369  -.25498  1.06575   .62940  -.25818  -.47180   .03358  -.38966   .16060  -.27854
       5  -.01902  -.15011   .97446   .66675   .21333  -.68202   .58280  -.23840   .65855  -.13566
       5  -.59616  -.34664   .48634   .25446   .37241  -.54443   .08664  -.19193   .92828  -.40835
       5   .37222   .07618  1.12760  -.50001   .30198   .19466   .07131  -.34506   .45512   .36074
       5   .16370   .01577  1.04550   .00619   .28735  -.66035   .61140   .64152   .79330  -.01340
       5  -.71456  -.92065   .76481  -.17659   .04013  -.64073  -.03873  -.66962   .53862  -.04873
       5  -.14563  -.38052  1.13991   .14477  -.28422  -.80953  -.02142  -.15500   .48110   .02102
       5  -.41581  -.09893   .95991  -.69318   .11668  -.42237  -.32851  -.14550   .58129  -.05483
       5  -.55230  -.36544   .83642   .48609  -.08735  -.63161   .65751   .18261   .29338   .15508
       5  -.08841  -.43305  1.16937   .08712   .54940  -.66578   .47746  -.25139   .51399  -.28219
       5  -.70661  -.35850   .42385  -.04263  -.15989  -.17255  -.20838   .46533  1.10884  -.23194
       5  -.31880  -.33642   .46561  -.07951  -.30750  -.15734   .42598  -.39341   .63181  -.15968
       5  -.02104   .01231   .89536  -.09382   .21141  -.29203   .31900  -.35373   .52825  -.00364
       5   .52882  -.19750   .57151   .18083  -.21128  -.39752  -.18213  -.00997   .54937   .20951
       5   .18084  -.35750   .98115   .10735  -.15109   .03623   .53613  -.63918   .13051   .41700
       5   .13607  -.39431  1.11954   .10425  1.13318  -.38586  -.06870  -.16331  -.02559  -.21759
       5  -.24449  -.22553   .72800   .77752  -.16310  -.39704   .20342   .09200   .12099  -.40769
       5   .15298 -1.05720   .27149   .21031  -.46929   .23929   .36514  -.08841   .31814   .03095
       5   .23835   .22692   .68748   .19415   .30307  -.00560   .13476  -.27850   .37748  -.39022
       5  -.46264  -.28041  1.19408   .00246   .92476  -.55617   .28429  -.12166   .52160   .52571
       5   .44002  -.41763  1.00519   .37208  -.21578  -.75036   .45686   .03575   .90486   .36819
       5  -.21521  -.24764   .77803   .16955   .49566  -.31356  -.16268  -.12568   .94447  -.14197
       5  -.01372  -.49895  1.15430   .34893  -.09728  -.00777   .56743  -.10422   .80134   .52607
       5  -.01684  -.41250   .64332   .38734   .31766  -.40441   .41563   .38074   .37687   .13039
       5  -.34362  -.80602   .89250   .52391   .84036  -.71583   .27336  -.48401   .83208   .40367
       5  -.29687  -.81651  1.09081   .72333   .12215  -.18851   .62242  -.34748  1.02239   .26486
       5   .15349   .04149  1.01170  -.32899   .18664  -.09604   .25532  -.21718   .31506  -.15446
       5   .29247 -1.06331   .95709   .59674   .02793  -.06086  -.15468   .42929   .25093   .15189
       5   .32936   .30622  1.19245   .68224   .15637  -.55656   .76485  -.97028   .69600   .15906
       5  -.23501  -.50298   .68681   .49305   .00555  -.31013   .62912  -.94264   .82368   .23385
       5  -.57203   .00568   .78144   .39737  -.37745   .11250  -.27170   .25716   .31007  -.15374
       5  -.07426  -.44952  1.05923   .35666  -.47327  -.61224   .19806  -.10865   .38987  -.23939
       5  -.37858   .07292   .63529   .37387  -.07960  -.41198   .31239  -.52773   .77381  -.47123
       5  -.09884  -.28480   .62785   .06343  -.45201 -1.05897   .35904   .29022   .41229   .20190
       5   .46010  -.69470  1.57011   .59279   .53526  -.09801   .22513  -.02678  1.05208   .53716
       5   .09403  -.18182  1.24049   .48869   .43755   .17045   .30166  -.04614   .60966   .06173
       5  -.01407  -.19142   .98127   .67015   .11047   .15014  -.54993   .13654   .09494  -.22172

The SSwithin (pooled within-cluster sum of squares of deviations from the cluster centres) is 217.342.

I performed a series of K-means cluster analyses in SPSS on these data. I used default settings to seed initial cluster centres (In SPSS, that is k-utmost points in the data cloud, found by a special procedure) each time. The 14 analyses formed k=2 to k=15 clusters. SSwithin for the obtained cluster solutions were:

k
 2   430.350 
 3   352.252 
 4   281.481 
 5   217.012 
 6   210.398 
 7   203.786 
 8   197.991 
 9   194.528 
10   190.398 
11   182.382 
12   183.242 
13   179.123 
14   179.375 
15   171.383

and on the plot

enter image description here

Clearly, for such clear clusters as ours had been generated there is a nice elbow, and at k=5 clusters (the clusters of that solution are very close to be what we generated).

But besides that we observe two minor "inversions" of SSwithin: for k=12 it is higher than for k=11, and for k=14 it is higher than for k=13.

Final centres in a K-means clustering solution are multivariate means (centroids) and as such they minimize SSwithin to the limit attainable with that specific assignment of points to k clusters. K-means algorithm tries to find the assignment which will yield the minimal SSw but does not guarantee to find it, the global optimum. The global optimum at k=i is lower SSw than at k=i-1 for any i but it may happen that the procedure succeeds to find the global (or near global) optimum at i-1 while badly fails to find it at i; then SSw observed in the obtained cluster solution may be lower at i-1 than at i.

Good initial cluster centres fed to the program play the crucial role. If those centres are already close to the real centres of the clusters hidden in the data or the data is such that they can easily "slide down" to them during the iterations then the solution will be about the global optimum: SSw utmost minimized. It is possible that in your example you somehow had had very good initial centres for k=5 and rather poor initial centres for most of other solutions. But SPSS's default gimmick to set initial centres to the utmost juts of the data cloud often provide easy "slide" to final centres which are not far from optimal. Hence the inversions of the SSw I've found in my example are slight, while yours was pronounced.

SSwithin elbow method isn't particularly reliable or sensitive as a criterion to select the "best" number of clusters. Its various standardized versions which compare SSw relative SSbetween or SStotal are far better. Two most well-known are Calinski-Harabasz and Davies-Bouldin clustering criterions. Shown applied to my example:

enter image description here

They unequivocally indicate 5-cluster solution. The two criterions are rather similar and are ideally suited to validate spherical clusters having compact centre (exactly the shape that K-means is inclined to). Calinski-Harabasz is somewhat in favour with similar-sized (in respect to the number of objects) clusters, while Davies-Bouldin a bit tends to prefer equally-distanced from each other clusters.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.