Generally, as mentioned in the comments, there are different concepts of what a cluster is, and whether "there are clusters" or not depends on your cluster concept. I wrote something about this as Chapter "Clustering strategy and method selection" in the "Handbook of Cluster Analysis", see here:
Here's one possibility to decide that there is no evidence for clustering. This requires you to specify two things, which should be related to the cluster concept of interest:
A null model that models your idea of homogeneous data (for example a Gaussian distribution, or a uniform distribution, as used for the gap statistic). The null model should be chosen so that non-clustering features of your data are matched, for example by using the estimated mean and covariance matrix; see also https://link.springer.com/article/10.1007/s11222-015-9566-5
A statistic that measures the degree of clustering. For $k$-means this can well be the $k$-means objective function, but it might also be something else such as the Average Silhouette Width.
You can then generate data sets from the null model, cluster the data into two (or more) clusters, compute the statistic lots of times (2000, say), which simulates a distribution of the test statistic under the null model, and compare the value(s) that you get on your data to the null distribution. If your data don't give you significantly better values, there is no evidence for clustering.
By the way, this is the idea of the gap statistic (using a uniform null and the log objective function of $k$-means), which gives you a formal rule for choosing "no clustering" (i.e., number of clusters 1) as opposed to the Silhouette Width or the elbow rule.