2

The one possible approach could be a relatively new trimap method, based on triplets (in contrast with usual pair-wise distances). It's explicitly optimize to keep the original form (in contrast to t-sne), while making better clusters then PCA. https://github.com/eamid/trimap


2

Since I'm looking into the gap statistics just now, I'll try to answer this old question. $\newcommand{\Gap}{{\rm Gap}}$ Once you have partitioned your datapoints into $k$ clusters using e.g., k-means, you want some metric that describes how compact the clusters are. One way of doing this is to compute all the pairwise distances between points within a ...


2

The answer is to use $X You find directions which explain the most variation in the data. You choose a few of them (eg. enough to explain 85% of the variance) and you project your data onto those directions. What you end up with is a lower dimensional dataset which captures much of the variance in the original dataset. This is what you want to use. From ...


2

The best thing to do here is really to get labeled data. It will do far more for you than any attempt to solve this problem. Especially in something as life-and-death crucial as COVID, deploying a poor model is unethical. What you’ve suggested is to take an unweighted average and threshold it: $\mathrm{COVID?} = \left[ 4 \geq \sum_{i=1}^6 f_i\right]$, using ...


2

In some applications the measurement units of different variables are the same, and the meaning is actually pretty much the same. Normalisation is bad if in such a situation the variance is actually informative in the sense that a larger variance implies that the variable is more relevant. One such example that I have come across is voltage traces on ...


2

Succinctly, Logistic regression is a supervised learning task. If you have inputs and a binary outcome then you can use logistic regression. For example, let's say I want to know the probability of a plant dying using the mass of herbicide I use as my predictor. The outcome is did the plant die (yes or no, hence binary) and the predictor would be the ...


1

The solution that I adopted in practice was to compare the correlation matrices for the two datasets (calculated using pearson/spearma/kendall or any other correlation coefficient of choice. It is possible to formulate this problems as rigorously testing the hypothesis that the two correlation matrices are identical (see here and here). But it is also ...


1

Here Samy Bengio explains CCA which is what you may try first. It can give you the info how similar are two matrices residing in some spaces with one dimension in common. CCA -- canonical correlation in the experimental context is to take two sets of variables and see what is common among the two sets. So it is general enough I would say. R has the standard ...


1

It actually creates a single cluster, which is identified with $0$. Try a different parameter for cutoff_scale like $0.01$. This will create the following clusters: +----+-----------+--------+----------+ | | cluster | user | values | |----+-----------+--------+----------| | 0 | 0 | 0 | 0.851429 | | 1 | 0 | 1 | 0.855663 | |...


1

K-means is a clustering (unsupervised) algorithm, but KNN is a classification/regression algorithm (supervised). KNN doesn't cluster data, it looks at the neighbouring points and assigns the target variable according to them. In that, you need specify a K as well.


1

The following is a proof that the objective function in the non-centered case is exactly equal to the objective function in the centered case. Recall that K-means finds clusterings by minimizing $\sum_{k=1}^K \sum_{y\in X_k}\|y-\mu_k\|_2^2$ over clusterings $\{X_k\}_{k=1}^K$, where $\mu_k$ is the centroid of $X_k$. Let $\{x_i\}_{i=1}^n\subset\mathbb{R}^D$ be ...


1

Forcing unit variances might be better because the standard k-means approximates spherical Gaussian distributions around centroids and, as a consequence, might favor inflated features like you said. But this is not effective in all cases since there are also other factors implied: deeply correlated features, scaling is global over all clusters, and so on. ...


1

There are a number of measures to do this, i.e., to compare two clusterings, one of which may be the true one if you have it. The probably most popular one is the adjusted Rand index, computed (not only) by the R-command adjustedRandIndex in package mclust. This is based on comparing for all pairs of observations whether they are in the same cluster in one ...


1

When you use hclust in R, all of the information that you need is stored in the output of the function. You can just save the output and read off what you need. However, the output may not be completely transparent, so I will go through a very small example and you can apply the ideas to your data. To get a small example, I will randomly sample eight ...


1

Trying to answer Xi'an comment, to check if I am understanding his comment. As far as I am understanding he means that upgrading $\pi$ to random variable would have the meaning of modelling our uncertainty of $\pi$ but not the sampling process of the data points. I try to write this better for exercise. In the standard setting, when we fit a Gaussian ...


Only top voted, non community-wiki answers of a minimum length are eligible