48

Two types of metric MDS The task of metric multidimensional scaling (MDS) can be abstractly formulated as follows: given a $n\times n$ matrix $\mathbf D$ of pairwise distances between $n$ points, find a low-dimensional embedding of data points in $\mathbb R^k$ such that Euclidean distances between them approximate the given distances: $$\|\mathbf x_i - \...


35

PCA yields the EXACT same results as classical MDS if Euclidean distance is used. I'm quoting Cox & Cox (2001), p 43-44: There is a duality between a principals components analysis and PCO [principal coordinates analysis, aka classical MDS] where dissimilarities are given by Euclidean distance. The section in Cox & Cox explains it pretty clearly:...


24

In case you will accept a concise answer... What questions does it answer? Visual mapping of pairwise dissimilarities in euclidean (mostly) space of low dimensionality. Which researchers are often interested in using it? Everyone who aims either to display clusters of points or to get some insight of possible latent dimensions along which points ...


19

PCA selects influential dimensions by eigenanalysis of the N data points themselves, while MDS selects influential dimensions by eigenanalysis of the $N^2$ data points of a pairwise distance matrix. This has the effect of highlighting the deviations from uniformity in the distribution. Considering the distance matrix as analogous to a stress tensor, MDS may ...


15

I had exactly the same question and posted it on a YouTube video of a CS231n lecture given by Andrej Karpathy a few weeks ago. Here is the question I posted followed by Andrej' response: https://www.youtube.com/watch?v=ta5fdaqDT3M&lc=z12ji3arguzwgxdm422gxnf54xaluzhcx Q: Does t-SNE need an entire batch of images (or more generally, data) to create ...


13

It would take more than a terabyte just to store a dense distance matrix of that size, using double precision floating point numbers. It's probably not feasible to attack such a large-scale problem head-on using a standard MDS algorithm. The runtime scales as $O(n^3)$, and you may not even be able to fit the distance matrix in memory on a single machine. ...


11

When dealing with streaming data, you might not want/need to embed all the points in history in a single t-SNE map. As an alternative, you can perform an online embedding by following these simple steps: choose a time-window of duration T, long enough so that each pattern of interest appears at least a couple of times in the window duration. scroll the ...


11

In multidimensional scaling, how can one determine dimensionality of a solution given a stress value? Having a stress value it is not possible to determine the dimensionality of the dataset. At best, you can evaluate whether the value is low or high (this evaluation is also a bit problematic to me). From what I understand, stress value is inversely ...


10

The function MDSplot plots the (PCA of) the proximity matrix. From the documentation for randomForest, the proximity matrix is: A matrix of proximity measures among the input (based on the frequency that pairs of data points are in the same terminal nodes). Based on this description, we can guess at what the different plots mean. You seem to have ...


8

You can look at the "GOF" component of the result ("goodness of fit"), if you specify the number of dimensions. It returns two numbers, that should be equal unless the distance matrix is not positive. You can also directly look at the eigenvalues: when they become small, you have enough dimensions. In the following example, two dimensions seem sufficient. ...


8

Update: I entirely removed my original answer, because it was based on a confusion between Euclidean distances and scalar products. This is a new version of my answer. Apologies. If by pairwise distances you mean Euclidean distances, then yes, there is a way to perform PCA and to find principal components. I describe the algorithm in my answer to the ...


8

You should perform feature normalization only on features - so only on your input vector $x$. Not on output $y$ or $\theta$. When you trained a model using feature normalization, then you should apply that normalization every time you make a prediction. Also it is expected that you have different $\theta$ and cost function $J(\theta)$ with and without ...


7

Here's a good paper comparing various dimension reduction techniques to PCA: http://www.iai.uni-bonn.de/~jz/dimensionality_reduction_a_comparative_review.pdf In brief, the paper covers the following techniques, though there are many more: (1) multidimensional scaling, (2) Isomap, (3) Maximum Variance Unfolding, (4) Kernel PCA, (5) diffusion maps, (6) ...


7

There is a recently published variant, called A-tSNE, which supports dynamically adding new data and refining clusters either based on interest areas or by user input. The paper linked below has some pretty nice examples of this: Citation: arXiv:1512.01655 Approximated and User Steerable tSNE for Progressive Visual Analytics Nicola Pezzotti, Boudewijn ...


6

@ttnphns has provided a good overview. I just want to add a couple of small things. Greenacre has done a good deal of work with Correspondence Analysis and how it is related to other statistical techniques (such as MDS, but also PCA and others), you might want to take a look at his stuff (for example, this presentation may be helpful). In addition, MDS is ...


6

I wouldn't place much stock in "rules of thumb" such as this. It is dependent upon so many things such as the number of variables, the number of sites, what dissimilarity you use etc. Also note that the vector fitting approach is inherently linear and we have no reason to presume that the relationship between the variable and the NMDS configuration is linear....


6

A general framework which addresses your problem is called dimensionality reduction. You would like to project data from N dimensions to 2 dimensions, while preserving the "essential information" in your data. The most suitable method depends on the distribution of your data, i.e. the N-dimensional manifold. PCA will fit a plane using least squares criterion....


6

I find it helpful to consider the singular value decomposition for questions like this with the assumption that $X$ is a real matrix. Writing $X = UDV^T$, we can see that $XX^T = UD^2U^T$ and $X^TX = VD^2V^T$. As we can see, the eigenvalues of both $XX^T$ and $X^TX$ are contained in the diagonal matrix $D^2$ and are indeed equal. Also, we see that the matrix ...


6

The Barnes-Hut approximation makes t-SNE highly scalable (at least, you can use it with 100 000 lines, I tried it). You can call it from R : Rtsne The complexity of the implemented algorithm is $O(n\log(n))$ whereas the naive implementation had a complexity of $O(n^2)$. The details of the underlying approximation can be found here Accelerating t-SNE using ...


6

Despite having 24 original variables, you can perfectly fit the distances amongst your data with 3 dimensions because you have only 4 points. It is possible that your points lie exactly on a 2D plane through the original 24D space, but that is incredibly unlikely, in my opinion. It is reasonable to imagine that the variation on the third dimension is ...


5

There is a very large literature on such data, since the 1980s at least usually called compositional. http://en.wikipedia.org/wiki/Compositional_data http://www.compositionaldata.com/ The largest single fraction of this literature is quite possibly in mathematical geology. Other applications are numerous. One is the study of expenditure budgets in ...


5

This may be only a partial answer because I don't think the plot that you expect is really what is in the data, especially the "parallelity and continuity" of the intermediate signals. I will speculate on reasons for that below. But I think I was able to get to what you look for in terms of the four basal signals A1, A5, E1, E5. Namely that they lie on the ...


5

These two books are in full agreement. Classical multidimensional scaling (where by "classical MDS" I understand Torgerson's MDS, following both Hastie et al. and Borg & Groenen) finds points $z_i$ such that their scalar products $\langle z_i, z_j \rangle$ approximate a given similarity matrix as well as possible. However, any dissimilarity matrix can ...


5

An interesting option would be exploring neural-based dimensionality reduction. The most commonly used type of network for dimensionality reduction, the autoencoder, can be trained at the cost of $\mathcal{O}(i\cdot n)$, where $i$ represents the training iterations (is an hyper-parameter independent of training data). Therefore, the training complexity ...


4

Comparison: "Metric MDS gives the SAME result as PCA"- procedurally- when we look at the way SVD is used to obtain the optimum. But, the preserved high-dimensional criteria is different. PCA uses a centered covariance matrix while MDS uses a gram matrix obtained by double-centering distance matrices. Will put the difference mathematically: PCA can be viewed ...


4

Most MDS implementations sort the obtained dimensions in the order of decreasing variance along them. Thus, the answer to your question is "typically, yes". In this respect MDS is similar to PCA. Both are techniques to map-in-low-dimensions. However, PCA is dimensions-reduction technique which cuts-off informatively "weak" dimensions, leaving just m ...


4

I'm not certain of your exact data, or the process you're using to analyze it, but what you describe makes me think of a correlation matrix. In R, generating the matrix, as well as the corresponding heat map (with dendrogram) is easy. The example below used example data to show correlations between usage rates of different IT applications, and generates the ...


4

PCA with a distance matrix exists, and it is called Multi-dimensional scaling (MDS). You can learn more on wikipedia or in this book. You can do it in R with mds function cmdscale. For a sample x, you can check that prcomp(x) and cmdscale(dist(x)) give the same result (where prcomp does PCA and dist just computes euclidian distances between elements of x)


4

You should not touch the weights. The way to proceed is to center and scale the training data and apply that same transformation to the test data. They complement each other. You may have a look at this video where the topic is nicely explained. Scaling is relevant from a practical point of view, because large scale methods are sensitive to unnormalized ...


4

The NMDS vegan performs is of the common or garden form of NMDS. If metaMDS() is passed the original data, then we can position the species points (shown in the plot) at the weighted average of site scores (sample points in the plot) for the NMDS dimensions retained/drawn. The weights are given by the abundances of the species. This is one way to think of ...


Only top voted, non community-wiki answers of a minimum length are eligible