1
$\begingroup$

I am running into some problems performing a multidimensional scaling image. First of my dataset is quite large (330.000 fields: 33000 rows, 10 columns). The output image needs to contain 10 dots, 1 for each column. The fields in the dataset contain intensity values which first need to be converted to a distance matrix, however when doing so I run into a memory limit in R.

I was hoping you could help solve this problem or perhaps point me to a program that can handle these kind of data sizes.

$\endgroup$

closed as off-topic by gung, COOLSerdash, Christoph Hanck, Nick Cox, Tim Dec 28 '15 at 11:35

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "This question appears to be off-topic because EITHER it is not about statistics, machine learning, data analysis, data mining, or data visualization, OR it focuses on programming, debugging, or performing routine operations within a statistical computing platform. If the latter, you could try the support links we maintain." – gung, COOLSerdash, Christoph Hanck, Nick Cox, Tim
If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ What are you computing and how are you computing it? How much memory would it need? What version of R? 32 or 64 bit? What platform? $\endgroup$ – Glen_b May 18 '13 at 2:43
  • $\begingroup$ @Glen_b I am trying to analyze a dataset of logaritmic scores which are the result from a research institute, I am using windows 7 x64 but I can dual boot into ubuntu if needed. The amount of memory needed to run it in R is 32gb just to put it into a distance matrix before I can run a cmdscale. That is why I ask for help here, I was hoping someone would know a program that could handle this amount of data with relatively little memory (10gb or so) which I know is a long shot. $\endgroup$ – Sinshz May 21 '13 at 8:17
0
$\begingroup$

If your problem is the method using too much memory, I do not have a suggestion, but if it is just your data taking up too much memory you could check out the dplyr package, which enables you to create a local database with your data freeing memory.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.