3
$\begingroup$

I will try to be as clear as I possibly can:

  • I want to populate a matrix of about 10 million rows by 43 columns as a test to an application
  • The first task is to generate a random number around 10 million
  • Second I need to fill out the columns. Let's say for column 2, random ints can only be generated between 1 and 25. Also, the distribution/frequency of value = 1 is about 80% of the 10 million, 9 to 25 is about 15% where 9 is 90% of the 15%, and 5% randomly distributed between everything else.
  • Let's say for columns 5 to 10, we know that if column_2 = 10 then there is about 80% possibility that the values will be roughly some_int_1.
  • I have a lot of these little rules.
  • None of them are discrete.
  • How can I go about generating such matrix?
$\endgroup$
4
$\begingroup$

If I think I've got what you're saying, here's some R code for basic stuff:

ss<-1000 #just in case of problems, change to 1000000 for your purposes
a1<-rnorm(ss)
a2<-sample(1:25, ss, replace =T, 
  prob=c(0.8, rep(0.05/length(2:8), length(2:8)), 0.135,  
  rep((0.15-0.135)/length(10:25), length(10:25))))
a5<- ifelse(a2 == 10, sample(1:5, 1, prob=c(0.8, rep(0.2/4, 4))), sample(1:5, 1))

df1<-cbind(a1, a2, a5)

If you want computational efficiency, well that's a little different.

$\endgroup$
1
$\begingroup$

One way to do this in Python is the following. You would of course have to change $nRow$ to 10 million. Depending on your computational needs and resources, may be better to not keep everything in memory. Generate each row, write out to file, and move on.

from numpy import random

nRow = 10
nCol = 43


def c1def(i):
    return(random.randn())

def c2def(i):
    x = random.rand()
    if x < 0.8:
        return(1)
    elif x < 0.95:
        if random.rand() < 0.9:
            return(9)
        else:
            return(random.randint(10,26))
    else:
        return(random.randint(2,9))

def c5def(vector):
    if vector[1] == 10:
        if random.rand() < 0.8:
            return(1)
    else:
        return(0)

rowVec = [0] * nCol
matrix = [rowVec] * nRow

for i in range(nRow):
    for j in range(nCol):
        matrix[i][0] = c1def(i)
        matrix[i][1] = c2def(i)
        matrix[i][4] = c5def(matrix[i])
        matrix[i][5] = c5def(matrix[i])
        matrix[i][6] = c5def(matrix[i])
        matrix[i][7] = c5def(matrix[i])
        matrix[i][8] = c5def(matrix[i])
        matrix[i][9] = c5def(matrix[i])
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.