I have data describing over a 15 million individuals where each item includes variables like these:
- A. Amount spent on airfare last year
- B. Brand of shoes
- C. Number of times visited some website in 6 months
- D. Speaks Malay (Y/N)
Here is some example data:
+----------+--------+-----+-----+ | A | B | C | D | +----------+--------+-----+-----+ | $4568.70 | Nike | 18 | - | | $220.17 | - | 25 | Yes | | $0 | - | 157 | No | | $2170.87 | Adidas | - | - | +----------+--------+-----+-----+
As you can see, some of the variables are categorical rather than numerical. The data for many items is incomplete but I prefer not to throw out any rows or columns.
What are some methods to estimate things like:
What is the average expected airfare and visits to some website of a group of 17 people given that 5 wear Nike shoes, 11 have Adidas shoes and 1 other has some other kind of shoes where 6 of them speak Malay?
In this case, I know the marginal distributions for columns B and D for a totally new group of people. Though I have reason to believe that none of the variables are independent, I would like to give the best possible estimates for the distributions of the unknown variables (A and C) for the new group of individuals.
How can I estimate things like the above? Are there any convenient non-parametric methods to estimate things like the above that scale well (suppose instead that I had hundreds of millions of rows by thousands of columns)? I'm looking for both techniques and packages with implementations.
