I have data describing over a 15 million individuals where each item includes variables like these:

  • A. Amount spent on airfare last year
  • B. Brand of shoes
  • C. Number of times visited some website in 6 months
  • D. Speaks Malay (Y/N)

Here is some example data:

|    A     |   B    |  C  |  D  |
| $4568.70 | Nike   | 18  | -   |
| $220.17  | -      | 25  | Yes |
| $0       | -      | 157 | No  |
| $2170.87 | Adidas | -   | -   |

As you can see, some of the variables are categorical rather than numerical. The data for many items is incomplete but I prefer not to throw out any rows or columns.

What are some methods to estimate things like:

What is the average expected airfare and visits to some website of a group of 17 people given that 5 wear Nike shoes, 11 have Adidas shoes and 1 other has some other kind of shoes where 6 of them speak Malay?

In this case, I know the marginal distributions for columns B and D for a totally new group of people. Though I have reason to believe that none of the variables are independent, I would like to give the best possible estimates for the distributions of the unknown variables (A and C) for the new group of individuals.

How can I estimate things like the above? Are there any convenient non-parametric methods to estimate things like the above that scale well (suppose instead that I had hundreds of millions of rows by thousands of columns)? I'm looking for both techniques and packages with implementations.

  • $\begingroup$ This question is too broad & not sufficiently statistical for this site. You may want to read about questions to ask here in our help center. FWIW, just use your software to extract all users who meet whatever conditions, & then average over the relevant column. $\endgroup$ – gung - Reinstate Monica Jan 6 '15 at 3:23
  • $\begingroup$ Thanks for the link. My question is not so much about calculating probabilities of specific conditions or taking averages. I'd like to be able to make inferences based on counterfactual conditional statements so that I can say, for example: given what we know about how A,B,C and D relate, if A instead had a new, different distribution, how should I expect the distributions of B,C and D to look? $\endgroup$ – whaupter Jan 6 '15 at 5:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.