0
$\begingroup$

Given you had a large dataset of people with a name and country. How would you calculate an estimate of the origin country from one of the names? If i looked up a name, Kim and just averaged the rows from each country, it seems to me that my numbers would be skewed since the dataset might contain a lot more names from a given country in the first place. How can i make up for this uncertainty?

$\endgroup$
  • $\begingroup$ @ Stromgren If I understand correctly from your comments below, this is a joint [P(Kim,US)] versus conditional [P(Kim|US)] probability question. I would suggest that you incorporate some of your explanations in your comments into the question to increase clarity. $\endgroup$ – Zhubarb Oct 8 '13 at 13:03
1
$\begingroup$

Welcome to the site.

I am not sure what you mean when you write "averaged the rows from each country". You can't average countries.

If your data set is a random selection of the world's population (lots of luck a true random sample of that!) then you can say, for each name, what proportion are from each country. You can also say, for each country, what proportion of people have each name.

But I don't see any sort of average you could take here.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Hey Peter. Thanks for your answer. I may have formulated my question badly. I'm not used to doing statistics. I will try to paint out my situation better. I have a huge dataset, each row containing a first name and a country. My goal is to use this data to provide a probabilistic guess of where a certain name is from. So, at first thought i wanted to fetch all the rows with a given name and then count how many instances of each country was returned. My problem is that some countries will have a lot more data than others. So my question is: How can i make up for this? $\endgroup$ – Stromgren Oct 6 '13 at 10:54
  • $\begingroup$ Oh I see. Why not just use proportions? E.g. "2% of people in America are named Michael, but only 0.01% of people in China are?" etc. $\endgroup$ – Peter Flom Oct 6 '13 at 11:06
  • $\begingroup$ Yes, this is exacly what i'm trying to do. My problem lies in how i would calculate these estimates. An example: I'm looking up all rows with name "Kim". I get 75 rows from "US" and 25 rows from "GB". So 75% and 25% of course. But my data is random. I might have 50.000 rows of people from "US" and only 10.000 rows of people from "GB". So "US" would get much higher estimates. I want to make up for this difference. My intuition tells me that i, in some way, will have to put the total number of rows from each country into my calculation? But i can't seem to think it longer than that :) $\endgroup$ – Stromgren Oct 6 '13 at 11:13
  • $\begingroup$ "My goal is to use this data to provide a probabilistic guess of where a certain name is from." You already know which country each name is from. My sense is that, given a new name, you want to guess which country that name is from. You could get into all sorts of complex methods such as neural networks, but why not simply assign as your guess the country that has the highest % of its names matching that name? $\endgroup$ – rolando2 Oct 6 '13 at 15:18
1
$\begingroup$

You have a lot of rows with names and states. The number of rows don't seem to match the size of the population of the state. You need to standardize the results.

In order to move forward we need to assume that the sample you have is random. If its not then this method may be flawed.

Here is how I would do it: Count the rows of the name and sum it up by state. Divide the sum of the name rows with the total number of rows of the state. The resulting number have to be standardized. I recommend using the population of the state as a weight. The standardization is done by multiplying the resulting number of the former division with the population of the state. Now you get a state by state comparable number. The next step is to do this calculation for the same name in every country. Calculate the sum of the comparabe numbers for each name in each state and divide each comparable number by the sum and multiply by 100. Now you get the percentage point that the name is from the specific country in question. The total percentage points for each name should equal 100 %.

Now, if you have a name and asks yourself which state this name comes from, you should have the probabilities from the above calculations.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I'm not saying you're wrong, but I'm confused as to why you'd standardize the results - could you provide further logic behind your reasoning? To me, I see it as there are ~30,000,000 people in the UK. Let's say there's 58,000 named "Kim." 58,000/30,000,000 = .0019333. If he wants to represent the probability as a decimal, there it is. If he wants to express it as a percent you just...obviously multiply by 100. If he meets the requirements with the size of his sample size per name per country, and the data in the database is truly random then: Central Limit Theorum. $\endgroup$ – Taal Oct 6 '13 at 22:36
  • $\begingroup$ Thus that decimal representing the true probability of the population (.0019333) of randomly finding someone named "Kim" in the UK would eventually converge with a p probability and +/- desired confidence interval of it not (or least getting quite a close approximation). There's probably some metric I'm stupidly missing that would actually describe one's approximate residual's as they converge the estimated probability with the central limit theorum though - as although there's a chance it will hit it exactly beforehand, there will be some form of variance. $\endgroup$ – Taal Oct 6 '13 at 22:46
  • 1
    $\begingroup$ Taal: I just want to answer the question put forward. "How would you calculate an estimate of the origin country from one of the names?" That is, we draw a name at random and want to find out the most probable origin country for that name. For me it is impossible to calculate an estimate of the origin country without weights. $\endgroup$ – Cookie Monster Oct 7 '13 at 17:48
  • $\begingroup$ I do genuinely appreciate your answer (as I was hoping my criticism would not come off as hostile, but instead "curious" for lack of a better word), but have a few more questions - unfortunately I have to finish a bit of work first - but I may ping you back here in a few days. $\endgroup$ – Taal Oct 8 '13 at 3:37
  • $\begingroup$ Taal: I will try to elaborate a little. What Stromgren is actully asking about is: "How do you compare averages in different countries". For me the question is about comparability and standardization. The easiest way forward is to attach weights. Why? Because even if the proportion namned Kim in the US (population around 300 million) and Sweden (9 million) is the same, the chance that a random person namned Kim is from the US is many times greater than from Sweden. $\endgroup$ – Cookie Monster Oct 8 '13 at 9:00
0
$\begingroup$

It sounds you want this: If you were to imagine all countries on the planet did not exist except for the UK (as an example) because, let's say...aliens invaded and took everyone else out (but the UK didn't know about it and never will because they live in a giant impenetrable bubble they don't know about - just making sure that our aliens are not messing with your results - at least as you survey their first names - so that the element of "randomness" is upheld....then again maybe you wanted the data only for that time period. Or after. Because the first name ratios may change if they got inside the bubble ....unless they took everyone out at complete random too, then you'd be fine.)

And then you go to the UK and met someone there at random. You want to know what the probability is that their name would be, for example, "Sam".

Essentially, the probability for each country for a given name.

Is this correct?

If so, then you can either

A) Find this data online already if you looked hard enough (opendata.stackexchange.com may help)

B) Or if you want to generate the values yourself (and depending upon your needed accuracy)...then the database you're using has to be very accurate (or actually just also completely random as well as big enough to also be big enough for you - I don't know how to calculate that metric.) and your first name data has to be completely random or at least completely randomly drawn from whichever specific countries your first names come from - there can be absolutely nothing in common with between who your first names represent or where they came from besides the fact that they were all random and thus will likely have random similarities (lol). Then again these rules lax depending upon how accurate you need/want to be.

You'd just do exactly what you said to get the best possible guess from the amount of names you have.

For each name you put in that is from the UK, record that that name appeared once and someone from the UK's population appeared once.

At the end, assuming you went through enough names for each countries sample size requirements (and your p value requirements are achieved, which is the probability that your conclusions/results occurred strictly by chance and you cannot trust your data, it's usually put at around .05 or 5%...there's also a confidence interval around that, but I don't think you need to worry about it much...you probably have enough names...maybe).

Then, as you said before, divide the number of unique names you recorded for each name for each country by the number of times you recorded that anyone's name appeared from that country.

(number of times the name "Sam" appeared and lived in the UK)
-----------------------------------------------------------
 (number of times any name appeared that lived in the UK)

=

The probability that name ("Sam") would be randomly selected from the population of that country (the UK here).

This has a p value in a -/+ confidence interval. The p value is probably 5% if that's what you chose (5% is "generally considered" to be acceptable that you're data is not wrong) when figuring out how many names you'll need. It is probably in a +/- confidence interval of 2.5% (also common figure) if that is also what you chose when figuring out how many names you'll need.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.