4
$\begingroup$

I have a personal project I'm working on and could use some help with the math. Let's say I had two lists.

FN: 1 million first names (dupes allowed).
LN: 2 million last names (dupes allowed).

And let's say we're looking at two names:

"Murray": Appears in FN 5,000 times and in LN 75,000 times.
"Harrison": Appears in FN 10,000 times and in LN 70,000 times.

I'm given "Murray Harrison" as a full name. How would you approach calculating the odds that the intended name was actually "Harrison Murray"? For the scope of this project, it is acceptable to assume that any last name is just as likely to appear next to any first name.

I think that:

P(Murray Harrison) = P(Murray as First) * P(Harrison as Last) = 5000/80000 * 70000/80000 = 0.055 P(Harrison Murray) = P(Harrison as First) * P(Murray as Last) = 75000/80000 * 10000/80000 = 0.117

Here's where my question might get tricky:

How would you take those two probabilities and determine the odds that the two names should be swapped? For example, say I decide that if there's a 95% chance they should be swapped, I'll swap them.

Would I need additional data to determine this probability? Would I just look at the probabilities of both name orders and if one's > 0.95 I would select that order?

Edit: To be clear, I have a list of full names and I want to programmatically determine which name is the first name, and which is the last. Sometimes the data will be in Last-First order (without a comma) and sometimes it will be in First-Last order. I'm trying to use a bit of stats to determine the most likely order.

Improve this question
$\endgroup$
7
  • $\begingroup$ Not sure if I understand. What do you mean by swapping? You say it like when you get a harrison or a murray, you do not know if it is a LN or FN. Is that so? $\endgroup$
    – Hotaka
    Commented Sep 29, 2013 at 22:09
  • $\begingroup$ That is correct (see my recent edit). I have a list of full names where the order is inconsistent. $\endgroup$
    – Donnie
    Commented Sep 29, 2013 at 22:33
  • $\begingroup$ If P(Murray Harrison)=.055 and P(Harrison Murray) = .117, you are .117/.055=2.1 times likely to get a Harrison Murray than a Murray Harrison. If you want percentages, the chance that you have a Harrison Murray is .117/(.117+.055)=68%. This should be a start, but I am still puzzled. How did you figure out that Murray appears in FN 5,000 times and in LN 75,000 times without knowing whether it is a FN or LN?? $\endgroup$
    – Hotaka
    Commented Sep 30, 2013 at 1:59
  • $\begingroup$ Imagine you have a list of full names (two "words" long) but you have no confidence that they're in the correct order. How do you determine which "word" is the first name, and which is the last name? What you can do is see how many times the first word appears in the entire world as a first name, and how often it appears as a last name. Then you can calculate odds. FN, for our purposes, is treated like it has every first name in the world. So, I take "Murray", and calculate the odds that it's being used as a first name rather than a last. Does that make more sense? $\endgroup$
    – Donnie
    Commented Sep 30, 2013 at 6:12
  • $\begingroup$ Sorry, I'm further lost. And it's probably why no one else is answering. As soon as you said "... the first word appears..." it doesn't make sense. You're saying there is no order to the combination of words, so it makes no sense that you do know which one is the first or last word. $\endgroup$
    – Hotaka
    Commented Sep 30, 2013 at 11:38

1 Answer 1

Reset to default
2
$\begingroup$

I have doubts about your assumption that any first name can appear next to any last name. In the real world there is a strong dependence. Drawing "Itzhak" or "Yo Yo" from the first names list would drastically alter the real distribution of the last names. It would be better to have a large list of full names. If you find two Murray Harrisons and no Harrison Murrays, that's stronger evidence. Nevertheless, let's continue with your independence assumption:

If your hypothesis is that the name under test is reversed, then your odds of the hypothesis given a particular name is

$$Odds(reversed|name) = Odds(reversed) \frac{P(name|reversed)}{P(name|correct)}$$

The likelihood of drawing "Murray Harrison" from your lists is 0.005 * 0.035 = 0.000175 and the likelihood of drawing "Harrison Murray" is 0.01 * 0.0375 = 0.000375. Your likelihood ratio increases the odds of the reversed hypothesis by a factor of 2.143. This is pretty weak evidence. Your prior odds of any given name being reversed would have to be almost 9:1 to meet your P > 0.95 threshold.

Improve this answer
$\endgroup$
3
  • $\begingroup$ In the real world, there is certainly a dependence. However, assuming independence is definitely acceptable for the scope of this project. I see you're calculating P(Murray Harrison) by calculating the odds of randomly drawing Murray from FN and randomly drawing Harrison from LN. Does it change things to know that the names I'm getting aren't generated, but are coming in from real people in the world? I'm checking to see if each "word" exists in my tables, and if so, how frequently each appears as a first name versus a last. I feel like table size shouldn't matter, but I could be wrong. $\endgroup$
    – Donnie
    Commented Sep 30, 2013 at 16:39
  • $\begingroup$ I had assumed that the frequencies of names in your tables are representative of those found in the US, so it doesn't change anything for me. And the likelihood ratio you calculated is identical to mine, so I think the table sizes cancel out. The point is that a ratio of ~2 is never going to overturn your prior odds. Your independence assumption weakens the value of the evidence. $\endgroup$
    – Eldan
    Commented Sep 30, 2013 at 18:15
  • $\begingroup$ I was wondering if perhaps they canceled out. Yes, they are assumed to be representative, I just wasn't sure if generating randomly was equivalent to being handed a name from the outside. In retrospect, I can see that they are equivalent. The evidence is weakened by the independence assumption, yes, but at the moment, the project does not require that level of accuracy. It will certainly be improved over time to include more realistic dependencies. At any rate, I think I can safely call this answered. Thank you so much! $\endgroup$
    – Donnie
    Commented Sep 30, 2013 at 19:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.