1
$\begingroup$

Bear with me:

I have a data set that is begging for the tf-idf transformation to account for a long tailed distribution of degree. Right now, it is in the form of a network where a tie represents co-purchases, but because some products are wildly more popular than others, they end up as the strongest ties for literally everything.

The problem is that the technology I'm using still doesn't have a built in log function. What are the alternatives to idf to adjust for uneven distribution of ties?

I'm also open to any other solutions in network analysis for adjusting strength of tie based on centrality.

Improve this question
$\endgroup$
4
  • $\begingroup$ Is there no way to load the data into another application and compute the log there? $\endgroup$
    – AdamO
    Jun 11, 2013 at 19:16
  • $\begingroup$ That's the back up plan, but there's a lot of data, so this would take many times longer than if I do in the application/original database. $\endgroup$
    – Olga Mu
    Jun 11, 2013 at 19:26
  • $\begingroup$ This question mixes statistical and computing facets, and that is not a problem in itself. But if the unnamed technology doesn't have a log function, what else does it lack? How are people supposed to advise if they can't know what is computable for you? $\endgroup$
    – Nick Cox
    Jul 10, 2013 at 8:49
  • $\begingroup$ No secret here-- I'm using Cypher (the lamguage for Neo4j, a graphical database), and there's no log, but plenty of string operations. $\endgroup$
    – Olga Mu
    Jul 10, 2013 at 18:37

2 Answers 2

Reset to default
1
$\begingroup$

A rough calculation of log to the base 10 can be done by taking the number of digits in the decimal representation of an integer (provided there is a way to do this in your technology). This computes ceiling(log10(n+1)) for any integer n > 0.

Improve this answer
$\endgroup$
5
  • $\begingroup$ You are clearly correct, but this presupposes that the OP has functions to do this, i.e. to calculate the length of a string representation of an integer. If the software doesn't have a log function, what else is it missing? (Clearly I don't know.) $\endgroup$
    – Nick Cox
    Jul 9, 2013 at 9:51
  • $\begingroup$ Yes, I agree. I will modify my answer to reflect this assumption. $\endgroup$
    – raghu
    Jul 10, 2013 at 5:49
  • $\begingroup$ That's true, but I am having hard time understanding how to use this to solve my problem. $\endgroup$
    – Olga Mu
    Jul 10, 2013 at 18:38
  • $\begingroup$ @OlgaMu You say you have a lot of string functions. Calculate N/df, truncate it to integer, convert to a string, and count the number of digits. This is a rough approximation for the log to base 10. $\endgroup$
    – Hong Ooi
    Aug 9, 2013 at 8:22
  • $\begingroup$ @HongOoi That makes complete sense now. Thank you for explaining it (I'm clearly a bit slow on this). Accepting answer! $\endgroup$
    – Olga Mu
    Aug 10, 2013 at 12:41
1
$\begingroup$

Why not simply use a linear term: idf = N/df instead of idf = log(N/df)? The linear term still captures the "essence" of idf, which is giving a higher score to rare terms. The log() function only tweaks or dampens the effect, but depending on your application it could be omitted.

Improve this answer
$\endgroup$
3
  • $\begingroup$ @COOLSerdash I disagree, this is a direct answer to the question "What are the alternatives to idf". An answer does not necessarily need to incorporate a complicated formula. $\endgroup$
    – Leeor
    Jul 9, 2013 at 8:45
  • 1
    $\begingroup$ I agree that an answer does not necessarily need to incorporate a complicated formula. I still think that the answer, as it stands now, would be better suited as a comment rather than an answer. The FAQ clearly states that fuller answer that are a bit more fleshed-out are desired. I do not - in any way - say that your answer is wrong. But a bit more explanation why you propose the linear term would be helpful. $\endgroup$
    – COOLSerdash
    Jul 9, 2013 at 8:52
  • $\begingroup$ Leeor-- the problem is that my distribution is really skewed. Without the log, the items at the high end of the distribution just dominate everything. $\endgroup$
    – Olga Mu
    Jul 10, 2013 at 18:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.