I have a set of Google search queries for a particular user. Every query has its frequency, that is the number of times the person searched it for. Now I have to infer some information about this person, let's say, age or gender. Now to represent this user, not all queries are useful. Queries which appear in abundance, like google.com or queries which appear only once or twice, aren't useful.

Overall if you plot the query VS its frequency distribution, it will be an exponential distribution with a mean μ and variance σ. Now I want to sample a few queries from this distribution, queries which have useful information. So what I do now is, I take a window of σ around the mean frequency μ, and take all the queries which have frequencies in this range.

But as mean is dependent on the query with maximum frequency, my mean μ shift towards that query and the window also lies close to that, ignoring other queries. Are there any better methods to do the sampling here on the basis of frequency of a query?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.