Jonathan

903 reputation

213

bio	website
	location	California
	age
visits	member for	2 years, 3 months
	seen	20 hours ago
stats	profile views	53

	903 reputation	bio	website		visits	member for	2 years, 3 months
213 badges		location	California		seen	20 hours ago

summary answers questions tags badges favorites bounties reputation activity

142 Actions

suggestions reviews revisions comments badges posts accepts all

1d	comment	How to plot a “differential word cloud” in R? this would be effective for large numbers of classes/tags, but for classes like UserType which divide the total population into a small number of groups, then the "document" definition may be too narrow - if there are 4 types, then DF would only be in increments of 25%! Instead, it might be better to treat each response as a document, and compute TF-IDF by summing up all occurrences in the class and divide by DF across the entire list of responses. Do the same for documents outside the class, and the difference between the TF-IDFs would be your score.
May 15	comment	Estimating repeat shoppers from an incomplete sampling I just saw the question stats.stackexchange.com/questions/2466/… so I will be playing with that to see if I can apply it here...
May 13	revised	Estimating repeat shoppers from an incomplete sampling added discussion of how mark-recapture framework applies
May 13	comment	Estimating repeat shoppers from an incomplete sampling @GaëlLaurans thanks! I will see what I can find.
May 13	comment	Estimating repeat shoppers from an incomplete sampling @Cam.Davidson.Pilon It is done by recording their ID and later tallying how often I recorded them. Interestingly, from another source, I have a "how many times..." type datapoint, but considered that even harder to work with as I only know the number up to that point in time that it was measured, but don't know what that number looks like later in time.
May 13	asked	Estimating repeat shoppers from an incomplete sampling
Apr 8	comment	How would you approach weighting of open response lists in a statistical manner? Adrian, perhaps it would be easier for us to answer if there was a more general question you could start with. You've jumped right to how you would weight responses (presumably for developing a score to apply back to list items). But, I don't think we know yet if weighting is even the right approach (ttnphns's comment suggests something else entirely).
Mar 16	comment	N-grams or vector space model for text mining? I'm having a hard time understanding how this is related to a vector space model or n-gram. It sounds like you're searching a database with binary values, not doing anything with text.
Mar 16	answered	How to store (and analyse) multi-answer multi-choice questionnaire data
Feb 20	awarded	Yearling
Jan 24	comment	Evaluating a text analytics engine for document concepts identification This sounds like a very hard question... your evaluation criteria will be subjective, but you want an automated process. I don't think the combination of the two can be done!
Jan 15	comment	Most interesting statistical paradoxes @Nick: this particular example is actually distinct from Simpson's Paradox, but it can be hard to know which fallacy/paradox applies in a particular situation because they look the same statistically. The difference is that SP is a "false effect" that appears only when analyzing subgroups. This trend shown is though to be a "true effect" that appears only when analyzing subgroups. In this case, it suggests that while income as a raw number doesn't affect voting patterns in aggregate, income as related to your neighbors (your state) does influence voting patterns.
Dec 31	comment	Use of text mining document classification via decision rules? This is a very open-ended question and seems to be based on your experience interacting with a specific text analysis package. Can you narrow down what your question is, and tell us what software you're basing your example from?
Nov 15	comment	Is there any proper way to fix a sample to adjust for known demographic overrepresentation? In general "weighting" is what you're looking for... but in the example you give, weighting may be insufficient to the task. Weighting corrects for over- and under-representation, but it won't correct for relevant categories of people being totally excluded, for example because they don't own a smartphone.
Nov 13	comment	Creating a recommender system for shoe sizing It might be a multi-pronged approach depending on the sparsity of the data. If you have "enough" specific matches of people who wear Nike size 9 and Adidas size x, then you might use the average of the specific matches. Then, if there aren't enough, you use a general sizing rule like +0.5.
Nov 5	comment	Can statistics based on crude units of measure be improved? @RioRaider while the normalization is definitely a good approach there would still be a problem with non-cup units... a recipe may e.g. combine "pounds of chicken" with "cups of flour".
Nov 5	comment	Application of Bayes' theorem "How does our brain automatically correct the order of letters?" Maybe it's not correcting the order. Just because we write words in order doesn't mean that's how our brain recognizes them. People are starting to process information a couple of words ahead while they're still starting to process the word in front of them. From the example it seems that the first/last letters (probably combined with the overall context) are used to pare down the possible space of words, and each of the letters present inside the word narrow it down even further. But, that's just further speculation.
Oct 25	comment	Variables for post-stratification weights? yeah, that seems like a lot of data quality issues :)
Oct 24	comment	Variables for post-stratification weights? What kind of data quality errors do you expect to see with demographics? In my experience, those are the most reliable. Most people know their age with a great degree of precision, for example. You are correct, on the other hand, that many of them tend to vary less than some behavioral metrics with different levels of effort to reduce non-response. But, most surveys tend to under-sample young people and hispanics. You would need to know what the "true" distribution is in order to weight them correctly, of course. US Census doesn't apply to organization memberships.
Oct 19	comment	Rates affecting rates I think there might be a little more to do here... I created a random dataset to test, and one of the sites tied for the highest contribution actually had below-average waste rate, because it was a larger site. I'll see if I can think of another approach...