Iterator

1,526 reputation

417

bio	website
	location
	age
visits	member for	1 year, 11 months
	seen	Feb 12 at 11:46
stats	profile views	222

I noticed that this profile receives views. Let me make your click a little more worthwhile, and explain Iterator's raison d'être...

I do some things again and again, or my code does (almost) the same thing again and again. Iterating isn't tedious per se, but it helps to iterate rapidly. I am mastering that.

I work on topics in many areas, usually solving stimulating problems with stimulating data. Sometimes there is a stimulating amount of stimulating data.

To contact me, look for me in the SO R chat room, where I am often learning from R experts, or leave a message there or in a separate chat (@Iterator) - SO will alert me.

	1,526 reputation	bio	website		visits	member for	1 year, 11 months
417 badges		location			seen	Feb 12 at 11:46

summary answers questions tags badges favorites bounties reputation activity

352 Actions

suggestions reviews revisions comments badges posts accepts all

Feb 1	awarded	Nice Answer
Sep 13	awarded	Taxonomist
Jul 1	awarded	Yearling
Jun 24	comment	How can I optimise computational efficiency when fitting a complex model to a large data set repeatedly? What's so bad about a little iteration? ;-)
Jun 24	comment	How can I optimise computational efficiency when fitting a complex model to a large data set repeatedly? One modification to this: running on m2.4xlarge (the 68.7GB RAM option) is one only way to guarantee you're getting the full machine, so that you don't necessarily hit RAM caching issues that may occur on VMs (virtual machines / AMIs) that run on a fraction of the machine.
Feb 19	comment	What programming language do you recommend to prototype a machine learning problem? I'd recommend R, Python, or Matlab. For reasons too extensive to address, I'd drop Matlab. For a stats person, I'd go with R, for a programmer, I'd go with Python. For the inner loops, I'd go with C/C++. At sufficient scale, Matlab's costs exceed any benefits.
Feb 19	comment	Software for easy-yet-robust data exploration Regarding JMP - If I'm not mistaken, there are trial versions and academic licenses that are within the reasonable.
Feb 19	comment	Software for easy-yet-robust data exploration Agreed (well with point #1). In some ways, it is even better than R for the EDA stuff, and can be inserted nicely into an iterative workflow. If one is already working without R, then JMP is a reasonable way to work without adding R. Because it can also link to R, an R programmer can create tools that then plug in to JMP, allowing the Eloi, errr, user to continue to believe that everything is just dandy.
Feb 19	comment	Software for easy-yet-robust data exploration I'll say that JMP is quite good. Even if one is very proficient in R, there are times when JMP is a faster way to navigate and analyze data.
Feb 15	revised	Why is there an R^2 value (and what is determining it) when lm has no variance in the predicted value? added 195 characters in body
Feb 14	comment	Why is there an R^2 value (and what is determining it) when lm has no variance in the predicted value? @drknexus I disagree. QR decomp is one of many numerical algorithms; if the underlying issue is numerical precision, this'll crop up in QR, matrix multiplication, non-linear solvers, and so many other places. The essential sequence is simple: the coefficients are very slightly off (should be (0,1)); this isn't unreasonable, yet produces the `mss` and `rss` "noise". It's the GIGO principle that assures that $R^2$ is precise, but incorrect. I'd rather insert a "garbage detector" before calculating $R^2$ than to modify the QR algo, because I doubt its validity could be improved.
Feb 14	comment	Why is there an R^2 value (and what is determining it) when lm has no variance in the predicted value? @eznme In order to get the output, one has to detect possible underflow issues. This is often, but not always, feasible. In a simple case like this, I think that there should be an underflow check. It's a ratio of two floats, which is an operation that's not a big part of the computational cost of the summary. I don't excuse not checking, but, as Ben points out, it'll take some time and effort to get these checks inserted in lots of places.
Feb 14	comment	Why is there an R^2 value (and what is determining it) when lm has no variance in the predicted value? (Continued) But, as a user, the choice of CV is a better site, for the simple reason that diligent statistical analysis is the responsibility of the user, not the developer. If the user sees an erroneous $R^2$ relative to the magnitude of the RSS, then they should do their own post-processing before reporting further. Programming-wise, I'd like to know how to avoid these numerical issues as much as possible, but I think that they can't be escaped, and that's where it's important to have a diligent user and to educate others.
Feb 14	comment	Why is there an R^2 value (and what is determining it) when lm has no variance in the predicted value? @drknexus I slightly agree, but am rather ambivalent. The expertise on FP numerical computing is likely more numerous on SO; on CV there's a lot more expertise in the statistical side. Either way, it's an R question, and Ben's comment below, about R Core, along with whuber's about Chambers' book, are likely the most insightful responses achievable on either site. Only with these answers can we learn whether there are statistical or programming blunders. It turns out that it's neither: it has more to do with numerical precision and the priorities of R core.
Feb 14	revised	Why is there an R^2 value (and what is determining it) when lm has no variance in the predicted value? formatting changes
Feb 14	revised	Why is there an R^2 value (and what is determining it) when lm has no variance in the predicted value? LaTeX formatting
Feb 14	comment	Why is there an R^2 value (and what is determining it) when lm has no variance in the predicted value? @eznme While an error is good, it is quite difficult to catch all kinds of places where floating point issues arise, notably in the world of IEEE-754 arithmetic. The lesson here is that even the bread and butter calculations with R should be handled delicately.
Feb 14	revised	Why is there an R^2 value (and what is determining it) when lm has no variance in the predicted value? added 486 characters in body
Feb 14	comment	Why is there an R^2 value (and what is determining it) when lm has no variance in the predicted value? +1 This is a fascinating question, especially with the curious progression of `NaN`s.
Feb 14	comment	Why is there an R^2 value (and what is determining it) when lm has no variance in the predicted value? @BrandonBertelsen Hehe... I address `NA`s over on StackOverflow. ;-) At least that's been my impetus for several recent questions - breaking my code doesn't happen often enough in some cases, and accounting for NAs: bring it on! :)