Does Julia have any hope of sticking in the statistical community?

Question

I recently read a post from R-Bloggers, that linked to this blog post from John Myles White about a new language called Julia. Julia takes advantage of a just-in-time compiler that gives it wicked fast run times and puts it on the same order of magnitude of speed as C/C++ (the same order, not equally fast). Furthermore, it uses the orthodox looping mechanisms that those of us who started programming on traditional languages are familiar with, instead of R's apply statements and vector operations.

R is not going away by any means, even with such awesome timings from Julia. It has extensive support in industry, and numerous wonderful packages to do just about anything.

My interests are Bayesian in nature, where vectorizing is often not possible. Certainly serial tasks must be done using loops and involve heavy computation at each iteration. R can be very slow at these serial looping tasks, and C/++ is not a walk in the park to write. Julia seems like a great alternative to writing in C/++, but it's in its infancy, and lacks a lot of the functionality I love about R. It would only make sense to learn Julia as a computational statistics workbench if it garners enough support from the statistics community and people start writing useful packages for it.

My questions follow:

1. What features does Julia need to have in order to have the allure that made R the de facto language of statistics?

2. What are the advantages and disadvantages of learning Julia to do computationally-heavy tasks, versus learning a low-level language like C/++?

Answer 1

I think the key will be whether or not libraries start being developed for Julia. It's all well and good to see toy examples (even if they are complicated toys) showing that Julia blows R out of the water at tasks R is bad at.

But poorly done loops and hand coded algorithms are not why many of the people I know who use R use R. They use it because for nearly any statistical task under the sun, someone has written R code for it. R is both a programming language and a statistics package - at present Julia is only the former.

I think its possible to get there, but there are much more established languages (Python) that still struggle with being usable statistical toolkits.

Answer 2

I agree with a lot of the other comments. "Hope"? Sure. I think Julia has learned a lot from what R and Python/NumPy/Pandas and other systems have done right and wrong over the years. If I were smarter than I am, and wanted to write a new programming language that would be the substrate for a statistical development environment in the future, it would look very much like Julia.

This said, it'll be 5 years before this question could possibly be answered in hindsight. As of right now, Julia lacks the following critical aspects of a statistical programming system that could compete with R for day-to-day users:

(list updated over time...)

• optionally-ordered factor types
• most statistical tests and statistical models
• literate programming/reproduce-able analysis support
• R-class, or even Matlab-class plotting

To compete with R, Julia and add-on stats packages will need to be clean enough and complete enough that smart non-programmers, say grad students in the social sciences, could reasonably use it. There's a heck of a lot of work to get there. Maybe it'll happen, maybe it'll fizzle, maybe something else (R 3.0?) will supercede it.

Update:

Julia now supports DataFrames with missing data/NAs, modules/namespaces, formula types and model.matrix infrastructure, plotting (sorta), database support (but not to DataFrames yet), and passing arguments by keywords. There is also now an IDE (Julia Studio), Windows support, some statistical tests, and some date/time support.

Answer 3

For me, one very important thing for a data analysis language is to have query/relational algebra functionality with reasonable defaults and interactively-oriented design, and ideally this should be a built-in of the language. IMO, no FOSS language that I've used does this effectively, not even R.

data.frame is very clunky to work with interactively - for example, it prints the whole data structure on invocation, the \$ syntax is hard to work programatically with, querying requires redundant self reference (i.e., DF[DF$x < 10]), joins and aggregation are awkward. Data.table solves most of these annoyances, but as it is not part of the core implementation, most R code does not make use of its facilities.

Pandas in python suffers from the same faults.

These gripes may seem nitpicky, but these faults accumulate and in the end are significant in aggregate as they end up costing a lot of time.

I believe if Julia is to succeed as a data analysis environment, effort must be devoted to implementing SQL type operators (without the baggage of SQL syntax) on a user friendly table data type.

Answer 4

I can sign under what Dirk and EpiGrad said; yet there is one more thing that makes R an unique lang in its niche -- data-oriented type system.

R's was especially designed for handling data, that's why it is vector-centered and has stuff like data.frames, factors, NAs and attributes.
Julia's types are on the other hand numerical-performance-oriented, thus we have scalars, well defined storage modes, unions and structs.

This may look benign, but everyone that has ever try to do stats with MATLAB knows that it really hurts.

So, at least for me, Julia can't offer anything which I cannot fix with a few-line C chunk and kills a lot of really useful expressiveness.

Answer 5

I can see Julia replacing Matlab, which would be a huge service for humanity.

To replace R, you'd need to consider all of the things that Neil G, Harlan, and others have mentioned, plus one big factor that I don't believe has been addressed: easy installation of the application and its libraries.

Right now, you can download a binary of R for Mac, Windows, or Linux. It works out of the box with a large selection of statistical methods. If you want to download a package, it's a simple command or mouse click. It just works.

I went to download Julia and it's not simple. Even if you download the binary, you have to have gfortran installed in order to get the proper libraries. I downloaded the source and tried to make and it failed with no really useful message. I have an undergraduate and a graduate degree in computer science, so I could poke around and get it to work if I was so inclined. (I'm not.) Will Joe Statistician do that?

R not only has a huge selection of packages, it has a fairly sophisticated system that makes binaries of the application and almost all packages, automatically. If, for some reason, you need to compile a package from source, that's not really any more difficult (as long as you have an appropriate compiler, etc, installed on your system). You can't ignore this infrastructure, do everything via github, and expect wide adoption.

EDIT: I wanted to fool around with Julia -- it looks exciting. Two problems:

1) When I tried installing additional packages (forget what they're called in Julia), it failed with obscure errors. Evidently my Mac doesn't have a make-like tool that they expected. Not only does it fail, but it leaves stuff lying around that I have to manually delete or other installs will fail.

2) They force certain spacing in a line of code. I don't have the details in front of me, but it has to do with macros and not having a space between the macro and the parenthesis opening its arguments. That kind of restriction really bugs me, since I've developed my code formatting over many years and languages and I do actually put a space between a function/macro name and the opening parenthesis. Some code formatting restrictions I understand, but whitespace within a line?

Answer 6

The Julia language is pretty new; it's time in the spot light can be measured in weeks (even though its development time can of course be measured in years). Now those weeks in the spot light were very exciting weeks---see for example the recent talk at Stanford where "it had just started"---but what you ask for in terms of broader infrastructure and package support will take much longer to materialize.

So I'd keep using R, and be mindful of the developing alternatives. Last year a lot of people went gaga over Clojure; this year Julia is the reigning new flavour. We'll see if it sticks.

Answer 7

Bruce Tate here, author of Seven Languages in Seven Weeks. Here are a few thoughts. I am working on Julia for the followup book. The following is just my opinion after a few weeks of play.

There are two fundamental forces at play. First, all languages have a lifespan. R will be replaced some day. We don't know when. New languages have an extremely difficult time evolving. When a new language does evolve, it usually solves some overwhelming pain point.

These two things are related. To me, we're starting to see a theme taking shape around languages like R. It's not fast enough, and it's harder than it needs to be. Those who can live within a certain performance envelope and stay within established libraries are fine. Those who can't need more, and they're starting to look for more.

The thing is, computer architectures are changing, and to take advantage of them, the language and its constructs need to be constructed in a certain way. Julia's take on concurrency is interesting. It optimizes the right thing for such a language: transparent distribution and the efficient movement of data between processes. When I use Julia for typical tasks, maps and transforms and the like, I am just calling functions. I don't have to worry about the plumbing.

To me, the fact that Julia is faster on one processor is interesting, but not overly damning for R. The thing that is interesting to me is that as processors depend more and more on multicore for performance, technical computing problems are just about ideally positioned to take the best possible advantage, given the right language.

The other feature that will help that happen is indeed macros. The pace of the language is just intense right now. Macros let you build with bigger, cleaner building blocks. Looking at libraries is interesting but doesn't tell the whole picture. You need to look at the growth of libraries. Julia's trajectory is pretty much spot on here.

Clojure is interesting to some because there's no technical language that does what R can, so some look to a general purpose language to fill that void. I am actually a huge fan. But Clojure is a pretty serious brain warp. Clojure will be there for programmers who need to do technical computing. It won't be for engineers and scientists. There's just too much to learn.

So to me, Julia or something like it will absolutely replace R some day. It's a matter of time.

Answer 8

Every time I see a new language, I ask myself why an existing language can't be improved instead.

Python's big advantages are

• a rich set of modules (not just statistics, but plotting libraries, output to pdf, etc.)
• language constructs that you end up needing in the long run (objected-oriented constructs you need in a big project; decorators, closures, etc. that simplify development)
• many tutorials and a large support community
• access to mapreduce, if you have a lot of data to process and don't mind paying a few pennies to run it on a cluster.

In order to overtake R, Julia, etc., Python could use

• development of just-in-time compilation for restricted Python to give you more speed on a single machine (but mapreduce is still better if you can stand the latency)
• a richer statistical library

Answer 9

Julia will not take over R very soon. Check out Microsoft R open.

https://mran.revolutionanalytics.com/open/

This is an enhanced version of R that automatically uses all the cores of your computer. It is the same R, same language, same packages. When you install it, RStudio will also use it in the console. The speed of MRO is even faster than Julia. I do a lot of heavy-duty computing and have used Julia more than a year. I switched to R recently because R has a better support and RStudio is an awesome editor. Julia is still in early stage and possibly not catching up Python or R very soon.

Answer 10

Julia 1.0 has just come out with a very usable IDE (Juno). It came out a bit late to the party as Python has already dominated Machine Learning, while R continues to dominate every other kind of statistical analysis. That being said, Julia is already rising to prominence in the area of finance and trading algorithms as fast development time AND execution are a must. In my opinion, unless another language comes along that is distinctly better, Julia's rise to prominence will probably look something like this:

(1) It starts to eat MATLAB's lunch. MATLAB users like the MATLAB syntax but hate pretty much everything else. The slowness, the expensive licenses, the very limited ways to deal with complex data structures that are not matrices. I remember one quote where it is said that "If Julia replaces MATLAB, it will be a huge service to humanity". MATLAB users can become proficient in Julia very quickly and will be impressed by the ease it is to write quality code that does so much more than what MATLAB can do (Structs that are fast that you can put in arrays and quickly iterate over?). Not only this, researchers can make serious toolboxes in Julia (a small team Ph.D. students wrote a world-class differential equations package) that would have been impossible with MATLAB.

(2) It starts taking over research in numerical methods and simulation. MIT is throwing its weight behind Julia, and the research community listen's to MIT. Numerical simulations and new numerical methods are ill-defined problems that have no libraries. This is where Julia as a language shines; if there is no libraries available, it is much easier to write fast quality code in Julia than any other language. It will be a numerical/simulation language that is written by mathematicians for mathematicians (sound similar to R yet?)

(3) Another breakthrough in Machine Learning happens that gives Julia the edge. This is a bit of a wildcard which might not happen. TensorFlow is great, but it is extremely hard to hack. Python has already started showing cracks and TensorFlow has started adopting Swift (with Julia getting an honorable mention). If another machine learning breakthrough happens, it will be much easier to implement and hack in a Julia package like Flux.jl.

(4) Julia starts slowly catching up to R, which will take a while. Doing stats in MATLAB is painful, but Juila is already way ahead of MATLAB with Distributions.jl. The fact is, R workflows can be easily translated to Julia. The only real advantage R has is the fact that there are so many packages are written by statisticians for statisticians. This process however, is also easy to do in Julia. The difference is that Julia is fast all the way down and you don't have to use another language for performance (the more "serious" R packages are written in languages like C). The problem with R is that packages written in R are too slow to handle large sets of data. The only alternative is to translate the packages into another language making development in R a slower process than Julia. If too many R packages need translating to handle larger datasets, R may start playing catch-up with Julia in these areas.

Answer 11

The following probably does not deserve to be an answer, but it is too important to be buried as a comment to someone else's response...

I have not heard much said about memory consumption, just speed. R's entire semantics being pass-by-value can be painful, and this has been one criticism of the language (which is a separate issue from how many great packages already exist). Good memory management is important, as is having ways of dealing with out-of-core processing (e.g. numpy's memory mapped arrays or pytables, or Revolution Analytics' xdf format). While PyPy's JIT compiler allows for some striking Python benchmarks, memory consumption can be quite high. So, does anyone have experience with Julia and memory usage yet? Sounds like there are memory leaks on the Windows "alpha" version that will no doubt be addressed, and I am still waiting on access to a Linux box to play with the language myself.

Answer 12

I think it's unlikely that Julia will ever replace R, for a lot of the reasons previously mentioned. Julia is a Matlab replacement, not a R replacement; they have different goals. Even after Julia has a fully-fleshed out statistics library, no one would ever teach an Intro to Statistics class in it.

However, an area in which it could be incredible is as a speed-optimized programming language that's less painful than C/C++. If it were seamlessly linked to R (in the style of Rcpp), then it would see a ton of use in writing speed-critical segments of code. Unfortunately no such link exists currently:

https://stackoverflow.com/questions/9965747/linking-r-and-julia

Answer 13

I am a Julia newbie, and am R competent. The reasons I find Julia interesting so far are performance and compatibility oriented.

GPU tools. I'd like to use CUSPARSE for a statistical application. CRAN results indicate there's not much out there. Julia has bindings available which seem to work smoothly so far.

using CUSPARSE
N = 1000
M = 1000
hA = sprand(N, M, .01)
hA = hA' * hA
dA = CudaSparseMatrixCSR(hA)
dC = CUSPARSE.csric02(dA, 'O') #incomplete Cholesky decomp
hC = CUSPARSE.to_host(dC)

HPC tools. One can use a cluster interactively with multiple compute nodes.

nnodes = 2
ncores = 12    #ask for all cores on the nodes we control
procs = addprocs(SlurmManager(nnodes*ncores), partition="tesla", nodes=nnodes)
for worker in procs
    println(remotecall_fetch(readall, worker, `hostname`))
end

Python compatibility. There's access to the python ecosystem. E.g. It was straightforward to find out how to read brain imaging data:

import PyCall
@pyimport nibabel

fp = "foo_BOLD.nii.gz"
res = nibabel.load(fp)
data = res[:get_data]();

C compatibility. The following generates a random integer using the C standard library.

ccall( (:rand, "libc"), Int32, ())

Speed. Thought I would see how the Distributions.jl package perfomed against R's rnorm - which I assume is optimised.

julia> F = Normal(3,1)
Distributions.Normal(μ=3.0, σ=1.0)

julia> @elapsed rand(F, 1000000)
0.03422067

In R:

> system.time(rnorm(1000000, mean=3, sd=1))
   user  system elapsed 
  0.262   0.003   0.266 

Answer 14

I am interested by the promise of better speed and easy parallelisation using different architectures. For that reason I will certainly watch Julia development but I am unlikely to use it until it can handle generalised linear mixed models, the has a good generic bootstrap package, a simple model language for building design matrices the capability equivalent to ggplot2 and a wide range from machine learning algorithms.

No statistician can afford to have a fundamentalist attitude to the choice of tools. We will use whatever enables us to get the job done most efficiently. My guess is I will be sticking with R for a few years yet, but but it would be nice to be pleasantly surprised.

Answer 15

The luxury of NA's in R does not come without performance penalties. If Julia supports NA's with a smaller performance penalty then it becomes interesting to a segment of the stats community, but NA's also impose considerable extra work when using compiled code with R.

Many of the packages in R rely on routines written in legacy languages (C, Fortran, or C++). In some cases the compiled routines were developed outside R and later used as the basis for R library packages. In others the routines were first implemented in R and then critical segments translated to a compiled language when performance was found lacking. Julia will be attractive if it can be used to implement equivalent routines There is an opportunity to design low-level support for NA's in a way that simplifies NA handling over what we have now when using R with compiled code.

The massive number of R libraries represents the efforts of many many users. This was possible because R provided capabilities that weren't otherwise available/affordable. If Julia is to become widely used, it needs a group of users who find it does what they need so much better than the alternatives that is worth the effort needed to supply very basic things (e.g., graphics, date classes, NA's, etc.) available from existing languages.

Answer 16

I will be up front, I have no experience with R, but I work with plenty of people that think it is an excellent tool for statistical analysis. My background is in data warehousing, and due to Julia's easily distributed, but more standard programming model, I think it could be a very interesting substitute for the transform portion of traditional ETL tools that generally do the job very poorly , most have no way of easily creating a standardized transform, or re-using the results of a transform already performed on a prior data-set. The support for tightly defined and typed tuples stands out, if I want to build an OLAP cube that basically needs to build more detailed tuples (fact tables) out of tuples already calculated, today's ETL tools have no 'building blocks' to speak of that can help, this industry has worked around this issue through various means in the past, but there are trade-offs. Traditional programming languages can help by providing centrally defined transformations, and Julia could potentially simplify the non-standard aggregations and distributions common in more complex data warehouse systems.

Answer 17

You can also use Julia and R together. There is Julia-to-R interface. With this packages you can play with Julia while calling R whenever it has a library that would be needed.

Answer 18

Julia has without doubt every chance of becoming a statistics power-users dream come true, take SAS for example, it's power lies in the numerous procs written in C - what Julia can do is give you the procs with the source code, with matrices as a built in data type dispensing with SAS/iml. I have no doubt that statisticians will flock to Julia once they get a handle on just what this puppy can do.

Answer 19

Oh yes, Julia will overtake R quite quickly. And the primary reasons will be "macros", 95% of the language is implemented in Julia, and its noise free, parsimonious syntax. If you don't have experience with lisp type of languages you might not understand it as yet, but you will see pretty quickly how R formula interface will became an obsolete and ugly mechanism, and will be replaced by specialized modeling micro languages akin to CL loop macro. Access to low level references of an object is also a big plus. I think R still didn't get that hiding internals from the user actually complicates than simplifies the things.

As I see it now (having years of heavy use of R behind, and just finished reading Julia manual), Julia's main drawbacks with respect to R is no support for structural inheritance (this was intentional). Julia's type system is less ambitious than S4; it also supports multiple dispatch and multiple inheritance, but with a catch - there is only one level of concrete classes. On the other hand I rarely see class hierarchies in R deeper than 3 levels.

Time will tell, but it will be sooner than most R users think:)

Answer 20

Julia's first target use cases are numerical problems. Basically, you can break these analysis and computational science fields into data science (data driven) and simulation science (model driven). Julia is dealing with the simulation science use cases first. They are also dealing with the data science cases, but more slowly. R will never be very useful for simulation science, but Julia will be very useful for both in a couple of years.

Answer 21

It needs to be able to apply any function to large datasets that don't fit on memory transparently for the user.
That includes at least running mixed effects models, survival models or MCMC on datasets that fit on the disk but not on memory. And if possible on datasets distributed on several computers.