How to indicate whether the improvement of a ranking is statistically significant over another one in R? I have two ranked lists of documents (List1, List2). After comparing each of them against the ground truth, I calculate precision at 5 (P@5) metric to measure how good are each of the ranked lists. List1 outperforms List2. 
Now, I would like to test and see if the improvement of List1 over List2 is statistically significant. Here are List1 and List2:
List1 = Doc10, Doc21, Doc2, Doc3, ...
List2 = Doc2, Doc10, Doc3, Doc21, ...
I have looked into the literature, and usually they do t-test for this matter, but I couldn't find any tutorials which explains how to do it in a tool such as R. I will appreciate if anyone can help me.
EDIT: In order to give you an example of a scientific paper in my area which has done it, please check this paper: http://romaindeveaud.github.io/publis/cikm15-deveaud.pdf
If you check the caption for Table 2, it says:

We use a two-sided pairwise t-test (with p < 0.05) to determine statistically significant differences over: (1) a random baseline that generates random category
  distributions ...

Note that each method returns a ranked list for each given query (or given user). Then, for each query in each system P@5 is calculated and each system's overall performance is the average of all P@5 values for each query. As an example:
           System 1          System 2
Query 1    P@5=0.5           P@5=0.6
Query 2    P@5=0.6           P@5=0.7
Query 3    P@5=0.7           P@5=0.8
---------------------------------------
Overall    P@5=0.6           P@5=0.7

 A: If you store your P@5 metrics in a couple of R vectors, say rs1 and rs2 you can carry out the t-test comparing their mean values as:
t.test (rs1, rs2)

This will compute a two sides unpaired test assuming unequal variances.
In your case, it seems that your alternative hypothesis is just one side so you can use the alternative parameter in the t.test function
t.test (rs1, rs2, alternative = "greater")

that will test for the alternative hypothesis that the mean value in rs1 is greater than the mean value in rs2 (check if this is the direction you need... it assumes that the greater the score the better the ranking)
You may also want to assume that the variance in both scores is the same. The use:
t.test (rs1, rs2, alternative = "greater", var.equal = TRUE)

Finally, if your samples are paired (I guess this means that your two lists contain different scores but for the same documents), you can use the paired parameter:
t.test (rs1, rs2, alternative = "greater", paired = TRUE)

Notice that in the paired analysis rs1 and rs2 should be in the same "order" meaning that each position in both vectors should refer to the same document... If you have named vectors in your R session you can reorder rs2 as rs1 doing:
rs2 <- rs2[names (rs1)]

I hope this helps
A: Let me try to answer on two levels.
1) How to compare rankings? To compare the full rankings of the two systems you need a) the correct ranking, or a ground truth ranking and b) you need a measure that tells how tow rankings are similar. This question Can I compare ordinal rankings (and if so, how)? on CV discusses 2 metrics to compare rankings. Notice that in this case you will be comparing the ranking of system 1 with the ground truth and then the ranking of system 2 to the ground truth, for each query or user. Then you will have something similar to the figure in your question, but instead of P@5, you may have the Kendal tau or the Spearman rho. 
But you may have settled in using P@5, which is not really comparing the full ranking of the two systems, but only how right they are on the first 5 entries. It is OK, if the publications in your area use P@5, you should use P@5. 
Which brings us to the second level, given that you have the sequence of measures (either Kendal tau, or Spearman rho or P@5) for System 1 and for System 2, for each query, how you should perform the test
2) How to perform a significance test on two sequences of measures for System 1 and System 2? 
Here, @dmontaner is basically correct and describes the mechanics of using the functions on R, but I will be a little bit more incisive.
If you are going to use a t-test than you should use this solution proposed by @dmontaner
t.test (rs1, rs2,  paired = TRUE)

your data is DEFINITELY paired by the query (or user). Methodologically you should not use the 
alternative = "greater"

option unless you have theoretical (not empirical) reasons to believe that your system is better than the competition. 
But there is an issue of whether you should or not use  t-test. If researchers in your area use t-test, if that paper you pointed out was a good paper in a good conference, that by all means use the t-test. But the t-test makes assumptions on your data that may not be true. For example, the t-test assumes your data is more or less distributed normally, and this is more important the less data you have. If you have less than 30 queries, that your data should be distributed very similar to a normal. But P@5 is usually not normally distributed (at least because it is limited to the 0-1 range). The same can be said about the other two measures (tau e rho). In this case you should use the Wilcoxon signed rank test. Fortunately there is a R function for that, and you only call 
    wilcox.test (rs1, rs2,  paired = TRUE)

