How is the chance-level confusion matrix calculated? I applied an ML technique on my dataset, and got this confusion matrix:
       0    1

0    162   62
1     27   50

Funnily, the overall accuracy is worse than predicting all to be zeros (note how the dataset is unbalanced). However, I think such model would be pretty useless. I'd like to find 1's as well as possible. How can I show that my model is better than one would get by chance?
 A: When datasets are unbalanced, accuracy is not always very useful.
One standard thing to consider is precision and recall; if you want a single number, the most common is probably the $F_1$ score, but the "right" thing to do depends on your problem setting.
If your classifier can output confidence scores, it's also common to consider performance as the cutoff for choosing one class varies, by the receiver operating characteristic (ROC) curve. To get a single number, people often use the area under that curve (often called the AUC, though that abbreviation just means "area under the curve" in general).
A: @Dougal has provided a good answer.  You would probably be better off using one of the metrics he suggests.  To answer your explicit question, I'll assume the true category labels are the columns, then:  
tab = as.table(rbind(c(162, 62),
                     c( 27, 50) ))
p   = margin.table(tab, 2)/sum(tab); p 
#        A        B 
# 0.627907 0.372093 

So if you were to predict A vs. B by chance alone, the probabilities would be:  
p%o%p
#           A         B
# A 0.3942672 0.2336398
# B 0.2336398 0.1384532

That matrix is constructed using $p(A)^2$, $p(A)p(B)$, and $p(B)^2$.  Scaled by your sample size it is:  
(p%o%p)*sum(tab)
#           A        B
# A 118.67442 70.32558
# B  70.32558 41.67442


As your classes become more imbalanced, it is often the case that you can get good accuracy by simply predicting the majority class.  It also becomes more difficult for your model to outpredict / be more accurate than this simple strategy.  That doesn't mean your model is worse than chance alone.  From a statistical perspective, the question of whether your model's predicted classes accurately match the true classes is one of agreement.  You can test for agreement using Cohen's $\kappa$:  
library(irr)
d   = as.data.frame(tab)
d2  = c();  for(i in 1:4){ d2 = c(d2, rep(d[i,1:2], times=d[i,3])) }
d2  = matrix(unlist(d2), ncol=2, byrow=T)
head(d2, 3)
#      [,1] [,2]
# [1,] "A"  "A" 
# [2,] "A"  "A" 
# [3,] "A"  "A" 
table(d2[,1], d2[,2])
#     A   B
# A 162  62
# B  27  50
tab
#     A   B
# A 162  62
# B  27  50
kappa2(d2)
# Cohen's Kappa for 2 Raters (Weights: unweighted)
# 
#  Subjects = 301 
#    Raters = 2 
#     Kappa = 0.324 
# 
#         z = 5.83 
#   p-value = 5.39e-09 

This result suggests there is more agreement than would be expected by chance alone.  
