Here are all the different ways of pairing one sock with another different one, arranged systematically:
"Random" in this context means that each pairing is equally likely. Therefore, to find the probability of a color match, count the color-matched pairs and divide by the total.
Counting is made easier by noting that the color-matched socks occur within squares of $6$ by $6$, $4$ by $4$, and $2$ by $2$, and indeed the full tableau of pairings is a square of $12=6+4+2$ by $12$. Within each square the diagonal is missing. Thus, the number of pairs within any such square-without-diagonal of sides $k$ must be $k\times k - k = k(k-1)$. Sum these values over the red socks ($k=6$), green socks ($k=4$), and blue socks ($k=2$), finally dividing by the total ($k=12$).
A general formula for any numbers of various kinds of socks follows immediately.