Many people claim to use ReLU to solve vanishing gradient problem, but wouldn't dying ReLU be a more serious problem? And someone also claims that ELU performs better, but wouldn't ELU also suffer from vanishing gradient? PReLU seems to avoid the problem, but it is not very popular. What's the catch?
ELU and ReLU both have zero or vanishing gradient "on the left". This is still a marked departure from $\tanh$ or logistic units, because those functions are bounded above and below; for ELU and ReLU units, the gradient updates will be larger "on the right". As a demonstration, work out the derivatives for each and note that the logistic and $\tanh$ units usually have smaller gradients for inputs in some interval around 0 such as $ [-2,2]$ than ELU and PReLU; $\tanh$ only attains a gradient of 1 at zero, and the logistic unit not at all! On the other hand, ReLU/ELU/PReLU have gradient 1 for all positive inputs.
On the other hand, you're correct that PReLUs avoid having a zero gradient everywhere. I'm not aware of a study exhaustively comparing ELU, ReLU and PReLU units. There's still a long way to go between these practical innovations in neural networks and a theoretical understanding of why they work well.