Is there a difference between $\beta$ and $\theta$? I've seen both $\beta$ and $\theta$ used to indicate model parameters in different publications. For example, Andrew Ng uses $\theta$ in his ML course and Gareth James et al use $\beta$ in ISLR. 
My understanding is that $\beta$ is used to refer to specific parameters while $\theta$ refers to general parameters, but I often see them used interchangeably.
Are they synonyms or do they have distinct meanings? 
 A: Those symbols do not have any specific meaning, what they mean should follow from context. Usually, in statistics we use Greek letters to denote parameters, but even this is not a general rule followed by everyone. $\beta$ and $\theta$ are popular choices by convention. Also by convention $\beta$ is popular choice for regression parameters. 
A: $\theta$ is the Greek letter used in the most generality for some parameter (could be a vector), which could include regression coefficients.
$\beta$ would be specific to regression coefficients.
Unless something specifically deals with regression coefficients, I think the preference would be to use $\theta$. Even in a context where the parameter could be regression coefficients but also could be a more general parameter, I think that $\theta$ would be preferable. If I see $\beta$, my mind is going into regression mode and then will get confused if there is no regression to be found.
These are just names and can be used however you want ("a rose by any other name..."), but these seem to be the conventions in statistics that will facilitate the easiest communication.
A: A search of my electronic copy of ISLR shows the symbol $\beta$ as appearing on 99 pages. A quick look at the context in which that symbol appears on those pages indicates that all uses of that symbol represent coefficients in linear forms, in contexts of: standard linear regression, logistic regression, penalized regressions, and support vector machines. That is consistent with my experience and with what @Tim notes in another answer: $\beta$ is a common choice for regression coefficients.
