What does Theta mean? I am a newbie to statistics and found this.

In statistics, θ, the lowercase Greek letter 'theta', is the usual
  name for a (vector of) parameter(s) of some general probability
  distribution. A common problem is to find the value(s) of theta.
  Notice that there isn't any meaning in naming a parameter this way. We
  might as well call it anything else. In fact, a lot of distributions
  have parameters which are usually given other names. For example, it
  is common use to name the mean and deviation of the normal
  distribution μ (read: 'mu') and deviation σ ('sigma'), respectively.

But I still don't know what that means in plain English?
 A: What $\theta$ refers to depends on what model you are working with. For example, in ordinary least squares regression, you model a dependent variable (usually called Y) as a linear combination of one or more independent variables (usually called X), getting something like
$Y_i = b_0 + b_1x_1 + b_2x_2 + ... + b_px_p$
where p is the number of independent variables. The parameters to be estimated here are the $\beta s$ and $\theta$ is a name for all the $\beta s$. But $\theta$ is more general can apply to any parameters we want to estimate. 
A: It is not a convention, but quite often $\theta$ stands for the set of parameters of a distribution.
That was it for plain English, let's show examples instead.
Example 1. You want to study the throw of an old fashioned thumbtack (the ones with a big circular bottom). You assume that the probability that it falls point down is an unknown value that you call $\theta$. You could call a random variable $X$ and say that $X=1$ when the thumbtack falls point down and $X=0$ when it falls point up. You would write the model
$$P(X = 1) = \theta \\
P(X = 0) = 1-\theta,$$
and you would be interested in estimating $\theta$ (here, the proability that the thumbtack falls point down).
Example 2. You want to study the disintegration of a radioactive atom. Based on the literature, you know that the amount of radioactivity decreases exponentially, so you decide to model the time to disintegration with an exponential distribution. If $t$ is the time to disintegration, the model is
$$f(t) = \theta e^{-\theta t}.$$
Here $f(t)$ is a probability density, which means that the probability that the atom disintegrates in the time interval $(t, t+dt)$ is $f(t)dt$. Again, you will be interested in estimating $\theta$ (here, the disintegration rate).
Example 3. You want to study the precision of a weighing instrument. Based on the literature, you know that the measurement are Gaussian so you decide to model the weighing of a standard 1 kg object as
$$f(x) = \frac{1}{\sigma \sqrt{2\pi}} \exp \left\{ -\left( \frac{x-\mu}{2\sigma} \right)^2\right\}.$$
Here $x$ is the measure given by the scale, $f(x)$ is the density of probability, and the parameters are $\mu$ and $\sigma$, so $\theta = (\mu, \sigma)$. The paramter $\mu$ is the target weight (the scale is biased if $\mu \neq 1$), and $\sigma$ is the standard deviation of the measure every time you weigh the object. Again, you will be interested in estimating $\theta$ (here, the bias and the imprecision of the scale).
A: In plain English:
Statistical distribution is a mathematical function $f$ that tells you what is the probability of different values of your random variable $X$ that has the distribution $f$, i.e. $f(x)$ outputs a probability of $x$. There are different such a functions, but for now let consider $f$ as some kind of "general" function.
However, for $f$ to be universal, that is, one that is possible to apply to different data (that share similar properties), it needs parameters that change its shape so that it fits different data. A simple example of such a parameter is $\mu$ in normal distribution that tells where is the center (mean) of this distribution and so it can describe random variables with different mean values. Normal distribution has another parameter $\sigma$ and other distributions also have at least one such a parameters. The parameters are often called $\theta$, where for normal distribution $\theta$ is a shorthand for both $\mu$ and $\sigma$ (i.e. is a vector of the two values).
Why is $\theta$ important? Statistical distributions are used to approximate the empirical distributions of data. Say you have dataset of ages of a group of people and on average they are 50 years old and you want to approximate the distribution of their ages using a normal distribution. If normal distribution didn't allow for different values of $\mu$ (e.g. had a fixed value of this parameter, say $\mu=0$), then it would be useless for this data. However, since $\mu$ is not fixed, normal distribution could use different values of $\mu$, with $\mu=50$ being one of them. This is a simple example, but there are more complicated cases where the values of $\theta$ parameters are not so clear and so you have to use statistical tools for estimating (finding the most appropriate) $\theta$ values.
So you could say that statistics is about finding the best $\theta$ values given the data (Bayesians would say: given the data and priors).
