Suppose I am modelling a binary outcome $y$ with $n$ covariates. If this model has the lowest AIC among all the other ones...would this be the one to use? Also in dealing with categorical variables, should you delete all levels or none? For example suppose a categorical variable $z$ has the following levels: $z_1, z_2$ and $z_3$. If $z_1$ is the reference level then I include $z_2$ and $z_3$ in the model. If I don't want to include the variable $z$ in the model....would I delete both $z_2$ and $z_3$? Or could I include one of them?
You are implying that it is good to remove variables/levels from a model. Using $Y$ to help decide how to model $X$ results in biased $\beta$s, standard errors that are too small, $P$-values that are too small, and confidence intervals that are too narrow. In the case of OLS it also results in an estimate of $\sigma^2$ that is too small. Removing levels of categorical predictors is another multiple comparisons problem. Think about doing an ANOVA on a 5-level factor then doing $t$-tests to decide which levels to pool, then doing an ANOVA on 3 levels (this is almost exactly what you are implying should be done through the use of dummy variables). The first ANOVA has a perfect multiplicity adjustment built into it, and the second will not come close to preserving type I error.
The AIC (and other, similar metrics) cannot tell you definitively which model to use, but is a very helpful guideline. In general, if the AIC for the full model is lowest, and if the AIC for other possible models is not really in the same ballpark, then there is a strong case to be made that the full model is the appropriate one.
As for factors, a factor with l levels burns l-1 degrees of freedom. These levels / parameters should be treated as a unit (e.g., tested together when testing a nested model without that factor vs. the model with that factor). If you decide to drop a factor from the model, you drop all the levels. Since a factor is represented by l-1 dummy coded vectors, that means you drop those l-1 predictors from the model, in your example, z2 and z3.