Why do we need so many dummy variables in a regression with categorical predictor? Why not use binary encoding instead of one-hot encoding? If we have $k$ categories of a categorical variable, why do we need $k-1$ dummy variables?
For example, if there are 8 categories, why don't we code them as
0 0 1
0 1 0
0 1 1
1 0 0
1 0 1
1 1 0
1 1 1?

Only 3 dummy variables would be needed as opposed to the commonly used $8-1 = 7$ dummy variables.
 A: The problem is not to encode the different id values of the categories.

if there are 8 categories, why don't we code them as
0 0 1
0 1 0
0 1 1
1 0 0
1 0 1
1 1 0
1 1 1?


Instead, the problem is to encode the different effects of the categories.

Your suggestion is to use only three dummy variables to model the effect as
$$\text{effect of category $=$ $\alpha_1$ (if category $=$ A, C, E, G) + $\alpha_2$ (if category $=$ A, B, E, F) + $\alpha_3$ (if category $=$ A, B, C, D)}$$
In this way you can indeed encode eight category values by a binary 3 bit binary number.
If we choose $\alpha_1 = 1$, $\alpha_2 = 2$ and $\alpha_3 = 4$ then we can get the values of the 8 id values
$$\begin{array}{}
A & 1\alpha_1 + 1\alpha_2 + 1\alpha_3 = 7 \\
B & 0\alpha_1 + 1\alpha_2 + 1\alpha_3 = 6 \\
C & 1\alpha_1 + 0\alpha_2 + 1\alpha_3 = 5 \\
D & 0\alpha_1 + 0\alpha_2 + 1\alpha_3 = 4 \\
E & 1\alpha_1 + 1\alpha_2 + 0\alpha_3 = 3 \\
F & 0\alpha_1 + 1\alpha_2 + 0\alpha_3 = 2 \\
G & 1\alpha_1 + 0\alpha_2 + 0\alpha_3 = 1 \\
H & 0\alpha_1 + 0\alpha_2 + 0\alpha_3 = 0 \\
\end{array}$$
But we are not interested in modelling the id-values 7,6,5,4,3,2,1,0. Instead we want to model the entire possible space of effects. That space is 8-dimensional and not 3-dimensional.
A: For one, the predictor variables would not be orthogonal. Generally, linear regressions work better (i.e., have tighter error estimates) when predictors are close to orthogonal (which can be quantified using the condition number of the Gram matrix $X^tX$). But the bigger issue with this proposed coding scheme is that it introduces similarity structure to the features that is completely arbitrary. For example the first row and the second row are less similar than the first row and the third row (as measured by, e.g., dot product similarity), but in a categorical variable the labeling of the values (or what is the same thing, the ordering of the rows of the encoding matrix) has no inherent meaning. So combining these two considerations, the only way that really makes sense to code a categorical variable with $K$ values is to choose $k$ orthogonal vectors, which requires a space of dimension $>=K$ (or really, $>=K-1$ since you can absorb one of them into the intercept). The typical coding using one-hotes is a particularly simple way to accomplish this, although in principle you could also use any set of $K$ orthogonal vectors.
A: You need a dummy variable for each level so that each level can have its own coefficient, independent of the other levels.
Think about how linear models work.  The predicted value is
$$
\hat{y}_i = \sum_j \beta_j x_{ji}
$$
With dummy variables (aka "one-hot encoding"), for each $i$, only one of the $\beta$, the one corresponding to the level of the categorical variable, is added to the sum.  Thus, each level gets its own coefficient that defines the outcome's response when the categorical variable takes on that value$^\star$.
Now, consider what happens with your proposed encoding.  Levels 1 and 2 each get their own response, but Level 3 is constrained to be the sum of the responses for levels 1 and 2.  There is no reason to suppose that this should be the case, so you shouldn't build that constraint into the model.  Having a separate dummy variable for each level allows each level to have its own response that does not depend in any way on the other levels.
$^\star$ Well, except for the one special level that gets its response rolled into the intercept term, but you shouldn't do that.  If there are $k$ levels, use $k$ dummy variables and no intercept instead of $k-1$ plus an intercept.  For example, if you're doing the model in R, use $y\sim x+0$ instead of $y\sim x$.
