Let's clarify the process of minimizing the Residual Sum of Squares (RSS) function, step by step. The function we want to minimize is given by $RSS(\beta) = (\mathbf{y} - \mathbf{X}\beta)^T(\mathbf{y} - \mathbf{X}\beta)$. This is a scalar function where $\mathbf{y}$ is an $n \times 1$ vector, $\mathbf{X}$ is an $n \times p$ matrix, and $\beta$ is a $p \times 1$ vector. Our goal is to find $\frac{\partial RSS(\beta)}{\partial \beta}$ and set it to zero to find the minimizing value of $\beta$. We start by expanding the RSS function: $$RSS(\beta) = (\mathbf{y} - \mathbf{X}\beta)^T(\mathbf{y} - \mathbf{X}\beta).$$ This can be rewritten as $$RSS(\beta) = (\mathbf{y}^T - \beta^T \mathbf{X}^T)(\mathbf{y} - \mathbf{X}\beta).$$ Distributing the multiplication, we obtain $$RSS(\beta) = \mathbf{y}^T\mathbf{y} - \mathbf{y}^T\mathbf{X}\beta - \beta^T\mathbf{X}^T \mathbf{y} + \beta^T \mathbf{X}^T \mathbf{X} \beta.$$ Notice that $\mathbf{y}^T\mathbf{X}\beta$ is a scalar, and its transpose $(\mathbf{y}^T\mathbf{X}\beta)^T = \beta^T \mathbf{X}^T \mathbf{y}$ is the same scalar. Therefore, $\mathbf{y}^T\mathbf{X}\beta = \beta^T \mathbf{X}^T \mathbf{y}$, and we can simplify the RSS function to $$RSS(\beta) = \mathbf{y}^T\mathbf{y} - 2\beta^T \mathbf{X}^T \mathbf{y} + \beta^T \mathbf{X}^T \mathbf{X} \beta.$$
Next, we differentiate with respect to $\beta$. When differentiating a scalar with respect to a vector, we use standard matrix calculus results. Specifically, we use the results that $\frac{\partial}{\partial \beta}(\beta^T A \beta) = (A + A^T)\beta$ (which simplifies to $2A\beta$ when $A$ is symmetric) and $\frac{\partial}{\partial \beta}(\beta^T b) = b$. In our case, $A = \mathbf{X}^T\mathbf{X}$, which is symmetric. So, differentiating term-by-term, we have $\frac{\partial}{\partial \beta}(\mathbf{y}^T\mathbf{y}) = 0$, $\frac{\partial}{\partial \beta}(- 2\beta^T \mathbf{X}^T \mathbf{y}) = -2 \mathbf{X}^T\mathbf{y}$, and $\frac{\partial}{\partial \beta}(\beta^T \mathbf{X}^T \mathbf{X} \beta) = 2\mathbf{X}^T \mathbf{X}\beta$. Combining these gives us the derivative: $$\frac{\partial RSS(\beta)}{\partial \beta} = -2 \mathbf{X}^T\mathbf{y} + 2\mathbf{X}^T \mathbf{X}\beta$$ or equivalently, $$\frac{\partial RSS(\beta)}{\partial \beta} = 2 \mathbf{X}^T (\mathbf{X}\beta - \mathbf{y}).$$ To find the minimizing $\beta$, we set the derivative to zero: $$2 \mathbf{X}^T (\mathbf{X}\beta - \mathbf{y}) = 0.$$ Dividing by 2 yields $$\mathbf{X}^T (\mathbf{X}\beta - \mathbf{y}) = 0,$$ which can be rewritten as $$\mathbf{X}^T(\mathbf{y} - \mathbf{X}\beta) = 0.$$ The equation you mentioned, such as $$\mathbf{X}^T(\mathbf{y}-\mathbf{X}\beta) + (\mathbf{X}^T(\mathbf{y}-\mathbf{X}\beta))^T = 0,$$ is not necessary and introduces confusion. The gradient is a vector, derived directly from matrix calculus rules. There's no need to add its transpose or apply a symmetry argument because the gradient is a $p \times 1$ vector, not a matrix that needs to be symmetric. We are setting the gradient vector equal to the zero vector. The confusion can arise from attempting to derive the gradient by manual product rule applications instead of utilizing known matrix calculus results. The result, $\mathbf{X}^T(\mathbf{y}-\mathbf{X}\beta)=0$, follows directly from setting the gradient to zero, and no extra symmetry condition is required.
