what it means, or, what the latin square does
The orthogonality of columns $i$ and rows $j$ means that their effect is being removed from the expectation values for some treatment $k$ (A,B,C,D).
See the formula (for a model without cross effects)
$Y_{ijk} = \alpha + c_i + r_j + \beta_k + \epsilon_{ijk}$
whose expectation for a certain level of $k$ (A, B, C or D) becomes the following
$E(Y_{ijk} \vert k) = \alpha + \beta_k$
provided that the treatment does not correlate (is orthogonal to) with the rows and columns.
the treatment for A (and similarly for B, C and D) is tested the same number of times in each row and so you can eliminate (average out) the effect of the row on the expectation value of the treatment A.
orthogonality
I am not sure if this is the origin of the etymology but this is what I imagine with orthogonality
In the example you have the following tests (column, row, treatment):
1,1,A
1,2,B
1,3,C
1,4,D
2,1,B
2,2,C
2,3,D
2,4,A
3,1,C
3,2,D
3,3,B
3,4,A
4,1,D
4,2,A
4,3,B
4,4,C
if you take this as a matrix $M$ and calculate $M^TM$ then you obtain in the non-diagonal elements a sum of products in which each term occurs the same number of times.
for example the product of the first and third column
$(1,1,1,1,2,2,2,2,3,3,3,3,4,4,4,4) \cdot (A,B,C,D,B,C,D,A,C,D,A,B,D,A,B,C) = (1+2+3+4)(A+B+C+D) = 16 \mu_i \mu_j $
and this property may be associated with orthogonality of columns in a matrix