Factorial in factorial design doesn't refer to the mathematical operation $n!$ during the analysis but instead is named for using factors (or categories) in the experiment. In R factors are represented as a vector of integers that are displayed with character values (e.g. some descriptive text). A factor usually comes with multiple levels. For example the factor
Fertilizer can have 3 levels, such as
In statistics, a full factorial experiment is an experiment whose design consists of two or more factors, each with discrete possible values or "levels", and whose experimental units take on all possible combinations of these levels across all such factors.
The terms factor (in context of experimental design) as well as factorial design were first mentioned in print by Ronald Fisher (see Wikipedia entry here).
The term "factorial" may not have been used in print before 1935, when Fisher used it in his book The Design of Experiments.
So in order to figure out why Fisher chose the term factorial for factorial designs, we should grab a copy of the book. But I am guessing it has something do with multiplying factor levels in order to get to the total number of treatment combinations in an experiment. But I am not sure.