There are imputation strategies which respect the ordinal nature of your data.

1. You could fill in the missing data with the mode (rather than the mean) of the non-missing data.
2. You can fill in the missing data by sampling from the non-missing data with probabilities proportional to the frequency of occurrence (possibly repeating this many times).
3. A "hot deck imputation" approach, where the data is filled in using data from a similar unit.
4. For a regression approach, you can use ordinal regression (rather than linear) to estimate the missing data

There are imputation strategies which respect the ordinal nature of your data.

1. You could fill in the missing data with the mode (rather than the mean) of the non-missing data.
2. You can fill in the missing data by sampling from the non-missing data with probabilities proportional to the frequency of occurrence (possibly repeating this many times).
3. A "hot deck imputation" approach, where the data is filled in using data from a similar unit.
4. For a regression approach, you can use ordinal regression (rather than linear) to estimate the missing data

More to the point of your question, a non-integer value like $6.34$ may or may not be inappropriate. This largely depends on how you plan to use the data and what statistical methods you plan to use. In my opinion, it is best to use strategies which maintain the desired structure of the data.