# Can categorical data only take finitely or countably infinitely many values?

I wonder if categorical data by definition can only take finitely or countably infinitely many values? And no more i.e. not uncountably many values?

Related question: is the distribution of a categorical variable always a discrete distribution or a continuous distribution?

Thanks and regards!

-
What do you mean by "Countably infinite"? Using the definition from here: mathworld.wolfram.com/CountablyInfinite.html I can't actually conceive of an applied example. Do you just mean a variable that can range toward + or - infinity? –  EpiGrad Oct 15 '11 at 7:35
No. I mean the cardinality of the set of the values which the categorical variable for the categorical data can take. –  Tim Oct 15 '11 at 8:03
Not sure that added clarity ;) I'm going to take a swing anyway. –  EpiGrad Oct 15 '11 at 8:05
I believe that the question should be rephrased can there be finite levels (or countably infinite) for a categorical variable. Categorical data can include infinite values - there's no reason to assume that an observation has a given cardinality. –  Iterator Oct 16 '11 at 22:22

"Categorical" is not a well-defined mathematical term, so to answer this question we have to look to how this word is intended to be used. It is employed in contrast to "ordinal," "interval," and "ratio." One way to understand the primary distinctions is in terms of the groups of allowable re-expressions of the values. In the case of the latter three, there is an order that must preserved, whence all re-expressions must be monotonic (order preserving). For categorical variables, any bijection (including permutations) is ok:

Beyond that, anything goes with the nominal [categorical] scale.

(Stevens, quoted in the Wikipedia article.)

Another concept of "categorical" is that each outcome must be distinguishable from every other. This strongly suggests that any probability measure must be totally discrete: that is, all subsets are measurable, implying that each category will have its own well-defined probability. (This is not the case for continuous distributions.)

This would seem to indicate that the number of categories should be finite or at most countable, but that is not evident in the literature. For instance, an archetypal example of a categorical variable is a set of names. The set of all possible names on any finite alphabet is countable but not finite. It is therefore useful to allow countably infinite sets to be categorical. For example, if we are studying names given to babies, it is convenient to let the sample space consist of all possible names (rather than all names that we know of).

A slightly less realistic, but still conceivable, example of a categorical variable would be one that uses real numbers for names. In effect, such a variable would ignore all the usual mathematical structure on this set. I don't see any problem with such a usage, but it's worth observing that the axioms of probability imply that any probability distribution valid in this context would (a) assign a non-negative value to each real number and (b) would assign a non-zero value to at most a countable infinity of the reals.

One application involving an uncountable sample space that supports categorical random variables of infinite, even uncountable, support is the study of random graphs. To understand the rate of growth of some property of graphs, we would want to contemplate graphs on 0, 1, ..., $n$, ... nodes, so it's convenient to allow graphs to have countably many nodes. Random variables defined on this set can have various types. For instance, the mean vertex degree (if finite) could be considered of ratio type; the total vertex degree could be considered of ordinal type (and, therefore--by forgetting the ordering--is a nice example of a countable discrete variable). If we also allow a graph to have arbitrarily many edges and are interested in, say, its connected components, then we would have a naturally occurring category that is uncountable (because each connected component determines the subset of nodes it contains and there are uncountably many subsets of a countable set).

To summarize, it is reasonable to allow categorical values to attain an uncountable infinity of possible values, while recognizing that at most a countable number of them could ever have positive probabilities. This must be a discrete distribution, because all subsets are measurable, which is not the case for continuous distributions.

-
"it is convenient to let the sample space consist of all possible names (rather than all names that we know of)" - how is that convenient? –  EpiGrad Oct 16 '11 at 21:30
I'm not sure why you say that "categorical" is not well-defined mathematically. I believe that you may need to clarify the meaning of category, level, and data. These terms must be defined mathematically for mathematical statistics to be relevant. :) Nonetheless, it's quite an interesting answer. –  Iterator Oct 16 '11 at 22:29
@Iterator I have no disagreement with you and appreciate your remarks. To clarify where I'm coming from, it appears to me that the scale "categorical, nominal, interval, ratio" is not a mathematical one, but rather is a qualitative description. One could indeed come up with mathematical definitions that capture most of the concept of "categorical," but that might even be counterproductive. I do not view this hierarchy as being any part of mathematical statistics, and in fact find it to be useless except as a suggestive guide in some applications. –  whuber Oct 16 '11 at 23:46
@whuber It sort of matters for issues like constructing hypothesis tests, and then for applications such as estimating necessary sample sizes. :) In any case, the OP was vague. Categorical is an adjective, which noun it modifies is the issue to address. Whether that noun is countable or describes something countable is the source of ambiguity. I am contemplating editing the question to reflect something that's actually answerable, because the vagueness yields an unanswerable question. –  Iterator Oct 16 '11 at 23:56
@Iterator I think you're on to something here. I have been wondering whether "categorical" refers to a sample space or to a random variable or to neither. In his original article, Stevens addresses scales of measurement. As such, it would seem he was interested in an additional property of a random variable. I would characterize this extra property in terms of the group of transformations of $\mathbb{R}$ that would be considered inconsequential for the data analysis. However, I'm unaware of any published definition of this sort. –  whuber Oct 17 '11 at 13:36

"Categorical data" is something of a fuzzy and problematic term to begin with. Similar to the "I know it when I see it" definition of obscene material. There are some very clear cases of categorical data, where the values of a variable fall into a small number of clearly defined categories.

Beyond that, there be dragons.

At some point you get enough categories that your "categorical" variable could likely be treated as a continuous variable. Or alternately, using either subject matter knowledge or a description of a distribution, you can break a continuous variable up into categorical chunks and treat it as categorical.

Theory Answer: No. You could have infinitely many categories, but for some reason decide not to call it a continuous variable. If you allow for decimal-based categories of an utterly unbounded variable, there's no reason I can see it wouldn't be be uncountably infinite.

I'm not sure how often this ends up coming up. In my experience at least, fairly rarely.

Applied Answer: The cardinality of most things that would ever reasonably be called categorical data have a cardinality vastly below the cardinality of N. As noted above, there are exceptions, often subject to vague judgement calls.

-
Thanks! So the silly answer is that the cardinality can be any, while the applied answer is that mostly it is finite? –  Tim Oct 15 '11 at 8:20
@PeterFlom: Cantor showed that the real numbers are uncountable. :) –  cardinal Oct 16 '11 at 2:05
@EpiGrad: I don't consider your "silly" answer quite so silly. There is no particular reason I see that an infinite number of categories should somehow be equated with some form of continuous random variable. Indeed, the difference seems to hinge on whether or not the variable can be represented in a meaningful, natural, and unambiguous way as a real number. –  cardinal Oct 16 '11 at 2:11
What would "vastly below the cardinality of N" mean? Is this just a long way of saying "finite"? If so, that would rule out many interesting applications involving categorical variables. –  whuber Oct 16 '11 at 15:36
Hi @cardinal yes, I know Cantor showed the reals are uncountable. But is there any set of categories that needs more than the rationals, which are countable? I have trouble imagining a set of categories that requires use of irrationals. –  Peter Flom Oct 17 '11 at 11:07
I believe that the usage is ambiguous. Categorical variables have levels. Data is merely observations of a categorical variable, with some specified level. One could abuse the notion of levels to produce infinitely many. A level itself could, say, count the infinities encountered in a data set (e.g. $X/Y$ where both are drawn from the two element set ${0,1}$. I do not disagree with your answer, but suggest clarifying to address the issue of levels (or categories). –  Iterator Oct 16 '11 at 22:27