I have a data frame in python where I need to find all categorical variables. Checking the type of the column doesn't always work because int type can also be categorical.
So I seek help in finding the right hypothesis test method to identify if a column is categorical or not.
I was trying below chi-square test but I am not sure if this is good enough
import numpy as np
data = np.random.randint(0,5,100)
import scipy.stats as ss
ss.chisquare(data)
Please advise.
Short answer: you can't.
There is no statistical test that will tell you whether a predictor that contains the integers between 1 and 10 is a numeric predictor (e.g., number of children) or encodes ten different categories. (If the predictor contains negative numbers, or the smallest number is larger than one, or it skips integers, this might argue against its being a categorical encoding - or it may just mean that the analyst used nonstandard encoding.)
The only way to be sure is to leverage domain expertise, or the dataset's codebook (which should always exist).
Whatever criteria -- or rules of thumb -- work for your dataset are welcome to you, but we can't see your data. In any case the problem is better pitched generally, and without reference to any particular software either.
It's worse than you think, even if you think it's worse than you think.
@Stephan Kolassa's answer already makes one key point. Small integers could mean counts rather than categories: 3, meaning 3 cars or cats, is not the same as 3, meaning "person owns a car" or "person is owned by a cat".
Decimal points could lurk within categorical variables, as part of coded classifications, e.g. of industries or diseases.
Measurements strict sense could just be integers by convention, e.g. heights of people may just be reported as integer cm or inches, blood pressures as integer mm Hg.
The number of distinct (a better term than "unique", which still has the primary meaning of occurring just once) values is not a good guide either. The number of different heights of people possible in moderate samples is probably much less than the number of different religious affiliations or ethnic origins.
Negative codes or a mix of low and high numbers may mean little or nothing. For example, there are many survey conventtions to choose from for indicating some kind of missing value (e.g. "not applicable", "no reply", "irrelevant answer", "incomprehensible answer"), which may be coded with numbers like -88, -99 or 999. It could be hard to tell whether such a variable was categorical or something else without access to a code book.
Well I think it's even worse than the other answers suggest: data aren't categorical or numeric sub specie æternatis—"level of measurement" is something stipulated by the analyst to answer a particular question on a particular occasion. See Glen_b's answer here.
It's of practical importance to understand that. For example, with a classification tree the distinction between ratio, interval, & ordinal level predictors is of no consequence: the only distinction that matters is that between ordinal & nominal predictors. Constraining the algorithm to split the predictor at a point along a line, separating higher from lower values, can have a significant effect on its predictive performance—for good or ill, depending on the smoothness of the (putatively ordinal) predictor's relation to the response & the size of the data-set. There's no sensible way to make the decision based solely on musing about how the predictor variable represents reality irrespective of the analysis you're about to undertake, let alone on what values you've found it takes in a sample.
This is an open research question. See for example the work by Valera et al. (paper) or extensions (e.g. one by Dhir et al. - paper).
Edit:
A common practice in statistics and machine learning is to assume that the statistical data types (e.g., ordinal, categorical or real-valued) of variables, and usually, also the likelihood model is known. However, as the availability of real-world data increases, this assumption becomes too restrictive. Data are often heterogeneous, complex, and improperly or incompletely documented. Surprisingly, despite their practical importance, there is still a lack of tools to automatically discover the statistical types of, as well as appropriate likelihood (noise) models for, the variables in a dataset.
(From the Valera paper.)
So when we say that this is an "open question" (oddly enough quoting myself), we mean to say that currently there are no good automatic methods for inferring the type of data given a finite sample. If you had an infinite sample this would be easy, but since that is not possible, we need to revert to other means.
