2
$\begingroup$

i'm writing a few convenience functions for exploring datasets. For that, I have to guess the type of data from the values. What I'm looking for are some good rules of thumb.

Some easy cases:

  • ['spam', 'eggs'] is nominal, because the dataset contains strings.
  • [true, false, false] is nominal, too.
  • [0, 1, 0, 0, 0, 1, 1] is either nominal or ordinal.
  • [0, 1, 4, 3, 1, 2, 2] too, but it's probably ordinal. Many values, and they have numbers, but not too many.
  • [0.2, 2, 1.6, 3] is cardinal, as the level of precision suggests there could possibly be many other values.

As you see the judgment is absolutely vague, but that's not the point: I just need heuristics that work in 80% of the cases. Or better, 95%. More ideas?

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ Although your intentions are noble, this strikes me as being a spectacularly misguided use of data types: usually people use a type indication as a hint to the software about how to treat the data. Data do not have any inherent type: it depends on the purpose of the analysis and what the data mean. Software (at least to date) knows neither of those. Asking it to guess the type seems tantamount to having the software automatically limit its own capabilities! Please visit the thread at stats.stackexchange.com/q/25776 for a little more about this. $\endgroup$
    – whuber ♦
    Commented Apr 4, 2012 at 15:48
  • 2
    $\begingroup$ Thanks @whuber for the comment and the link! Of course I don't want to limit what the software can do, I merely want to provide some good default values. The default values are primarily used to output a rough description of your dataset (which variables are present? What are their ranges and means? Latter doesn't make sense for nominal or ordinal data, so treat that differently). So, yes, I know that data doesn't have an inherent type, the question is: how well can we guess what a user is probably going to do with the data from the data itself? $\endgroup$
    – Manuel Ebert
    Commented Apr 4, 2012 at 16:14
  • 1
    $\begingroup$ Ah, good. It sounds like you are thinking then about characterizing the nature of the data rather than assigning a "type". For instance, with $(0,1,0,0,0,1,1)$ you could report that this has (only) two distinct values (and even give a list of them, sorted according to the software's default assumptions), that they are all multiples of 1 (and therefore integral), that they are non-negative, but they do contain zeros, and no missing values appear. That would empower the user to go in an appropriate analytical direction. $\endgroup$
    – whuber ♦
    Commented Apr 4, 2012 at 16:18
  • 1
    $\begingroup$ Sorry, yes, that's what I want. I come from a machine learning background and am notoriously weak on definitions and terminology ;-) But more than just characterizing the nature of the data it's really more about given some characteristics, what would somebody likely want to know about that data? $\endgroup$
    – Manuel Ebert
    Commented Apr 5, 2012 at 9:32
  • 1
    $\begingroup$ I suspect--based on many personal attempts at mind-reading on this site!--that guessing what somebody would want to know about data is a difficult and uncertain activity, even when the characteristics of the data are in clear evidence. People have far more interests, applications, and cognitive models than can possibly be revealed in a mere set of data. To put it another way: what I may want to learn from a set of data might be completely different than what someone else might want to learn from it: this indicates the question at the end of your comment has no definite answer. $\endgroup$
    – whuber ♦
    Commented Apr 5, 2012 at 14:25

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.