0
$\begingroup$

I'm storing files and giving them random numbers as the name, using a 32 bit unsigned range, and writing it as hex. Eg: 087b8a08.

To avoid having too many files in any one directory I split the name like: 08/7b/8a/08.

I've noticed that at the top level I have a reasonable number of file names, but each level down that I go there are fewer and fewer files in each directory. (Since I obviously don't store every possible name, just a small subset, eg 10,000, or a 1 million, out of a possible 4 billion.)

What's a good strategy to divide up the hex-digits to have approximately the same number of names at each level, given: an expected number of files that I store, and how many level to divide into?

I read that dividing by e (rounded to 3) each time minimizes the number of items in a level vs the number of levels. But I think that doesn't apply here since I don't use every number name.

Improve this question
$\endgroup$
8
  • 2
    $\begingroup$ Why on earth would you name files like this? Most programming languages have build in functions to uniquely assign file names without the risk of random duplication, however small. Why is there a concern of having too many names in a directory? Have you given any thought to writing your files to a database, as they are particularly well suited for tasks like this? $\endgroup$
    – StatsStudent
    Commented May 12, 2016 at 3:01
  • 2
    $\begingroup$ @Stats In many operating systems, having large numbers of files in a directory degrades file searches and thus impairs all operations requiring access to files. Having more than ten or so files per directory slows down human navigation considerably. Although the database idea is a good one, if those files need to be available directly to other software it might not be an option. $\endgroup$
    – whuber ♦
    Commented May 12, 2016 at 13:53
  • $\begingroup$ Well certainly I can appreciate the issue of possible degradation of performance. There are many of programs written to do do exactly what the OP is asking without having to rely on possibly dupblicating file names. I'd recommend the OP check out stackoverflow.com/questions/197162/… and consider posting there. I view this more as a computer/program issue, rather than a statistical one, really. $\endgroup$
    – StatsStudent
    Commented May 12, 2016 at 15:08
  • $\begingroup$ I also mean to include: serverfault.com/questions/98235/… $\endgroup$
    – StatsStudent
    Commented May 12, 2016 at 15:14
  • 1
    $\begingroup$ @Ariel, I don't necessarily agree with your comments regarding the database, but I'll assume you know what is best for your particular situation. That being said, maybe I'm just not understanding what you are trying to do, but it seems like maybe you are making things too hard for yourself. Rather than using hex digits, why not simply use sequential integers, and write, say the first n, files to director_n, the next n files to directory_n*2? If you are simply interested in properties of random digits, I think you can make the question a more clear by removing the superfluous background info. $\endgroup$
    – StatsStudent
    Commented May 13, 2016 at 3:18

1 Answer 1

Reset to default
0
$\begingroup$

You have 32 bits, and assuming you have a uniform distribution, the expected size is halved with every bit that you use. It's straightforward to estimate the number of files at each level. But I suggest you don't overthink this. Use either ab/abcdef01 (reduce by 256) or ab/cd/abcdef01 (reduce by 256^2) because you probably won't get as much benefit from any further nesting as you expect. At some point the actual number of files and data is the problem, not only in one directory.

E.g. the well-known squid proxy cache uses 16*256 folders (so probably a/bc/. I doubt it uses random numbers (that would be rather useless) but most likely it uses a hash code of the web host name.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.