I'm writing a duplicate file finder application, and I do not want to have to read every byte of every file to determine which files are duplicates.
Hard drive storage is segmented into clusters of X number of bytes. For example, on Windows, the average cluster size is 4096 Bytes (or 4K).
Therefore, my current algorithm first groups all the files by size. Then for each same-size-file-set, I calculate the total number of clusters, then randomly select 5 of these clusters and calculate a hash for each file in that set.
Q1: Five 4K bytes equals a sample size of 20,480. Assuming we can have very large files (i.e., the population size?), how confident can I be that all the files within this set, which have the same hash value are identical?
What I have tried:
Using 2 different online Sample Size calculators, I've calculated the following:
For a Confidence Level of 99.5%, Margin of Error of 1.15%, Population Proportion of 50%, and unlimited population size, I need 20,462 samples, which is very close to the 20,480 samples I'm taking.
The other site, CL: 99%, CI: 0.893, Population: 1 Million; states I need: 20,441 samples.
But these sites talk about taking surveys.
Q2: But do these numbers apply in my case?
Q3: If so, how should I read and apply them?
Q4: Or am I doing some wrong and if so what should I be doing instead?