2
$\begingroup$

How can I modeling a distribution for these kind of data?

  • Frequency of clock of a CPU [I have 100 observations between 100 MHz and 4 GHz]
  • RAM of a Computer [I have 150 observations between 128 MB and 32 GB]
  • Weight of a phone [I have 50 observations between 30 g and 300 g]
  • Internal Memory of a CPU [I have 100 observations between 100 MB and 5 TB]

For everyone of them I have some boundary and I can calculate mean and standard deviation.

The second question is: If I receive a new value for the attribute RAM ... for example 64 GB that is out of bound... there is a way to get a probability that 64 GB could be a new value???

For example, 64 GB could be a new value for the RAM attribute of a pc (in my distro it is out of bound). If I get 512 GB as a new value I know that it can be referred to the RAM attribute of a pc.

So what I need is a way to estimate that the 64 GB could be a new value of RAM because it is very close to my distribution... but 512 GB can't be in any way a new RAM value.

So I need a function that return me a number close to 1 for the 64 GB and close to 0 for the 512 GB value... How can I do that? Gaussian kernel density?

NOTE 1 I have a catalogue and I can extract all possible value for a specific attribute and I need to model their distribution. I don't need the negative part... as all the attributes that I'm considering haven't negative values

NOTE 2 It's clear that a normal distro can't fit well of the values of this kind of attributes... Yes RAM, memory are discrete and not unimodal... also the weight of phones is not unimodal... so my question is which is the distribution that can fit the values of this kind of values?

NOTE 3

These are a portion of my data. Between parentheses there is the frequency of the value. I want to do it because if a receive a new value I want to know if it a fair value.

I think that for the RAM attribute all values must be converted to KB.

Example of weight (frequency is the value in parentheses)
77.9 (3)
78 (114)
78.6 (3)
79 (114)
79.3 (3)
80.4 (3)
80 (297)
80.5 (3)
81 (75)
81.4 (3)
82 (150)
82.5 (3)
83 (159)
83.6 (3)
84 (111)
84.5 (6)
85 (330)
85.5 (3)
86 (129)
86.3 (3)
87 (66)
87.5 (3)
87.8 (3)
88 (147)
89 (123)
89.3 (3)

Example of RAM 
1 GB (918)
1 MB (3)
128 MB (78)
16 MB (27)
192 MB (12)
2 GB (357)
24 MB (3)
256 MB (141)
288 MB (18)
3 GB (84)
3 MB (3)
32 MB (30)
384 MB (15)
4 GB (3)
4 MB (12)
448 MB (3)
45 MB (3)
512 MB (570)
576 MB (9)
64 MB (141)
768 MB (39)
8 MB (15)
96 MB (3)

Example of Frequency
1.2 GHz (528)
1.3 GHz (417)
1.4 GHz (102)
1.5 GHz (174)
1.6 GHz (30)
1.7 GHz (114)
1.8 GHz (9)
1.9  GHz (15)
104 MHz (3)
184 MHz (3)
2.1 GHz (12)
2.2 GHz (36)
2.26  GHz (21)
$\endgroup$
  • 4
    $\begingroup$ Normally a value doesn't have a distribution at all--it's just a number. Could you please edit this question, then, to explain the sense in which your values could be varying or uncertain? $\endgroup$ – whuber Jun 26 '15 at 17:35
  • $\begingroup$ "so what you suggest to model that distribution of values" - is it possible to show the distribution you want to model? There is no way that someone who has not seen your catalogue, can know what the distribution of RAM in your catalogue is, for instance. $\endgroup$ – Silverfish Jun 27 '15 at 17:13
  • $\begingroup$ Thank you for the additional information. However, it is not enough to answer your question: as @Silverfish requested, you would need to provide details of the data if you want objective opinions about suitable distributions to fit to them. $\endgroup$ – whuber Jul 1 '15 at 13:30
  • $\begingroup$ Great--I voted to reopen your question. You might consider explaining why you want to model these distributions, because that can help people identify the better solutions that are available. $\endgroup$ – whuber Jul 1 '15 at 15:16
  • $\begingroup$ Agree with whuber that you should explain why you want to model these numbers. Also, where are they coming from? The distribution of amount of RAM in computers drawn at random from "our office" vs from "our customers" will differ a great deal. A concrete example of such a difference mattering would be phone weight--if they're all the same phone model, then a normal distribution would probably work pretty well, if they're different models, the distribution is probably multimodal. $\endgroup$ – Stumpy Joe Pete Jul 6 '15 at 7:01
3
+50
$\begingroup$

A technique I have had some success with, is to use a kernel density estimate (KDE) to model arbitrary and non-analytical continuous distributions. This would work well for your distribution of phone weights, for example. You can then use the KDE to estimate the likelihood of an additional observation. This tells you the probability of observing the new data point, under the p.d.f. given by your model (the KDE, in this case).

Some care is required with correcting the amplitude to get a valid p.d.f., or to convert it to a reasonable likelihood function.

vfWeightSamples = [... your samples of measured weight ...]; % Your data
vfWeightLikelihoodRange = linspace(min(vfWeightSamples)-10, max(vfWeightSamples)+10, 200);  % For plotting the estimated function
vfWeightLikelihood = ksdensity(vfWeightSamples, vfWeightLikelihoodRange, 'function', 'pdf', 'bandwidth', 0.5);
figure, plot(vfWeightLikelihoodRange, vfWeightLikelihood);

Kernel density estimate of weight density

You can play with the bandwidth parameter until you are happy that the estimated density captures the structure of your data. You then need to ensure the function has an appropriate amplitude: for example, the probability of obtaining a weight of "89" should be something like 126/1860 for your dataset. You can incorporate this by computing a weighting factor for the density:

fDensityFactor = ksdensity(vfWeightSamples, 89, 'bandwidth', 0.5);
fDensityFactor = (126/1860) / fDensityFactor;

% - To compute a new likelihood
fLikelihood = fDensityFactor * ksdensity(vfWeightSamples, vfNewSamples, 'bandwidth', 0.5);
$\endgroup$
  • $\begingroup$ I think that it is what I looking for. But I need an example... than I'll choose your answer as best. For example could you show me how to apply it to the RAM attribute in python? I also think that all value must be converted in KB or MB... $\endgroup$ – Usi Usi Jul 6 '15 at 16:14
  • $\begingroup$ This will only work properly for a continuous variable. I'll add some example Matlab code. $\endgroup$ – Dylan Richard Muir Jul 7 '15 at 7:04
  • $\begingroup$ Kernel density estimation is among various good methods of checking how far a distribution is normal (or indeed any other specified distribution), although with a specific hypothesis in mind I'd prefer a quantile-quantile plot. But it is not especially useful for fitting a distribution: if anyone was determined to fit a normal distribution (a bad idea in this thread), then the mean and SD inferred from a kernel density estimate should not differ much from mean and SD calculated directly from the data. Note that interest in a Gaussian doesn't imply a Gaussian kernel especially. $\endgroup$ – Nick Cox Jul 7 '15 at 9:32
  • $\begingroup$ I agree. My example was definitely not designed to fit a normal distribution, but to estimate an arbitrary distribution informed by the data. $\endgroup$ – Dylan Richard Muir Jul 7 '15 at 10:03
  • 1
    $\begingroup$ @MattKrause: Yes; as I wrote above, this is only really suitable for continuous variables such as weight. It would be a bit of a misuse to apply a kernel density estimate to a distribution known to be discrete... But as you said, it depends on how strict one needs to be. $\endgroup$ – Dylan Richard Muir Jul 8 '15 at 6:45
5
$\begingroup$

Individual values themselves don't have distributions--the number 3 is just the number 3, so if you have a specific computer in mind, then it doesn't make a whole lot of sense to talk about the "distribution" of RAM in this specific computer.

However, you could define a random variable that describes a property of a set of computers. For example, "Suppose you grabbed a computer at random from {this office, that warehouse, one of our users}. Let's call that computer's RAM capacity $R$?" $R$ is a random variable and it does make sense to talk about the distribution of $R$.

It's unlikely that $R$ or any of other properties are exactly normally distributed. The normal distribution is unimodal (it's got one peak) and continuous (it assigns a probability to any value). In practice, CPU speed, cache size, and RAM amount are pretty discrete. You can't buy a chip with 3,534,234 bytes of RAM on it--your choices are typically 1, 2, 4,...32 GB. Likewise, while CPU speeds can vary a little bit due to over/unclocking (etc), they're going to be clustered around 2.2, 2.4, 2.7 GHz. The weight of a phone is likely to be more continuous, but even so...it's unlikely to be unimodal: there's a market for enormous phones with high-quality screens, as well as a market for small, cheap phones, but there might not be much in the middle.

Finally, there's no way for values to be "outside" a normal distribution. Its two tails extend to $\pm \infty$, although the probability of seeing something very far from its mean gets incredibly small. This actually points out another problem with your proposed approach: all the quantities you've described are strictly positive (you can't have a phone that is 0 grams or a computer with -4 Gb of RAM), while the normal distribution assigns probability to all positive AND negative numbers.


Update: As people have noted in the comments, the "best" model depends on your goal as well as the data. If you're attempting to model (e.g.) price as a function of the phone's specs, you might be able to leave the data as is--the distribution of residuals matters more than the distribution of the data per se.

Alternatively, you could use your catalogue as a a discrete distribution. This might be a good choice if you're modelling a relatively "closed" population (e.g., Users prefer machines that run at 2 GHz; how often are "acceptable" nodes available in our lab?) There are 1464 examples (in the CPU Frequency part), and 528 of them have a speed of 1.2 GHz, so P(speed=1.2 GHz) = 528/1464, or about 0.36. There are a few options for handling "new" values that don't show up in the original data set. You could add an "unknown" bin, or two bins (smaller than the min, bigger than the max). One common trick is to treat "rare" data as unknown, so you might initially use the 21 samples with a 2.26 GHz CPU to seed the P(>2.2 GHz) bin. As time passes, you may end up with more and more data in that bin, in which case you may need to revisit the model.

$\endgroup$
  • $\begingroup$ Thanks for your great answer, so what you suggest to model that distribution of values? I have a catalogue and I can extract all possible value for a specific attribute and I need to model their distribution. $\endgroup$ – Usi Usi Jun 27 '15 at 14:52
  • $\begingroup$ I read well your answer and I have understood what you says... Yes ram, memory are discrete and not unimodal... also the weight of a phone is not unimodal... so my question is which is the distribution that can fit the values of this kind of values? $\endgroup$ – Usi Usi Jul 1 '15 at 11:56
0
$\begingroup$

As I realize from your question, you can run a test to see if data follow a normal distribution or not. This is a definition from wikipedia :

In statistics, normality tests are used to determine if a data set is well-modeled by a normal distribution and to compute how likely it is for a random variable underlying the data set to be normally distributed.

see this page: https://en.wikipedia.org/wiki/Normality_test

see this page for simple R commands: http://www.dummies.com/how-to/content/how-to-test-data-normality-in-a-formal-way-in-r.html

for QQ-plot you can use qqnorm() in R

Your second question is not clear because normal distribution spans all over $(-\infty,\infty)$ and it is not bounded. Then you can have some values but the probability of occurrence is too low.

$\endgroup$
  • $\begingroup$ How can I model this kind of values?? I don't need the negative part $\endgroup$ – Usi Usi Jun 27 '15 at 14:53
  • $\begingroup$ Then for example you can assume $X^2$ that follows non-canonical (if mean of the data is zero) chi-squared distribution. $\endgroup$ – TPArrow Jun 27 '15 at 21:50
  • $\begingroup$ This answer looks less relevant largely because the OP changed the question drastically after 6 of 7 days in the bounty period. To that extent the single downvote is unfair and I reversed it. $\endgroup$ – Nick Cox Jul 7 '15 at 16:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.