2
$\begingroup$

I have a large data set of parcels (>100k) with the variables length, width, height and weight. For simulation purposes I'm only able to use a sample of roughly 10 parcels. To make it as practical as possible I'd like to build a sample that is representative of the whole data set. What is the best way to do this?

I tried to pick random items from the data set but there always seem to be outliers and just shuffling through randomly picked samples until I'm content with the outcome doesn't seem practical. The two-sample KS test however, does give me p-values of >0.2 for each variable.

I thought about building it manually by creating histograms for each variable and then scaling them down to 10 elements but then there is a problem that some of the variables are highly correlated with each other and others not so much. (Kendall's tau gives me values ranging from 0.2 to 0.7)

Edit: Some information about the simulation: In the course of my bachelor's thesis I have to run a DEM simulation of a separation mechanism of parcels on a conveyor belt. My job is to determine the optimal combination of parameters (with DoE) for a high degree of separation and speed. The computation is very slow, so I can only work with a very limited sample size.

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ Have you considered clustering? $\endgroup$
    – John Madden
    Commented Aug 7, 2015 at 13:51
  • 2
    $\begingroup$ The practical meaning of "representative" in this instance will have to depend on what you are simulating and the purpose of that simulation. Could you tell us a little about those? $\endgroup$
    – whuber ♦
    Commented Aug 7, 2015 at 13:52
  • 1
    $\begingroup$ If you are randomly sampling, the data should be representative (in the ordinary sense of the word), nor could there be any true outliers. $\endgroup$
    – gung - Reinstate Monica
    Commented Aug 7, 2015 at 14:00
  • $\begingroup$ A friend also just suggested clustering to me. I'm not familiar with cluster analysis so I guess I'll have some reading up to do. I edited my question to give some information about the simulation. I hope this helps. $\endgroup$
    – Shamp0o
    Commented Aug 7, 2015 at 14:47
  • 1
    $\begingroup$ Thank you for the edit (+1). This adds considerable information to the question, because it indicates you are in an experimental design situation and that you are seeking to estimate a response surface in order to optimize a response. That suggests you might not want a statistically representative sample--instead, you need one that will give you the best possible estimate of the optimum. If you know nothing about the response, you might want your sample to exhibit a variety of extreme characteristics, whereas if you know about where the optimum is, you want the sample to be near it. $\endgroup$
    – whuber ♦
    Commented Aug 7, 2015 at 15:37

1 Answer 1

Reset to default
0
$\begingroup$

As far as I can see, you are allowed to pick a limited number, N, of points in the 4-dimensional space, and you need them to cover your region of interest "as evenly as possible". In that case, low discrepancy numbers can help, e.g. try the Hammersley set described here.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.