Skip to main content
Cross Validated

Return to Question

information about the simulation
Source Link
Shamp0o
Shamp0o
  • 21
  • 2

I have a large data set of parcels (>100k) with the variables length, width, height and weight. For simulation purposes I'm only able to use a sample of roughly 10 parcels. To make it as practical as possible I'd like to build a sample that is representative of the whole data set. What is the best way to do this?

I tried to pick random items from the data set but there always seem to be outliers and just shuffling through randomly picked samples until I'm content with the outcome doesn't seem practical. The two-sample KS test however, does give me p-values of >0.2 for each variable.

I thought about building it manually by creating histograms for each variable and then scaling them down to 10 elements but then there is a problem that some of the variables are highly correlated with each other and others not so much. (Kendall's tau gives me values ranging from 0.2 to 0.7)

Edit: Some information about the simulation: In the course of my bachelor's thesis I have to run a DEM simulation of a separation mechanism of parcels on a conveyor belt. My job is to determine the optimal combination of parameters (with DoE) for a high degree of separation and speed. The computation is very slow, so I can only work with a very limited sample size.

I have a large data set of parcels (>100k) with the variables length, width, height and weight. For simulation purposes I'm only able to use a sample of roughly 10 parcels. To make it as practical as possible I'd like to build a sample that is representative of the whole data set. What is the best way to do this?

I tried to pick random items from the data set but there always seem to be outliers and just shuffling through randomly picked samples until I'm content with the outcome doesn't seem practical. The two-sample KS test however, does give me p-values of >0.2 for each variable.

I thought about building it manually by creating histograms for each variable and then scaling them down to 10 elements but then there is a problem that some of the variables are highly correlated with each other and others not so much. (Kendall's tau gives me values ranging from 0.2 to 0.7)

I have a large data set of parcels (>100k) with the variables length, width, height and weight. For simulation purposes I'm only able to use a sample of roughly 10 parcels. To make it as practical as possible I'd like to build a sample that is representative of the whole data set. What is the best way to do this?

I tried to pick random items from the data set but there always seem to be outliers and just shuffling through randomly picked samples until I'm content with the outcome doesn't seem practical. The two-sample KS test however, does give me p-values of >0.2 for each variable.

I thought about building it manually by creating histograms for each variable and then scaling them down to 10 elements but then there is a problem that some of the variables are highly correlated with each other and others not so much. (Kendall's tau gives me values ranging from 0.2 to 0.7)

Edit: Some information about the simulation: In the course of my bachelor's thesis I have to run a DEM simulation of a separation mechanism of parcels on a conveyor belt. My job is to determine the optimal combination of parameters (with DoE) for a high degree of separation and speed. The computation is very slow, so I can only work with a very limited sample size.

Source Link
Shamp0o
Shamp0o
  • 21
  • 2

Building a representive sample from a large data set

I have a large data set of parcels (>100k) with the variables length, width, height and weight. For simulation purposes I'm only able to use a sample of roughly 10 parcels. To make it as practical as possible I'd like to build a sample that is representative of the whole data set. What is the best way to do this?

I tried to pick random items from the data set but there always seem to be outliers and just shuffling through randomly picked samples until I'm content with the outcome doesn't seem practical. The two-sample KS test however, does give me p-values of >0.2 for each variable.

I thought about building it manually by creating histograms for each variable and then scaling them down to 10 elements but then there is a problem that some of the variables are highly correlated with each other and others not so much. (Kendall's tau gives me values ranging from 0.2 to 0.7)