What do big data and high dimensional data mean? Is high dimensional data a special case of big data? What are the complications that arise in the analysis of high dimensional and big data each?
$\begingroup$
$\endgroup$
3
-
8$\begingroup$ To a large degree, "big data" is a term for marketing hype. The term has a more legitimate meaning when you get to the point where the size forces you to use different methods, but it's mostly used by people using standard methods who just was to look more impressive. $\endgroup$– gung - Reinstate MonicaCommented Aug 28, 2019 at 18:57
-
$\begingroup$ Big data could be fat, tall or both. Fat a.k.a. high-dimensional ~ many variables, tall ~ many observations. $\endgroup$– Richard HardyCommented Aug 28, 2019 at 19:45
-
$\begingroup$ i think big data would also include data coming in really fast stream $\endgroup$– DreadfulWeatherCommented Aug 29, 2019 at 3:05
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
Big data implies large numbers of data points, while high-dimensional data implies many dimensions/variables/features/columns.
It's possible to have a dataset with many dimensions and few points, or many points with few dimensions. But if you have high-dimensional datasets with few data points, you're unlikely to be able to learn much from it. So high-dimensional data is generally going to be big data as well.
The converse is not true - big data does not need many dimensions for you to learn from it. But if you are only working with a few dimensions, it's probably not as necessary to collect a large number of data points to do your analysis. Note that there are important exceptions to this - noisy measurements, high frequency spatial or temporal data, and so on.
So it's probably generally true that giant datasets with many points also happen to have many variables/dimensions as well. In other words, the terms mean different things, but big data is usually high-dimensional data and vice versa.
Regarding the complications associated with each, here is a very incomplete answer: big data poses computational challenges (loading data in memory, for example), while analysis of high-dimensional data falls prey to the curse of dimensionality.
-
2$\begingroup$ I disagree slightly with the statement "if you have high-dimensional datasets with few data points, you're unlikely to be able to learn much from it." Time-series data is usually high-dimensional, but you can often get good insights from relatively few (dozens, hundreds) of data examples relative to the number of dimensions in each example of data (hundreds, thousands, more?). Usually these insights come from the fact that you can represent the original high-dimensional data well using a lower-dimension representation, like the first few principal components. $\endgroup$ Commented Aug 29, 2019 at 2:55
-
$\begingroup$ @JohnDavis Fair enough, but I stand by what I wrote. There are no particularly strong claims in the answer because there's obviously a lot of variation between domains. $\endgroup$– mktCommented Aug 29, 2019 at 20:51