19
$\begingroup$

What are main differences between sparse data and missing data? And how does it influences machine learning? More specifically, what effect sparse data and missing data have on classification algorithms and regression (predicting numbers) type of algorithms. I'm talking about a situation, where percentage of missing data is significant and we can't drop the rows containing missing data.

$\endgroup$
  • 4
    $\begingroup$ Sparse data means that many of the values are zero, but you know that they are zero. Missing data means that you don't know what some or many of the values are. $\endgroup$ – Anna SdTC Mar 14 '17 at 6:54
  • $\begingroup$ Thanks. That's what I also thought, but wanted to confirm. Also, as mentioned in question, would like to know how, in general, these types datasets are handled in machine learning problems.. $\endgroup$ – tired and bored dev Mar 14 '17 at 7:40
  • 1
    $\begingroup$ I think that your question is a little vague. "Machine learning" includes a wide range of methods and tools, so the answer depends on what you have or what you want to do. Here they discuss some methods for handling missing data: stats.stackexchange.com/questions/103500/… $\endgroup$ – Anna SdTC Mar 14 '17 at 7:50
  • $\begingroup$ Thanks. I'm aware of broad range of tools and types of ml algorithms. But wanted to know if there are any general approaches. $\endgroup$ – tired and bored dev Mar 14 '17 at 9:23
15
$\begingroup$

For the ease of understanding, I'll describe this using an example. Let's say that you are collecting data from a device which has 12 sensors. And you have collected data for 10 days.

The data you have collected is as follows: enter image description here

This is called sparse data because most of the sensor outputs are zero. Which means those sensors are functioning properly but the actual reading is zero. Although this matrix has high dimensional data (12 axises) it can be said that it contains less information.

Let's say 2 sensors of your device is malfunctioning.
Then your data will be like: enter image description here

In this case, you can see that you cannot use data from Sensor1 and Sensor6. Either you have to fill data manually without affecting the results or you have to redo the experiment.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.