3
$\begingroup$

Our company is developing self-learning software that extracts data from documents and enters it automatically into text-fields in some other software, on behalf of the user. These suggestions aren't always correct though, so the user occasionally has to correct them. We are collecting the results and use them for future calculations and as statistics in general.

I am currently developing a statistics tool so that we can spot how well the software is performing over time. If we make an update to the code and spot that the data suggested is worse off than previously we would like to be able to spot this change, preferably in a diagram.

Example:

The software knows nothing about what the user is looking for, but learns to nail it down every time after 10 documents.

Every point of data is either a hit or miss, there isn't anything in between. However, displaying individual data points of 100 % (hit) or 0 % (miss) isn't very useful in a diagram.

Bad diagram

I will need to group data-points together in order to get a more readable diagram.

Ideal diagram

The question is, how?

Example data: 0 0 0 1 0 0 1 0 1 0 1 0 1 1 0 1 1 1 0 0 1 1 1 1 0 1 1 1 1 1

30 values. 1 = hit, 0 = miss. The trend should show a steady increase in accuracy since it gets more and more of the data right.

Actual question:

How do I group together data-points of percentages in a way so that changes over time is easily distinguishable?

Improve this question
$\endgroup$
6
  • $\begingroup$ Can you edit your question to include a sample of your original data? $\endgroup$
    – Stephan Kolassa
    Commented May 18, 2016 at 8:58
  • $\begingroup$ It's been done :) $\endgroup$
    – HashedPassword
    Commented May 18, 2016 at 10:21
  • $\begingroup$ Why would the underlying percentage start at zero (your second diagram, at time point 1)? Even purely random guessing would be expected to have some non-zero percentage correct, wouldn't it? $\endgroup$
    – Glen_b
    Commented May 18, 2016 at 10:40
  • $\begingroup$ If the software doesn't have any information on how the user fills in the data it won't make a randomized guess. It's more of a hassle for the user to erase the data suggested and then type the right data, than it is to simply fill it in directly. I'm not involved in the development of the software (yet), but that's how it currently works. If we did make a guess it wouldn't be 0 %. $\endgroup$
    – HashedPassword
    Commented May 18, 2016 at 11:18
  • $\begingroup$ I wasn't trying to imply that the program would use random guessing at time 1 -- but it would be weird to do anything that did worse. The point of the comment was only to find this out: is the proportion correct at time 1 certain to be zero or not? $\endgroup$
    – Glen_b
    Commented May 18, 2016 at 11:20

1 Answer 1

Reset to default
4
$\begingroup$

I'd recommend that you calculate and plot a moving average, also known as a running mean. You will need to supply a bandwidth parameter (i.e., how many adjacent data points to average over), which you should pick based on what works best in your data set - for your sample data, it seems like $k=5$ or $k=9$ works well. In R:

library(caTools) # for runmean()
foo <- c(0,0,0,1,0,0,1,0,1,0,1,0,1,1,0,1,1,1,0,0,1,1,1,1,0,1,1,1,1,1)

plot(foo,type="o",pch=19,ylab="")
lines(runmean(foo,k=5),col="red",lwd=2)

moving average

You can easily implement a moving average in MS Excel.

Alternatively, you could use any other smoother. For instance, here is a loess smoother:

plot(foo,type="o",pch=19,ylab="")
lines(lowess(foo),col="blue",lwd=2)

loess

Improve this answer
$\endgroup$
2
  • $\begingroup$ I did try something similar to the moving average, but both of these look great! I will try to implement both and return with the result! :) $\endgroup$
    – HashedPassword
    Commented May 18, 2016 at 12:14
  • 1
    $\begingroup$ Loess wasn't optimal for us, but the moving average is working out really well. We actually have tons of data already. This one has a bandwidh of 50 and around 1000 points of data: i.gyazo.com/b9ab25bd9d7ce55bef76223b4fed294a.png Thanks again! :) $\endgroup$
    – HashedPassword
    Commented May 20, 2016 at 5:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.