We have data from the border firewalls pertaining to all ports used on our networks. For example, we have counts of all connection attempts to all ports each day on our networks.
date, port, count 8/17/2016,23,123444523 8/17/2016,80,45653345 ... 8/18/2016,23,244533556 ...
We are trying to create normal patterns around these counts. For example, in the last 2 weeks, port 23 observed these sets of counts pertaining to connection attempts each day:
122334234,12445523,123454332,.... If somewhere around those counts are normal then
555335534343 will be detected as an anomaly on a particular day and send an alert.
Also, create a list of top 50 ports each day (by connection attempt counts). If a port has jumped more than, let's say, 20 places in this list then there's something fishy probably and we can generate an alert.
What other patterns could be formed in the data to detect out-of-the-ordinary observations and, more importantly, what machine learning or pattern recognition algorithms can we use for these purposes?