How to model the problem of predicting failure in Server Clusters The problem goes as follows -
There is a cluster of Servers. Whenever there is failure/anomaly in any of the server, a report is logged. Some of the features of the log report are

*

*Time of Failure


*Service that led to the failure


*Product for which the server was running


*Cluster Number


*No. of Servers in the cluster
... to name a few
Now the problem is to find patterns from available data as to when such anomalies can occur. Now I tried creating a model by simply using the features to predict the time stamp of failure and obtained a normal prediction score by using a 80-20 Train Test split. However the actual problem is that in real life, since logs are generated only when there is a failure, there is no way to get the various features/Data before hand.
Thus this problem is more about finding patterns. I have been dealing with ML problems quite actively lately that involves existing Train/Test data. However, I have no clue how to go about this one. Any help would be much appreciated.
 A: Rather than focus first on ML tools, I suggest that you re-frame your analysis using the tools and method of Survival Analysis (https://en.wikipedia.org/wiki/Survival_analysis), for which there is a very large literature.  This will help you understand the statistical patterns associated with failures/anomalies and what you can learn from those patterns.
Second, I suggest you learn about root cause analysis.  This probably means that you will need data beyond error logs for servers. It may involve data associated with server installation, server configuration, updates/upgrades, and facility information such as power and temperature fluctuations.  It may also involve  data on the personnel who maintain the servers, such as how long they have been working, how much training they have, and which servers they are responsible for.
After you do both of these steps, you can then think again about what ML tools might be most effective and appropriate.
A: Just a thought: what if you converted the problem of "anomaly detection" to the problem of "normality detection"? 
That is, you train your model on anomalies and then try to apply it to all the data hoping that it'll be able to tell anomalies apart from normal data points. 
Not sure though if the features you have are descriptive enough for this ("time of failure" certainly doesn't seem good), but this is definitely something I'd try doing. You can start with One-Class SVM (available in sklearn).
A: i am not sure if this is relevant anymore but i will still have a go.. 
This is typical problem which most monitoring tool try to address. All monitoring tools have been programmed to highlight when there is a event making it difficult for anyone to predict. At most this allows the resolution teams with root cause analysis and resolution. 
There are few things you will need to
1) Continues data feed from sensors which provide you the data. 
2) A root cause analysis of previous failures, providing you with all the contributors. ( e.g. a hardware failure may occur because of over heating. which is actually caused by cooling failure because of power outage ). 
As you piece the old failure data together, the continues sensor data will help you identify failures before or as they occur. 
