One of the major additions to version 3.3 of Prelert Anomaly Detective® for Splunk was a feature called StatsReduce. This feature enables Anomaly Detective to take advantage of Splunk’s distributed processing to analyse immense volumes of data quickly enough to deliver real-time insights.
Due to the way Anomaly Detective is designed, adding this feature wasn’t particularly difficult. Tom’s last post explained how our anomaly detection algorithms are designed to work with aggregated data to solve the problem of data inertia. One of the dimensions we aggregate over is time: we divide time into periods called buckets, calculate summary statistics of the data for each time bucket and do our analysis based on these statistics.