Implementing StatsReduce in Anomaly Detective

One of the major additions to version 3.3 of Prelert Anomaly Detective® for Splunk was a feature called StatsReduce.  This feature enables Anomaly Detective to take advantage of Splunk’s distributed processing to analyse immense volumes of data quickly enough to deliver real-time insights.

Due to the way Anomaly Detective is designed, adding this feature wasn’t particularly difficult.  Tom’s last post explained how our anomaly detection algorithms are designed to work with aggregated data to solve the problem of data inertia.  One of the dimensions we aggregate over is time: we divide time into periods called buckets, calculate summary statistics of the data for each time bucket and do our analysis based on these statistics.

Read More

Anomaly Detection on Large Data Sets via Aggregation

In our last release we announced the “Stats Reduce” feature. The functionality was a key ingredient of the original conception for how we'd get our anomaly detection algorithms to work on very large data sets. This post is going to discuss the rationale behind our use of aggregation, which underpins this feature.

Read More

IoT Won’t Work Without Artificial Intelligence

As originally published by Wired.

As the Internet of Things (IoT) continues its run as one of the most popular technology buzzwords of the year, the discussion has turned from what it is, to how to drive value from it, to the tactical: how to make it work.

IoT will produce a treasure trove of big data – data that can help cities predict accidents and crimes, give doctors real-time insight into information from pacemakers or biochips, enable optimized productivity across industries through predictive maintenance on equipment and machinery, create truly smart homes with connected appliances and provide critical communication between self-driving cars. The possibilities that IoT brings to the table are endless.

Read More

How to Find Anomalies in Splunk's Internal Performance

Clearly we have had many discussions on this blog about different use cases of the Prelert Anomaly Detective App for Splunk - those in IT Operations, Performance Management, and IT Security. But one area of applicability that shouldn’t be overlooked is using Anomaly Detective to find performance or operability problems in Splunk itself.

Read More

C++11 mutex implementations

C++11 brought concurrency to standard C++ for the first time.  Prior to this the only choice for writing multi-threaded C++ programs was to use a separate C++ library, such as Boost Thread or Intel Thread Building Blocks, or roll your own wrappers around the low-level operating system facilities, such as POSIX threads or Windows threads.

Whilst it’s great to have standard headers like <thread>, <mutex> and <atomic> available, the brutal truth is that under the covers all the implementations of the standard C++ classes use those same low-level operating system facilities.

Last year I looked into the performance of different types of locks on different platforms.  The variation in performance is surprisingly wide.  Prelert’s codebase pre-dates C++11, so we have our own wrappers around the low-level operating system facilities.

Read More

Machine Data is Different

A couple of years ago one of Prelert's taglines was “Machine Learning for Machine Data”.  Whilst the marketing drive has moved on, this phrase is still very relevant to what Prelert's products do.  But why is machine data different?

To answer this question, let’s start by considering different perspectives on what constitutes unstructured data.

At the end of February Rich wrote a blog about anomaly detection in unstructured dataPrelert Anomaly Detective® for Splunk has the ability to categorise log messages and then detect anomalies in the rates of the different message categories.

Read More

Ready to Talk Anomaly Detection & Advanced Math at Splunk's User Conference

The Prelert team, along with partners and customers, will share insights on using machine-based anomaly detection to find value in Big Data in front of over 4,000 IT and business professionals at Splunk’s fifth annual Worldwide Users’ Conference, .conf2014. The event will take place from October 6-9 at the MGM Grand in Las Vegas, Nevada.

The event will kick off with a Partner Soiree from 5-7 p.m. on Oct. 6th. We don’t have a booth number, but have no fear - exhibiting as a Level 1 sponsor means Prelert will have prime real-estate right near all the action. There will be food and drinks in the booth partner pavilion, so stop by and say hello!

Read More

Anomaly Detection to Reduce the Noise

If you have followed some of my other recent blogs, you’ll have noticed that automated anomaly detection is a great technique to find anomalous behaviors in data by effectively contrasting the difference between “normal” and “abnormal." Most people equate this with contrasting between “good” and “bad,” but that isn’t always necessarily true. What if the data set you’re looking at are “all bad things,” such as Intrusion Detection (IDS) alerts? If that’s the case, you need to change your frame of reference slightly.


In the case of IDS Alerts, or any other alerts, there (sadly) could always be a “background noise” of alerts going on. Your task now is to find the signal in that noise. For example, suppose a set of systems all should, in general, generate a similar number of IDS alerts per unit time  (i.e. ServerA: 10/hr, Server B: 12/hr, Server C: 15/hr….Server X: 500/hr). In this case, Server X is interesting to know about because it is disproportionately getting targeted more aggressively than other systems.

Read More

Will You be Replaced by Machine Intelligence?

While humans are definitely needed for the expertise-dependent and creative functions, many aspects of IT operations and performance management could be done more effectively by machine intelligence. Here are just a few examples.

Read More

How to Detect (and Resolve) IT Ops/APM Issues Before Your Users Do

As originally published by APMdigest.

Among the most embarrassing situations for application support teams is first hearing about a critical performance issue from their users. With technology getting increasingly complex and IT environments changing almost overnight, the reality is that even the most experienced support teams are bound to miss a major problem with a critical application or service. One of the contributing factors is their continued reliance on traditional monitoring approaches.

Whoops.

Traditional tools limit us to monitoring for a combination of key performance indicator thresholds and failure modes that have already been experienced. So when it comes to finding new problems, the best case is alerts that describe the symptom (slow response time, transaction fails, etc.). A very experienced IT professional will have seen many behaviors, and consequently can employ monitoring based on best practices and past experiences. But even the most experienced IT professional will have a hard time designing rules and thresholds that can monitor for new, unknown problems without generating a number of noisy false alerts. Anomaly detection goes beyond the limits of traditional approaches because it sees and learns everything in the data provided, whether it has happened before or not.

Read More

Subscribe to updates