Machine Data is Different

A couple of years ago one of Prelert's taglines was “Machine Learning for Machine Data”.  Whilst the marketing drive has moved on, this phrase is still very relevant to what Prelert's products do.  But why is machine data different?

To answer this question, let’s start by considering different perspectives on what constitutes unstructured data.

At the end of February Rich wrote a blog about anomaly detection in unstructured dataPrelert Anomaly Detective® for Splunk has the ability to categorise log messages and then detect anomalies in the rates of the different message categories.

Read More

Ready to Talk Anomaly Detection & Advanced Math at Splunk's User Conference

The Prelert team, along with partners and customers, will share insights on using machine-based anomaly detection to find value in Big Data in front of over 4,000 IT and business professionals at Splunk’s fifth annual Worldwide Users’ Conference, .conf2014. The event will take place from October 6-9 at the MGM Grand in Las Vegas, Nevada.

The event will kick off with a Partner Soiree from 5-7 p.m. on Oct. 6th. We don’t have a booth number, but have no fear - exhibiting as a Level 1 sponsor means Prelert will have prime real-estate right near all the action. There will be food and drinks in the booth partner pavilion, so stop by and say hello!

Read More

Anomaly Detection to Reduce the Noise

If you have followed some of my other recent blogs, you’ll have noticed that automated anomaly detection is a great technique to find anomalous behaviors in data by effectively contrasting the difference between “normal” and “abnormal." Most people equate this with contrasting between “good” and “bad,” but that isn’t always necessarily true. What if the data set you’re looking at are “all bad things,” such as Intrusion Detection (IDS) alerts? If that’s the case, you need to change your frame of reference slightly.


In the case of IDS Alerts, or any other alerts, there (sadly) could always be a “background noise” of alerts going on. Your task now is to find the signal in that noise. For example, suppose a set of systems all should, in general, generate a similar number of IDS alerts per unit time  (i.e. ServerA: 10/hr, Server B: 12/hr, Server C: 15/hr….Server X: 500/hr). In this case, Server X is interesting to know about because it is disproportionately getting targeted more aggressively than other systems.

Read More

Will You be Replaced by Machine Intelligence?

While humans are definitely needed for the expertise-dependent and creative functions, many aspects of IT operations and performance management could be done more effectively by machine intelligence. Here are just a few examples.

Read More

How to Detect (and Resolve) IT Ops/APM Issues Before Your Users Do

As originally published by APMdigest.

Among the most embarrassing situations for application support teams is first hearing about a critical performance issue from their users. With technology getting increasingly complex and IT environments changing almost overnight, the reality is that even the most experienced support teams are bound to miss a major problem with a critical application or service. One of the contributing factors is their continued reliance on traditional monitoring approaches.

Whoops.

Traditional tools limit us to monitoring for a combination of key performance indicator thresholds and failure modes that have already been experienced. So when it comes to finding new problems, the best case is alerts that describe the symptom (slow response time, transaction fails, etc.). A very experienced IT professional will have seen many behaviors, and consequently can employ monitoring based on best practices and past experiences. But even the most experienced IT professional will have a hard time designing rules and thresholds that can monitor for new, unknown problems without generating a number of noisy false alerts. Anomaly detection goes beyond the limits of traditional approaches because it sees and learns everything in the data provided, whether it has happened before or not.

Read More

Automated Anomaly Detection: A Connector for Amazon CloudWatch

Amazon CloudWatch is a monitoring service for AWS cloud resources and the applications you run on AWS. You can use Amazon CloudWatch to collect and track metrics, collect and monitor log files, and set alarms.

At the time of writing, CloudWatch is currently available to all AWS users, with the free version giving basic monitoring metrics (at 5 minute frequency) and generous usage limits. You can also add up to 10 custom metrics and 10 alarms.

In this blog I shall briefly explain why it is important to use unsupervised machine learning to effectively manage your AWS environments. I shall then point you at the developer resources we have made available that will allow you to configure this for Amazon Elastic Compute Cloud (EC2) which includes a free developer license for the Anomaly Detective® Engine & API.

The case for unsupervised machine learning in AWS

Read More

Rogue User Detection via Behavioral Analysis

Similar in concept to my last blog on data exfiltration, finding “rogue users” or “rogue systems” using behavioral analysis and automated anomaly detection takes a different approach than the traditional methods of manual data inspection, or the application of rules or signatures to identify specific behavioral violations.


Read More

Prelert Takes Home a Silver Stevie Award

Last Friday marked the twelfth annual American Business Awards and Prelert was honored with a Silver Stevie Award in the New Product or Service of the Year - Software - Big Data Solution category. The announcement was made at the organization’s first ever New Product & Tech Awards banquet at the Palace Hotel in (where else but the tech mecca) San Francisco.

The Stevies are the nation’s premier business awards program and any organization operating in the U.S. is eligible – big, small, for profit, non-profit, you name it. This year, more than 3,300 nominations were submitted, representing organizations of all sizes and in virtually every industry to be considered in a wide range of categories. Winners were selected by more than 240 executives nationwide who participated in the judging process.

Read More

It's Time to Democratize Data Science!

The biggest trend we’ve seen in the analytics industry is both the increase in understanding of data’s value and the desire of executives – who aren’t data scientists – to gain insights from it. This leads us to believe that it’s truly time to democratize data science.

In the early 1990s, the HTML protocol was invented, the government opened the internet to private enterprise and the first internet mail and shopping experiences came online. In that transition from government funded research network to ‘open to the public’ the internet was democratized.

It took the intentional development of tools like Netscape’s Mosaic web browser, Intel’s Pentium processor and Sun Microsystems’ Java to package that technology for widespread consumption. But that started the ball rolling and today we couldn’t imagine a world that did not have the internet.

Read More

Why What You Don't Know May Hurt You, & How Security Analytics Can Help

As originally published by Infosec Island.

Managing security in today’s enterprise is far different than it was ten to fifteen years ago. In the past, companies were able to set up proxy agents, firewalls and strong virus protection software and feel pretty secure that their company’s information was safe.

However, in today’s world, things have changed. We are no longer dealing with teenage hackers or disgruntled young adults with a political or social ax to grind. The real threat to your security comes from advanced cybercriminal organizations. They are well versed in your typical defenses and spend all their time figuring out ways to bypass them. These are professionals with the skills, knowledge, talent, creativity and motivation to succeed.

If you consider your organization to be a likely target, then it’s a safe bet that your defenses have already been infiltrated – and that it’s only a matter of time until the real theft begins. This means your organization needs to immediately focus on detecting nefarious activities inside of your perimeter.

Read More

Subscribe to updates