Winning the battle against bots

It's six months since we brought anomaly detection out of beta and added IP filtering: the ability to block IPs used by known attackers, data centres, TOR connections, etc, out of the gate. 

Since these features went live, we've been excited – and surprised! – to see how our customers have been using them to win the battle against the bots. 

Findings from the last six months

We knew a high percentage of traffic was bot-generated but, since we've been tracking the stats, we've been surprised at just how high that percentage is.

CrowdHandler is now blocking thousands of suspicious IPs every week for each of those sites with IP filtering turned on.

Most importantly: these six months of observation, across multiple sites, have allowed us to detect some interesting trends and patterns in bot behaviour, which in turn have informed further improvements. This shared intelligence means our security features are now working hand-in-hand to protect sites all over the world.

New: IP reports now contribute to the anomaly score

To give you an example of why this is useful: iPhones allow users to hide their IP address, as if they are hiding behind a proxy network. On its own, CrowdHandler's anomaly detection would probably flag this user as a possible bot – a false positive. But because the IP address it is coming from has a good reputation, our anomaly detection will now take this into account and the user’s anomaly score won't be as low. 

Likewise, if a user comes to the site from an IP range with a bad reputation, it is more likely to get a bad score. The bots are more likely to be turned away. 

When an IP is blocked, the block only lasts for an hour. But CrowdHandler will record the decision and use the information to look for clusters of blocks around certain IP ranges. What's more, this information is shared across all the other sites that CrowdHandler is protecting. So, if the IP is used for suspicious behaviour a second time, it will be blocked again – but this will damage its reputation much further, and it will also damage the reputation of IP addresses in the same range, across every site that uses CrowdHandler. 

New: customers can review IP range recommendations

In an attempt to avoid IP blocks, bots will often enter on a single IP address, then rotate onto new addresses within that range, every few requests. They do this because, whilst one of these addresses may have been flagged on an abuse list, it's possible that not all have. 

This behaviour triggers our anomaly detection algorithms but, by the time the behaviour has been identified and the IP blocked, the bot has already moved onto its next IP. This is why we analyse customer logs to identify the ranges that trigger high anomaly scores, IP blocks and warnings. 

Now, CrowdHandler customers can review the bad ranges we've identified for them, and they can choose to block these ranges permanently.

New: customers can subscribe to the global block list

Bad actors tend to hide behind a relatively small number of bad proxy networks. So, CrowdHandler customers can subscribe to a range of block lists, allowing you to filter out IPs based in data centres, for example, or IPs on known abuse lists. 

Now, the choice of filters to use also includes CrowdHandler's global block list: a constantly updated list of IP ranges we have identified as being flagged across our customer base. It's based on the recommendations we are making on a global level, across all our clients who are using anomaly detection.

And it works. People subscribing to global block lists have found an 80% reduction in the amount of traffic that they need to block, specific to their domain.

Undercover agents: exposed

All bots are, of course, computer programs. Sometimes, this is obvious because they've been badly programmed and report the tools they are using as a user agent, such as Curl or Axios. Usually, they spoof the agent string and report that they are a mainstream browser, such as Chrome on Windows. 

However, even when a bot has declared itself to be something it's not, CrowdHandler can detect this. 

Because we are analysing millions of hits a day, our anomaly detection can spot the tell-tale signs of a spoof, and we can use this to block the imitators – even when they are using thousands of globally distributed IPs. 

Again, information is added to the anomaly score and used across our customer base. (In fact, we have already used this to identify several ticket scalping bot networks.)

Even less reason to be worried about false positives 

We already knew that the percentage of false positives was small, and the last few months have shown us it's even smaller than we thought (less than 0.25%). Sometimes, we even see anomaly detection flag up user journeys that look OK to us… only to find, upon closer investigation, that it was correct and they were indeed bad actors. 

However, we understand that customers can be nervous about blocking anomaly scores above a certain level. That's why, even if a customer decides only to block users with a very high anomaly score (or perhaps even not to block any users at all), CrowdHandler will still offer warnings. And those warnings continue to count towards our overall intelligence and improve security, as CrowdHandler works behind the scenes to log information and build the bigger picture. 

What about the good bots?

You might be wondering – given all the automation, what if we accidentally identify and block a good bot, like Google? 

We know it's not always obvious; Microsoft, Google and Amazon are all good companies, but they also rent servers to anyone, and it's not uncommon for people to base attacks out of Amazon Web Services, Google Cloud or Microsoft Azure hosting. 

The first thing to say is: if you're not sure, don't block. When you look at the recommendations list, you will see that we give more information for each suggestion which should help you make your own decision. 

But, to reassure you further, CrowdHandler also reviews the rules of our global block list regularly, adding global 'bypass' and 'ignore' lists for common 'good' bots, such as monitoring services. Furthermore, your own rules will always override the global block lists - so if you disagree with an entry on the global block list, you can still choose to allow it on your domain. 

Start using the new security tools today

If you’re already using anomaly detection and IP filtering, then you’re all set. All of the new tools are available now to Professional and Enterprise customers at no extra charge.

We can't wait to see how our customers use them in the continued battle against the bots.