A user agent is a device (and more specifically, the software) that accesses a website. Web browsers like Chrome, Edge, and Safari are all user agents, and so is Google when it scans and indexes websites (i.e. “Googlebot”).
The user agent identifies itself in every HTTP request using a written description in the “User-Agent” header. This HTTP header is primarily what we will be discussing today.
User agents are awesome. They give you simple and useful information about the specific browsers, bots, and other services accessing your website. Web developers can use this information to optimize their content for the technology they were requested by. And, of course, they can be used to filter unwanted traffic from your servers.
But user agents also have some major drawbacks.
For starters, user agents are completely non-standardized. Some include version numbers, special characters, or domain names while others are simply brand names or even acronyms.
Additionally, they are unbelievably easy to spoof. I know at least 3 easy ways for you to load this blog post with whatever user agent you want. In fact, you could even do it right there in your browser in about 30 seconds!
Finally, user agents are not even required. It is perfectly valid to request a web page without any user agent at all. At least we can be glad that you can’t use multiple user agents for a single request.
All of this adds up to two inescapable facts that do not appear compatible:
Thankfully, most software - including all major browsers - reliably and accurately include a consistent user agent in every request. Even the majority of bots report some sort of relevant user agent. And if someone requests a page with a spoofed user agent, it is relatively safe to assume that they want the server to respond as if it were that user agent.
So go ahead and customize your content for different browsers or user agents. Or even make up your own - there’s no rule against that. If you want to build a site that displays secret content when requested by “Programmers are Awesome”, then do it.
And yes, it is good to block known “bad bots”. After all, there is no reason to subject your servers to “WebFu**” or “Kozmosbot” traffic unless you really want to.
Just don’t be surprised when “malicious actors” pretend to be someone else. It’s just as easy for their spam factories to pretend to be “Google Chrome” or “Programmers are Awesome” as it is for you.
Here are some sample user agents. Yeah - these don’t all make sense unless you understand the history behind them.
For a more comprehensive list of common user agents, WhatIsMyBrowser.com has a huge database of them.
Here is a small sampling of some “bad” user agents you might want to block:
Customizing content for a specific user in Marketpath CMS is incredibly easy. In your template, simply reference {{ request.user_agent }}. You can even do this directly in HTML content areas using liquid markup!
And if you want to block known bad bots that can be identified by their user agents, we make that as easy as enabling a site setting. Simply navigate to Site -> Site Settings. Scroll down to “Live Settings” and check the “Block Bad User Agents” checkbox.
No worries - we make it painless to try out using our Free plan!
Get Started Today
