Block AI agents and every other web category from one file
An AI agent list answers one question: is this an agent? Your policy asks many more. The filtering domains database labels the 20 million most visited domains with 58 categories, so the rest of your rules get the same care as your AI rules.
Real rows, shown exactly as they appear in the file
AI Agent Blocklist + 20M Filtering Domains Database
Every AI agent domain and device identifier we track, plus 20 million categorized domains for the rest of your web policy.
Quote on request for one organisation
The gap an AI agent list leaves open
Blocking the agent is the first step. But an agent, or the person who runs it, still reaches ordinary websites to get work done. Those websites belong to categories your policy already names.
An AI browser or autonomous agent starts a task with a user's access.
It opens a link shortener, a free proxy or a remote access page.
It uploads a file to a file host or pastes text into a web app.
It reaches a shop, a payment page or a crypto exchange.
Agent domains blocked by the agent list, destinations ruled by category.
Policies name categories
An acceptable use policy rarely stops at AI. It also rules out anonymizers, gambling, adult content and unapproved file sharing.
Agents reach normal sites
Agent traffic ends on file hosts, shops and web apps. Your AI list does not know those, and it should not have to.
Manual lists stay small
A hand-kept list rarely grows past a few hundred domains. The web your users reach is many millions of domains wide.
Two suppliers, two formats
AI data from one vendor and categories from another means two update cycles and two sets of names to align.
One label hides the rest
A code hosting site is also a forum and a learning resource. A single label per domain leads to rules that miss.
Auditors ask for the list
"Which categories do you block?" is easy to answer when every rule points at a named category in one file.
What the 20M filtering domains database contains
One data set with a deliberately simple shape: a domain and its categories. That keeps it easy to load, compare between versions and turn into the lists your devices read.
The file, column by column
- A header row:
Domain,Web Filtering Category - Column one: the domain, one per row
- Column two: its categories, joined by a pipe character
| - Plain text, opens in any spreadsheet, script or database
What comes with it
- The full AI Agent Blocklist: 2,600+ agent domains and 5,300+ device identifiers
- One list per category on request, for devices that read plain lists
- Our category names mapped to yours on request
- Updates daily or monthly, as agreed in your quote
Real rows from the database
These eight rows show the kinds of sites that matter most next to an AI agent policy. Several carry more than one category, which is normal for sites that do more than one thing.
| Domain | Web Filtering Category | Why it matters next to AI agents |
|---|---|---|
replit.com | AI / LLM Tools|Developer Tools / Code Hosting | Also a coding agent in the AI Agent Blocklist, so both files agree on it |
kaggle.com | AI / LLM Tools|Developer Tools / Code Hosting | Data and notebooks: a common place for files to leave |
wetransfer.com | Cloud Storage / File Hosting | A file host an agent or a person can upload to |
box.com | Cloud Storage / File Hosting|Web-based Applications | Often approved by name while the category stays blocked |
anydesk.com | Remote Access Tools | Remote control of a device, a route around local controls |
coinbase.com | Cryptocurrency|Financial Services | A place where an agent with a wallet could spend |
zoho.com | Business|Email and Messaging|Web-based Applications | Three categories: mail, apps and business tools in one domain |
stackoverflow.com | Developer Tools / Code Hosting|Education/Reference|Forums / Message Boards | Developers need it, so allow rules must win here |
See 1,000 rows in the real format. The CSV has the same two columns and the same category names as the full database.
Download 1,000 rows (CSV)All 58 categories, sorted by what an agent could do there
We sorted the categories by the risk they carry once an AI agent, or a person, can act on a site. The suggested action is a starting point. In the file, each domain simply lists its category names.
Help to hide or get in
BlockSites that route traffic around your controls or open a device to someone else.
Where files and text can go
Allow by nameUploads, pasted code and outgoing messages end up here. Keep your sanctioned services open by name and close the category.
Where money moves
ReviewSites where an agent with saved cards, accounts or a wallet could buy, pay or sign up.
Content no workplace allows
BlockOff limits on nearly every network you run, whether the reason is the law, your HR rules or young users.
Time and attention
LimitAllowed in many offices, often limited on guest networks, in schools or during working hours.
Infrastructure and software
Allow, logThe plumbing of the web. Keep it open, and log downloads where your policy asks for it.
Information, learning and daily life
AllowReference, public life and everyday topics. Most policies allow them and use the labels in reports.
Two layers, one policy: the agent list and the categories
The two data sets do different jobs. Use each one where it is strongest, and let them meet in the same firewall, DNS filter or proxy.
AI Agent Blocklist
- 2,600+ AI agent domains in eleven agent types, from AI browsers to MCP servers
- 5,300+ extension, app, package and process identifiers for device policies
- A risk grade on every entry, for staged blocking
- Updated every day as new agents appear
20M Filtering Domains Database
- 20 million domains, the sites your users and agents actually reach
- 58 categories covering security, content, data movement and daily life
- Several categories per domain, so rules match how sites really work
- Updated daily or monthly, as you choose
| Question your policy asks | Which file answers it |
|---|---|
| Is this an AI browser, agent or MCP server? | AI Agent Blocklist, with the agent type |
| Is this agent extension or app on a laptop? | AI Agent Blocklist device identifiers |
| Is this a file host, a proxy or a remote access tool? | 20M database categories |
| Is this site adult, gambling or otherwise off limits? | 20M database categories |
| What kind of sites did a team use last month? | 20M database, joined to your logs |
| Is this a general AI tool rather than an agent? | The AI / LLM Tools category in the 20M database |
New to agent blocking? Start with our guide to blocking AI agents, and see the full contents of the AI Agent Blocklist.
How to write rules when a domain has several categories
A row with three categories belongs to three lists. That is the point: the domain is caught by whichever category your policy cares about. It also means rule order matters.
Put named allows first
Services you approved, such as your company file host or your code platform, go at the top. They must win against any category block below them.
Then block the agent list
AI agents and AI browsers are blocked by the agent list, whatever categories their domains also carry.
Then block risky categories
Anonymizers, remote access, link shorteners and the content you never allow. A domain in any of them is stopped.
Then limit or log the rest
Time and money categories get limits or logging. Everything else stays open and labelled for reports.
Turning the CSV into one list per category takes one line in most shells. The pattern below matches a category name whether it is first, last or in the middle of the second column:
# one plain list for one category (file names are your own)
awk -F, 'NR>1 && ("|" $2 "|") ~ /\|Remote Access Tools\|/ {print $1}' domains.csv > remote_access_tools.txt
# count the domains in each category
awk -F, 'NR>1 {n=split($2,c,"|"); for(i=1;i<=n;i++) k[c[i]]++} END {for(x in k) print k[x], x}' domains.csv | sort -rn
Monitor first, then block. Load a new category with a log or monitor action for a few days, check what it would have stopped, then switch it to block. Allow rules you forgot show up quickly that way.
Where each part of the file goes
The same CSV feeds four kinds of tools. Pick the place by the size of the category and by where your users connect.
Firewalls
Small, high-risk categories such as remote access and link shorteners fit well as external lists. Check your model's list size limit first.
DNS filters
Large content categories belong here, because resolvers handle long lists well. Our DNS guide shows the agent side.
Proxies
Data movement categories with named exceptions sit well in a proxy, where you can allow one file host and block the others.
Logs and reports
Join the CSV to your DNS or proxy logs and every line gets a category. That turns raw logs into a monthly picture of web use.
Firewall owners can follow our firewall guide for the agent list, then add category lists the same way.
Daily or monthly updates: choose by what you do with the file
Both rhythms deliver the full database. The difference is how fresh your rules need to be.
For blocking rules on firewalls, DNS filters and proxies. New sites get their categories quickly, so new file hosts and proxies are covered soon after they appear.
Fits teams that already pull the AI Agent Blocklist every day.
For reporting, review and log enrichment, where a stable monthly version is easier to compare and keep on file.
Fits audit teams and organisations with a monthly change window.
Replace, do not append. Load each new version in full instead of adding lines to the old one. Domains change category over time, and a full replace keeps your lists honest.
Five mistakes to avoid when you add categories to an agent policy
Most problems with category blocking come from rollout, not from the data. These are the ones we see teams correct in their first month.
| Mistake | What happens | What to do instead |
|---|---|---|
| Blocking AI / LLM Tools as a whole | Assistants the company licensed stop working, and staff look for workarounds | Block agents with the agent list, and allow the AI tools you approved by name |
| Treating file hosting as one decision | The company file host breaks along with the unknown ones | Allow your sanctioned service by name above the category block |
| Skipping the anonymizer category | One proxy site routes around every other rule you wrote | Block VPN / Proxy / Anonymizers on every enforcement point |
| Using only the first category of a row | Sites that do several things slip past rules on their other categories | Split the second column on the pipe character and keep every category |
| Loading an old copy once and forgetting it | New sites stay unlabelled and moved domains keep old labels | Schedule the daily or monthly update and replace the whole file |
Who licenses the agent list and the database together
The combined licence suits teams that run their own enforcement and want one source for AI and for everything else.
Security teams with their own firewall
Agent domains and risky categories in the same policy, from one data source, with one renewal date.
DNS filter and proxy owners
Resolvers and proxies without a category service of their own get the full category picture next to the agent list.
Regulated organisations
Pharma, finance and health teams that must show which categories they block, and why, with files they can version.
Schools and districts
AI agents, adult content and anonymizers handled in one place. See our page on students and AI agents.
Teams with several sites
One data set keeps head office, branches and remote users on the same categories, whatever filter each site runs.
Analysts and SOC teams
Category labels on every log line, so web use, agent attempts and policy hits can be reported side by side.
What the licence covers
Plain terms, set out in your quote. The full text is in our terms of service.
One organisation
Use the agent list and the database in every firewall, DNS filter, proxy and report your organisation runs.
Quote and invoice
A written quote, an invoice to your company and payment by bank transfer or purchase order.
Your category names
On request we map our 58 categories to the names your policy already uses.
Paperwork
Supplier forms, security questionnaires and a data processing agreement if you need one.
No resale
No resale, passing on or publication of the raw data. Product makers ask about a product licence.
Access
After the order we set up your download access within one business day.
Block AI agents and set every other category from one supplier
AI Agent Blocklist + 20M Filtering Domains Database, quoted on request for one organisation.
- 2,600+ agent domains and 5,300+ device identifiers, updated every day
- 20 million domains in 58 categories, as one CSV, daily or monthly
- See the format first: 1,000 rows (CSV)
Ask us or order a plan
Tell us what you need to block and where your rules live. We reply within one business day.
- Files for firewalls, DNS filters, secure web gateways, MDM and browser policies
- Updated every day as new AI browsers, agents and extensions appear
- Plans for one organisation, for MSPs and for product makers, or a hosted filter we run for you
Questions about the filtering domains database
What is a filtering domains database?
A file that tells a web filter what kind of site each domain is. Ours lists the 20 million most visited domains, each with one or more of 58 filtering categories, so your firewall, DNS filter or proxy can allow, block or log by category instead of by hand-typed lists.
Why do I need web categories if I already block AI agents?
The AI agent list stops the agents themselves. Categories cover the places any user or agent can go next: file hosts, remote access tools, link shorteners, anonymizers, shops and payment sites. Most acceptable use policies name those too.
How are several categories shown for one domain?
In the second column of the CSV, separated by a pipe character, for example Developer Tools / Code Hosting|Education/Reference|Forums / Message Boards. A rule on any of those categories matches the domain.
How often is the database updated?
Daily or monthly. You choose the rhythm when we prepare your quote. Daily suits blocking rules, monthly suits reporting and review.
What does the database cost?
It is quoted on request, together with the AI Agent Blocklist, for one organisation. The quote depends on how you use the data and how often you want updates. Send the form on this page and we reply within one business day.
Can we see real rows before we order?
Yes. A CSV with 1,000 rows of the database is linked on this page, in exactly the format of the full file.
Can you use our own category names?
Yes, on request. We map our 58 categories to the category names your policy already uses, so the file fits your rules without renaming.
Replit, Kaggle, WeTransfer, Box, AnyDesk, Coinbase, Zoho and Stack Overflow are trademarks of their respective owners. Their domains appear only as examples of rows in the database.
Names are used only to identify the products discussed. AI Agent Blocklist is not affiliated with, sponsored or endorsed by these companies.