20M filtering domains database

Block AI agents and every other web category from one file

An AI agent list answers one question: is this an agent? Your policy asks many more. The filtering domains database labels the 20 million most visited domains with 58 categories, so the rest of your rules get the same care as your AI rules.

domains.csv2 columns
DomainWeb Filtering Category
replit.comAI / LLM Tools|Developer Tools / Code Hosting
wetransfer.comCloud Storage / File Hosting
anydesk.comRemote Access Tools
coinbase.comCryptocurrency|Financial Services
stackoverflow.comDeveloper Tools / Code Hosting|Education/Reference|Forums / Message Boards

Real rows, shown exactly as they appear in the file

Next step

AI Agent Blocklist + 20M Filtering Domains Database

Every AI agent domain and device identifier we track, plus 20 million categorized domains for the rest of your web policy.

Quote on request for one organisation

The gap an AI agent list leaves open

Blocking the agent is the first step. But an agent, or the person who runs it, still reaches ordinary websites to get work done. Those websites belong to categories your policy already names.

The agent

An AI browser or autonomous agent starts a task with a user's access.

Finds a route

It opens a link shortener, a free proxy or a remote access page.

Moves data

It uploads a file to a file host or pastes text into a web app.

Spends or signs up

It reaches a shop, a payment page or a crypto exchange.

Your answer

Agent domains blocked by the agent list, destinations ruled by category.

Policies name categories

An acceptable use policy rarely stops at AI. It also rules out anonymizers, gambling, adult content and unapproved file sharing.

Agents reach normal sites

Agent traffic ends on file hosts, shops and web apps. Your AI list does not know those, and it should not have to.

Manual lists stay small

A hand-kept list rarely grows past a few hundred domains. The web your users reach is many millions of domains wide.

Two suppliers, two formats

AI data from one vendor and categories from another means two update cycles and two sets of names to align.

One label hides the rest

A code hosting site is also a forum and a learning resource. A single label per domain leads to rules that miss.

Auditors ask for the list

"Which categories do you block?" is easy to answer when every rule points at a named category in one file.

What the 20M filtering domains database contains

One data set with a deliberately simple shape: a domain and its categories. That keeps it easy to load, compare between versions and turn into the lists your devices read.

20Mmost visited domains
58filtering categories
1 to 4categories on a typical row
CSVone file, two columns
Dailyor monthly updates

The file, column by column

  • A header row: Domain,Web Filtering Category
  • Column one: the domain, one per row
  • Column two: its categories, joined by a pipe character |
  • Plain text, opens in any spreadsheet, script or database

What comes with it

  • The full AI Agent Blocklist: 2,600+ agent domains and 5,300+ device identifiers
  • One list per category on request, for devices that read plain lists
  • Our category names mapped to yours on request
  • Updates daily or monthly, as agreed in your quote

Real rows from the database

These eight rows show the kinds of sites that matter most next to an AI agent policy. Several carry more than one category, which is normal for sites that do more than one thing.

DomainWeb Filtering CategoryWhy it matters next to AI agents
replit.comAI / LLM Tools|Developer Tools / Code HostingAlso a coding agent in the AI Agent Blocklist, so both files agree on it
kaggle.comAI / LLM Tools|Developer Tools / Code HostingData and notebooks: a common place for files to leave
wetransfer.comCloud Storage / File HostingA file host an agent or a person can upload to
box.comCloud Storage / File Hosting|Web-based ApplicationsOften approved by name while the category stays blocked
anydesk.comRemote Access ToolsRemote control of a device, a route around local controls
coinbase.comCryptocurrency|Financial ServicesA place where an agent with a wallet could spend
zoho.comBusiness|Email and Messaging|Web-based ApplicationsThree categories: mail, apps and business tools in one domain
stackoverflow.comDeveloper Tools / Code Hosting|Education/Reference|Forums / Message BoardsDevelopers need it, so allow rules must win here

See 1,000 rows in the real format. The CSV has the same two columns and the same category names as the full database.

Download 1,000 rows (CSV)

All 58 categories, sorted by what an agent could do there

We sorted the categories by the risk they carry once an AI agent, or a person, can act on a site. The suggested action is a starting point. In the file, each domain simply lists its category names.

Help to hide or get in

Block

Sites that route traffic around your controls or open a device to someone else.

Parked / For-Sale DomainsRemote Access ToolsRestricted / SuspiciousURL ShortenersVPN / Proxy / Anonymizers

Where files and text can go

Allow by name

Uploads, pasted code and outgoing messages end up here. Keep your sanctioned services open by name and close the category.

AI / LLM ToolsCloud Storage / File HostingDeveloper Tools / Code HostingEmail and MessagingFile Sharing/P2PWeb-based Applications

Where money moves

Review

Sites where an agent with saved cards, accounts or a wallet could buy, pay or sign up.

Advertising/MarketingCryptocurrencyFinancial ServicesJob Search/CareersMarketplaces / Classifieds / AuctionsOnline BankingReal EstateShopping

Content no workplace allows

Block

Off limits on nearly every network you run, whether the reason is the law, your HR rules or young users.

AdultAlcoholDatingGamblingHate/DiscriminationIllegal DrugsLingerie/Swimsuits/Intimate ApparelTobaccoViolenceWeapons

Time and attention

Limit

Allowed in many offices, often limited on guest networks, in schools or during working hours.

Arts & EntertainmentBlogs/Personal SitesCommunity SitesForums / Message BoardsGamingPodcastsSocial NetworkingStreaming Media

Infrastructure and software

Allow, log

The plumbing of the web. Keep it open, and log downloads where your policy asks for it.

BusinessCDN / Hosting / InfrastructureComputers & TechnologyFreeware/SoftwareSearch Engines & Platforms

Information, learning and daily life

Allow

Reference, public life and everyday topics. Most policies allow them and use the labels in reports.

Automotive & TransportationEducation/ReferenceFood & DrinkGovernment & LegalHealth & MedicineHome & GardenMilitaryNews/MediaPets & AnimalsPoliticalReligion & SpiritualityRestaurants & DiningScienceSports & RecreationTravelWebinars/E-learning

Two layers, one policy: the agent list and the categories

The two data sets do different jobs. Use each one where it is strongest, and let them meet in the same firewall, DNS filter or proxy.

AI Agent Blocklist

  • 2,600+ AI agent domains in eleven agent types, from AI browsers to MCP servers
  • 5,300+ extension, app, package and process identifiers for device policies
  • A risk grade on every entry, for staged blocking
  • Updated every day as new agents appear

20M Filtering Domains Database

  • 20 million domains, the sites your users and agents actually reach
  • 58 categories covering security, content, data movement and daily life
  • Several categories per domain, so rules match how sites really work
  • Updated daily or monthly, as you choose
Question your policy asksWhich file answers it
Is this an AI browser, agent or MCP server?AI Agent Blocklist, with the agent type
Is this agent extension or app on a laptop?AI Agent Blocklist device identifiers
Is this a file host, a proxy or a remote access tool?20M database categories
Is this site adult, gambling or otherwise off limits?20M database categories
What kind of sites did a team use last month?20M database, joined to your logs
Is this a general AI tool rather than an agent?The AI / LLM Tools category in the 20M database

New to agent blocking? Start with our guide to blocking AI agents, and see the full contents of the AI Agent Blocklist.

How to write rules when a domain has several categories

A row with three categories belongs to three lists. That is the point: the domain is caught by whichever category your policy cares about. It also means rule order matters.

  1. Put named allows first

    Services you approved, such as your company file host or your code platform, go at the top. They must win against any category block below them.

  2. Then block the agent list

    AI agents and AI browsers are blocked by the agent list, whatever categories their domains also carry.

  3. Then block risky categories

    Anonymizers, remote access, link shorteners and the content you never allow. A domain in any of them is stopped.

  4. Then limit or log the rest

    Time and money categories get limits or logging. Everything else stays open and labelled for reports.

Turning the CSV into one list per category takes one line in most shells. The pattern below matches a category name whether it is first, last or in the middle of the second column:

# one plain list for one category (file names are your own)
awk -F, 'NR>1 && ("|" $2 "|") ~ /\|Remote Access Tools\|/ {print $1}' domains.csv > remote_access_tools.txt

# count the domains in each category
awk -F, 'NR>1 {n=split($2,c,"|"); for(i=1;i<=n;i++) k[c[i]]++} END {for(x in k) print k[x], x}' domains.csv | sort -rn

Monitor first, then block. Load a new category with a log or monitor action for a few days, check what it would have stopped, then switch it to block. Allow rules you forgot show up quickly that way.

Where each part of the file goes

The same CSV feeds four kinds of tools. Pick the place by the size of the category and by where your users connect.

Firewalls

Small, high-risk categories such as remote access and link shorteners fit well as external lists. Check your model's list size limit first.

DNS filters

Large content categories belong here, because resolvers handle long lists well. Our DNS guide shows the agent side.

Proxies

Data movement categories with named exceptions sit well in a proxy, where you can allow one file host and block the others.

Logs and reports

Join the CSV to your DNS or proxy logs and every line gets a category. That turns raw logs into a monthly picture of web use.

Firewall owners can follow our firewall guide for the agent list, then add category lists the same way.

Daily or monthly updates: choose by what you do with the file

Both rhythms deliver the full database. The difference is how fresh your rules need to be.

Daily

For blocking rules on firewalls, DNS filters and proxies. New sites get their categories quickly, so new file hosts and proxies are covered soon after they appear.

Fits teams that already pull the AI Agent Blocklist every day.

Monthly

For reporting, review and log enrichment, where a stable monthly version is easier to compare and keep on file.

Fits audit teams and organisations with a monthly change window.

Replace, do not append. Load each new version in full instead of adding lines to the old one. Domains change category over time, and a full replace keeps your lists honest.

Five mistakes to avoid when you add categories to an agent policy

Most problems with category blocking come from rollout, not from the data. These are the ones we see teams correct in their first month.

MistakeWhat happensWhat to do instead
Blocking AI / LLM Tools as a wholeAssistants the company licensed stop working, and staff look for workaroundsBlock agents with the agent list, and allow the AI tools you approved by name
Treating file hosting as one decisionThe company file host breaks along with the unknown onesAllow your sanctioned service by name above the category block
Skipping the anonymizer categoryOne proxy site routes around every other rule you wroteBlock VPN / Proxy / Anonymizers on every enforcement point
Using only the first category of a rowSites that do several things slip past rules on their other categoriesSplit the second column on the pipe character and keep every category
Loading an old copy once and forgetting itNew sites stay unlabelled and moved domains keep old labelsSchedule the daily or monthly update and replace the whole file

Who licenses the agent list and the database together

The combined licence suits teams that run their own enforcement and want one source for AI and for everything else.

Security teams with their own firewall

Agent domains and risky categories in the same policy, from one data source, with one renewal date.

DNS filter and proxy owners

Resolvers and proxies without a category service of their own get the full category picture next to the agent list.

Regulated organisations

Pharma, finance and health teams that must show which categories they block, and why, with files they can version.

Schools and districts

AI agents, adult content and anonymizers handled in one place. See our page on students and AI agents.

Teams with several sites

One data set keeps head office, branches and remote users on the same categories, whatever filter each site runs.

Analysts and SOC teams

Category labels on every log line, so web use, agent attempts and policy hits can be reported side by side.

What the licence covers

Plain terms, set out in your quote. The full text is in our terms of service.

One organisation

Use the agent list and the database in every firewall, DNS filter, proxy and report your organisation runs.

Quote and invoice

A written quote, an invoice to your company and payment by bank transfer or purchase order.

Your category names

On request we map our 58 categories to the names your policy already uses.

Paperwork

Supplier forms, security questionnaires and a data processing agreement if you need one.

No resale

No resale, passing on or publication of the raw data. Product makers ask about a product licence.

Access

After the order we set up your download access within one business day.

Block AI agents and set every other category from one supplier

AI Agent Blocklist + 20M Filtering Domains Database, quoted on request for one organisation.

  • 2,600+ agent domains and 5,300+ device identifiers, updated every day
  • 20 million domains in 58 categories, as one CSV, daily or monthly
  • See the format first: 1,000 rows (CSV)

Ask us or order a plan

Tell us what you need to block and where your rules live. We reply within one business day.

  • Files for firewalls, DNS filters, secure web gateways, MDM and browser policies
  • Updated every day as new AI browsers, agents and extensions appear
  • Plans for one organisation, for MSPs and for product makers, or a hosted filter we run for you

We use your details only to answer you. No newsletters.

Questions about the filtering domains database

What is a filtering domains database?

A file that tells a web filter what kind of site each domain is. Ours lists the 20 million most visited domains, each with one or more of 58 filtering categories, so your firewall, DNS filter or proxy can allow, block or log by category instead of by hand-typed lists.

Why do I need web categories if I already block AI agents?

The AI agent list stops the agents themselves. Categories cover the places any user or agent can go next: file hosts, remote access tools, link shorteners, anonymizers, shops and payment sites. Most acceptable use policies name those too.

How are several categories shown for one domain?

In the second column of the CSV, separated by a pipe character, for example Developer Tools / Code Hosting|Education/Reference|Forums / Message Boards. A rule on any of those categories matches the domain.

How often is the database updated?

Daily or monthly. You choose the rhythm when we prepare your quote. Daily suits blocking rules, monthly suits reporting and review.

What does the database cost?

It is quoted on request, together with the AI Agent Blocklist, for one organisation. The quote depends on how you use the data and how often you want updates. Send the form on this page and we reply within one business day.

Can we see real rows before we order?

Yes. A CSV with 1,000 rows of the database is linked on this page, in exactly the format of the full file.

Can you use our own category names?

Yes, on request. We map our 58 categories to the category names your policy already uses, so the file fits your rules without renaming.

Replit, Kaggle, WeTransfer, Box, AnyDesk, Coinbase, Zoho and Stack Overflow are trademarks of their respective owners. Their domains appear only as examples of rows in the database.

Names are used only to identify the products discussed. AI Agent Blocklist is not affiliated with, sponsored or endorsed by these companies.