About this crawler

This page describes automated web requests you may have seen from the DivergenceResearchBot user agent.

Who is running it

Independent academic research conducted by Damilola Ibosiola. Contact: research@davonix.co.uk. Previous published work in this area includes studies of online complaint and takedown systems (The Web Conference, 2019) and illicit content ecosystems (ICWSM, 2018).

What it is for

A study of whether websites serve different content to AI assistants and agents than they serve to people. As AI systems increasingly read the web on a user's behalf, we are measuring whether the page an assistant sees matches the page a person sees, and where it does not, why.

Findings are published as peer-reviewed academic research. The project does not sell data, does not build a commercial product, and does not use collected content for model training.

What it does

• Requests a small, fixed set of public pages.
• Reads only. It does not submit forms, log in, create accounts, make purchases, click adverts, or interact with payment flows.
• Requests the same page a handful of times with different client configurations, which is the measurement itself.
• Respects robots.txt.
• Limits itself to one connection per site at a time, with a minimum interval between requests.

How to verify it is really us

Our requests are not cryptographically signed. Every request carries our contact address in a From header and this page's address in its User-Agent, but neither proves anything on its own, and we would rather say so than imply a guarantee we are not offering. If you need to confirm that particular traffic was ours, email us with a timestamp and we will tell you.

Important: some of our measurement requests deliberately present the user agent of a third-party AI crawler, because measuring how sites respond to those identities is the point of the study. Those requests always carry our contact address in a From header. Equally, a request claiming to be from a major AI company may not be from that company at all, and may not be from us either. Impersonation of AI crawler user agents by unrelated parties is common, which is part of what this research examines.

How to opt out

Email research@davonix.co.uk with the domain or domains you want excluded. No justification needed. We apply exclusions within one working day, across all current and future parts of the project, and we confirm when it is done.

You can also block us conventionally:

User-agent: DivergenceResearchBot
Disallow: /

Ethics

The studies have institutional ethics approval, granted by Queen Mary University of London (IRB), reference IRB/26/EECS/03288, 2026-08-17. On top of that we work to a published protocol, based on the Menlo Report and adopted before data collection began, which sets out what we do in the field in more detail than any approval process examines. It is available in full at:

https://davonix.co.uk/crawler/ethics

It sets out, among other things, why some requests carry third-party user agents, how we treat robots.txt, and what we do if we find something that should be reported to a site owner rather than only published.

Reporting a problem

If this crawler has caused load problems, triggered alerts, or behaved in a way that looks wrong, please email research@davonix.co.uk. Include a timestamp and, if possible, a log excerpt. We will respond promptly and, if we are at fault, stop immediately.

Last updated: 2026-08-19