Status: Adopted Version: 1.0 Adopted: 2026-08-17 Framework: The Menlo Report (DHS, 2012) and its Companion Review route: Institutional ethics approval, recorded below
The studies have institutional ethics approval, recorded below. Approval settles whether the research may proceed; it does not settle the operational decisions that determine whether it proceeds well. This protocol is that second layer: the specific commitments the project holds itself to in the field, at a level of detail no approval process examines — what the crawler does and does not touch, how it identifies itself, what happens to a site owner who objects, and what we do if we find something a site owner should hear about before a reviewer does.
Its date matters for the same reason approval's does. A protocol written after data collection reads as justification rather than governance, and reviewers can tell. The adoption date above precedes the first request of the first measurement wave, and the git history of this file is the evidence of that.
| Institution | Body | Reference | Date | Scope |
|---|---|---|---|---|
| Queen Mary University of London | IRB | IRB/26/EECS/03288 | 2026-08-17 | All three studies in section 1 |
Approval is not a licence to stop thinking, so the following hold regardless of what it permits:
robots.txt position — are stated plainly rather than buried, so a reader can
disagree with them specifically.This protocol covers three studies:
No study recruits participants, administers interventions to people, or collects data from private spaces. All three nonetheless carry non-trivial ethical weight, addressed below.
Human subjects. None of the studies engage human subjects directly. There is no interaction with people, no deception of people, and no manipulation of anyone's information environment.
Personal data appears incidentally and is minimised. Takedown notices name senders, complainants and targets; removed content may concern identifiable individuals; crawled pages may contain personal data. Mitigations:
Note on the "subject" of active measurement. In parallel_web the entity
whose behaviour we observe is a website operator, not a person in the
human-subjects sense. We nonetheless treat operators as stakeholders with
legitimate interests: they should be able to identify our traffic, understand
it, and stop it. See §3.
Anticipated benefits. Each study addresses a documented gap with a public interest dimension: whether the agentic web is being used to deceive at scale; whether copyright process is being used to suppress journalism; and whether regulator-facing removal claims correspond to reality. In each case the beneficiaries are the public, platform operators, and regulators, none of whom currently have this evidence.
Anticipated harms and mitigations.
| Harm | Mitigation |
|---|---|
| Load imposed on crawled sites | Per-domain rate limiting (default 1 concurrent, ≥5 s interval), no recursive crawling, bounded URL sets, off-peak scheduling where identifiable |
| Distortion of a site's analytics or business metrics | No form submission, no authentication, no transactions, no ad clicks, no interaction with payment flows |
| Our traffic mistaken for an attack | Descriptive User-Agent with contact URL on every request; Web Bot Auth signatures where applicable; a published crawler page; prompt response to any operator contact |
| Publishing an exploitable technique | Coordinated disclosure before publication; see §5 |
| Misattributing deception to an innocent operator | Divergence claims require exceeding an empirically established noise floor; findings that name an operator require manual review and a right of reply |
| Amplifying a censorship campaign by describing it | Reporting focuses on method and scale rather than reproducing the suppressed material or the suppression playbook in operational detail |
Harms of not doing the work are also weighed. Agent-targeted deception, fraudulent takedown campaigns and unverified removal claims all currently operate without measurement, which is itself a harm to the public interest.
Burdens fall on website operators and platforms; benefits accrue broadly. Sampling is stratified and documented so that burden is not concentrated arbitrarily on small or under-resourced operators: rate limits are absolute rather than proportional to site size, meaning large sites absorb a proportionally smaller share.
Where a study population is skewed (e.g. abuse-enriched seeds), this is declared in the paper rather than presented as representative of the web.
Compliance. No circumvention of access controls, no authentication to services we are not entitled to use, no violation of paywalls, no access to non-public data. All notice data is obtained through sanctioned channels under their published terms.
Terms of service. Lumen API use is governed by its terms; the project complies and does not redistribute data contrary to them. Where a site's terms forbid automated access, see the robots.txt position in §3.
Transparency. Methods, the profile matrix, the ethics protocol and the disclosure record are published with each paper.
Reviewers will focus on these. Both are stated plainly rather than buried.
Several parallel_web arms send user-agent strings belonging to third parties: a
training crawler and a user-triggered agent for each of three vendors (GPTBot
and ChatGPT-User; PerplexityBot and Perplexity-User; ClaudeBot and
Claude-User), plus Googlebot as the established-crawler control.
Why it is necessary. The research question is whether sites serve different content when they believe an AI agent is asking. That question cannot be answered without presenting the identity a site keys on. There is no observational substitute: we cannot ask operators, because self-report is precisely what would be unreliable, and we cannot obtain the agent operators' own traffic.
Why the harm is limited. These arms fetch public pages, read-only, at low rates. They do not authenticate, transact, or consume anything of value. The impersonated parties are large operators whose crawlers already fetch these pages legitimately; our volume is negligible against theirs.
Mitigations.
- These arms are rate-limited below the honest arms.
- Every request carries a From header with our contact address, so the
operator can always reach us even when the UA is borrowed.
- We never claim Web Bot Auth attestation for an identity that is not ours.
Signed requests are only ever signed as us. Forging attestation would cross
from measurement into fraud.
- The agent operators whose identities are used are notified through the
disclosure process before publication.
Alternatives considered and rejected. Using only our own honest agent identity would measure how sites treat us, not how they treat the agents that matter, and would miss the entire phenomenon. Coordinating with agent operators to use their infrastructure was considered infeasible and would compromise independence.
Default position: honoured by all arms. This is the stricter choice and it is the default in the profile matrix.
A tension exists, and it is declared rather than hidden: a site that blocks
GPTBot in robots.txt and also serves that agent deceptive content is
precisely the interesting case, and honouring robots.txt means we do not
observe it. We accept that loss of coverage rather than override operator
preference by default.
Any exception must be enumerated in the profile matrix with a written justification, approved by the external reviewers, and reported in the paper. The presumption is against exceptions.
Before submission, and in any case promptly after a finding is confirmed:
A disclosure log is maintained at docs/ethics/disclosure-log.md, recording
date, recipient, content and response.
Operators may request exclusion by contacting the address on the crawler page. Requests are honoured within one working day, applied across all studies, and recorded in an exclusion list committed to the repository. No justification is required from the requester. Exclusions are reported as a count in the paper so their effect on coverage is visible.
This protocol is revisited when: a new study is added; the profile matrix gains an arm with a new ethical character; an operator complains; a disclosure produces an unexpected response; or twelve months elapse.