technical

AIPREF: a common language for AI usage preferences

The IETF AIPREF working group is developing a standardized way for publishers to express how their content should be used by automated systems. This guide covers the vocabulary, the two attachment mechanisms, and how to deploy them while the drafts are still moving.
Originary Team6 min read
Who this is for. Publishers, content platforms, and infrastructure teams who want a vendor-neutral way to declare AI usage preferences in HTTP and robots.txt, and a verifiable record that downstream agents observed those preferences.

Suppose you want search engines to keep indexing your site, but you do not want the same pages used to train generative models. robots.txt can say which crawler may fetch which path. It cannot say what the fetched content may be used for. The IETF AI Preferences (AIPREF) working group is defining both a vocabulary for that kind of preference and mechanisms for attaching it to content.

AIPREF builds on infrastructure publishers already run (robots.txt and HTTP headers) and adds purpose-specific semantics that robots.txt alone cannot express. Unlike informal conventions or platform-specific controls, it is being developed as an Internet-wide specification.

What is AIPREF?

AIPREF consists of two complementary specifications currently in draft at the IETF:

1. Vocabulary specification (draft-ietf-aipref-vocab)

Defines a structured vocabulary for expressing preferences about how content should be used by automated systems. The vocabulary includes categories like bots, train-ai, train-genai, and search, with allow (y) or disallow (n) values.

Latest cited version at writing: draft-ietf-aipref-vocab-06. AIPREF is an active IETF working-group effort; check the working group page for the current draft before integrating.

2. Attachment specification (draft-ietf-aipref-attach)

Specifies how to associate preferences with content using HTTP headers and robots.txt. This includes the Content-Usage HTTP header field and updates to RFC 9309 (robots.txt) to support preference directives.

Latest version: draft-ietf-aipref-attach-03 (September 2025)

Usage categories

The vocabulary defines four primary categories, organized hierarchically:

Automated processing (bots)

The broadest category covering all automated processing of content. This is the parent category for more specific usage types.

Use case: Blanket permission or restriction for any automated access.

AI training (train-ai)

A subset of automated processing specifically for training machine learning models. This includes both generative and non-generative AI systems.

Use case: Allow search indexing but restrict model training.

Generative AI training (train-genai)

A subset of AI training focused specifically on training models that generate synthetic content (text, images, audio, etc.).

Use case: Allow classification models but restrict generative models.

Search (search)

Content indexing and discovery for search applications that direct users to original content locations.

Use case: Maintain search visibility while restricting AI training.

Hierarchical inheritance. Categories inherit from their parents. If you set bots=n but do not specify search, search will inherit the disallow preference. However, explicit values always override inherited ones: bots=n, search=y allows search while disallowing other automated processing.

How to attach preferences

AIPREF defines two mechanisms for associating preferences with content:

1. HTTP Content-Usage header

The most granular method. Add the Content-Usage header to HTTP responses to specify preferences for specific resources:

HTTP/1.1 200 OK
Content-Type: text/html
Content-Usage: train-ai=n

<!DOCTYPE html>
<html>...</html>

Implementation examples

Nginx

location / {
  add_header Content-Usage "train-ai=n" always;
}

Apache (.htaccess)

<IfModule mod_headers.c>
  Header set Content-Usage "train-ai=n"
</IfModule>

Express.js

app.use((req, res, next) => {
  res.setHeader('Content-Usage', 'train-ai=n');
  next();
});

Cloudflare Workers

export default {
  async fetch(request) {
    const response = await fetch(request);
    const headers = new Headers(response.headers);
    headers.set('Content-Usage', 'train-ai=n');
    return new Response(response.body, {
      headers,
      status: response.status
    });
  }
};

2. robots.txt Content-Usage directive

For path-scoped preferences, add Content-Usage directives to your robots.txt file:

User-Agent: *
Allow: /
Content-Usage: train-ai=n

User-Agent: *
Allow: /public-research/
Content-Usage: /public-research/ train-ai=y, train-genai=n
Path matching rules. The robots.txt mechanism uses longest-prefix matching. If a resource path matches multiple Content-Usage directives, the one with the longest matching path prefix applies. This allows you to set site-wide defaults and override them for specific paths.

Preference resolution rules

When preferences come from multiple sources or specify overlapping categories, AIPREF defines clear resolution rules:

1. Explicit values win

An explicit y or n for a category takes precedence over inherited values from parent categories.

2. Specific overrides general

More specific categories override broader ones. If train-genai is not specified, it inherits from train-ai, which inherits from bots.

3. Multiple sources: disallow wins

When combining preferences from HTTP headers and robots.txt, if any source indicates n (disallow), the usage is disallowed. Otherwise, if any indicates y (allow), it is allowed.

4. Unknown is valid

If a category is not specified and cannot be inherited, the preference is "unknown." This is a valid state: not every publisher needs to express preferences for every category.

Example resolution. Given: bots=y, train-ai=n, train-genai=y
  • bots: allow (explicit)
  • train-ai: disallow (explicit, overrides parent)
  • train-genai: allow (explicit, overrides parent train-ai)
  • search: allow (inherits from bots)

Practical considerations

Work in progress

AIPREF is currently in draft status at the IETF. While the core concepts are stable, details may change before final standardization. Early adopters should track the working group's progress and be prepared to update implementations.

Draft versions cited at writing: draft-ietf-aipref-vocab-06 and draft-ietf-aipref-attach. AIPREF is moving; always check the IETF working group page for the current revisions before relying on specific identifier names or syntax in production.

No built-in enforcement

AIPREF provides a mechanism for expressing preferences, not enforcing them. The specification does not define compliance mechanisms, auditing, or consequences for ignoring preferences. Publishers seeking enforcement should layer AIPREF with contracts, terms of service, or technical access controls.

Legal context matters

The specification explicitly notes that preferences do not automatically create legal rights. Recognized priorities (accessibility, security, legal obligations) may override preferences. For example:

  • Accessibility tools may ignore bots=n to serve users with disabilities.
  • Security researchers may process content despite restrictions.
  • Existing licensing agreements supersede AIPREF preferences.

Relationship to other signals

AIPREF complements rather than replaces existing mechanisms:

robots.txt (RFC 9309)

Handles crawl access control. AIPREF extends robots.txt with purpose semantics but does not replace its core function of controlling crawler access.

ai.txt

An informal convention for AI-specific permissions. AIPREF provides a standardized alternative with formal IETF backing and richer semantics.

C2PA / Content Credentials

Handles content provenance and authenticity. AIPREF expresses usage preferences; C2PA verifies content lineage. They answer different questions: AIPREF states how content may be used, and C2PA describes where it came from and how it was made.

Implementation roadmap

For organizations looking to adopt AIPREF today:

Phase 1: express baseline preferences

Start with robots.txt directives for site-wide or path-based preferences. This requires minimal infrastructure changes and provides broad coverage.

User-Agent: *
Content-Usage: train-genai=n, search=y

Phase 2: add HTTP header support

Implement Content-Usage headers at your CDN, reverse proxy, or application layer. This enables resource-specific preferences and more granular control.

Phase 3: document and communicate

Publish your AIPREF policy in human-readable form. Link to it from your terms of service. Make it clear to AI system operators what your preferences are and why.

Phase 4: monitor and enforce

Track which systems respect your preferences. Consider pairing AIPREF with technical access controls (authentication, rate limiting) and legal agreements (licenses, terms of service) for enforcement.

What PEAC does not do

  • PEAC does not author AIPREF; AIPREF is an IETF working-group effort and any cited drafts evolve independently.
  • PEAC does not enforce AIPREF upstream of the publisher; enforcement stays with the publisher and its infrastructure.
  • PEAC does not assert that an agent obeyed a preference; it carries a signed record of what the agent attested at the boundary.
  • PEAC does not replace robots.txt, terms of service, licensing agreements, or legal review.

Originary's position

We support the IETF AIPREF effort. A shared, machine-readable vocabulary lets a publisher state a preference once and have any automated system read it the same way, instead of configuring each platform's controls separately.

Originary reads AIPREF preferences where publishers expose them and pairs preference signals with signed records, so publishers can produce verifiable records of how their content was accessed and used. See the PEAC Protocol overview for the records model and the RFC 9309 guide for how AIPREF composes with crawl access control.

As the specification matures, we will continue to track the working group's progress and update our implementations to stay aligned with the final standard.

Further reading

related