Crawling, Indexing & Technical SEO Patents Reference
85+ patents across 9 categories covering how Google discovers, processes, and indexes web content.
Crawling (12 patents)
Core mechanisms:
- Crawl budget allocated based on page importance (PageRank-weighted)
- Crawl frequency determined by update patterns (fresh content = more frequent crawls)
- Priority crawling for high-value pages (sitemaps signal importance)
Crawl budget factors:
- Site authority and PageRank
- Update frequency (freshness signals)
- Internal link structure (deeply buried pages get fewer crawls)
- Response time and server reliability
- Robots.txt and noindex tag usage
Indexing (15 patents)
Caffeine architecture (2010): Near-real-time indexing system that replaced the batch-based index. Allows new content to appear in search within minutes to hours instead of weeks.
Document processing pipeline:
Crawl → Parse HTML → Extract content + links + metadata → NLP processing →
Entity extraction → Quality scoring → Index entry + ranking signalsIndex partitioning:
- Separate index partitions for fresh content vs established content
- High-quality signals promote pages to better index partitions
- Thin/low-quality pages may remain in supplemental index
URL Canonicalization (8 patents)
| Patent | Description |
|---|---|
| US9081861B2 | URL normalization rules — the definitive patent |
Canonicalization rules from US9081861B2:
- HTTP vs HTTPS → HTTPS preferred (consolidate all to HTTPS)
- www vs non-www → Pick one, redirect the other
- Trailing slashes → Consistent usage site-wide
- URL case sensitivity → Lowercase preferred
- Parameter handling → Filter tracking params, index content-affecting params
- Duplicate URL consolidation → One canonical URL per piece of content
Sitemaps (5 patents)
What Google uses from XML sitemaps:
- Priority attribute: directional hint only (not binding)
- Changefreq: crawl frequency guidance (not binding)
- lastmod: credible if consistently accurate; ignored if always set to today
- Discovery: robots.txt
Sitemap:directive ensures sitemap is found
Duplicate Detection (10 patents)
| Patent | Description |
|---|---|
| US7734627B1 | Document fingerprinting via term relationships |
Types of duplicates detected:
- Exact duplicates: Identical content at different URLs
- Near-duplicates: Minor variations (pagination, session IDs)
- Cross-domain duplicates: Same content on multiple domains
- Syndicated content: Articles republished elsewhere
JavaScript Rendering (8 patents)
The rendering challenge: Google crawls HTML first, then renders JavaScript in a separate queue (can take days to weeks).
Implications:
- Content dependent on JS execution may be delayed in indexing
- Critical content (headings, body text, schema) should be in initial HTML
- Dynamic rendering: Serve pre-rendered HTML to Googlebot, JS to users
- Progressive enhancement: Core content visible without JS, enriched with JS
Real-Time Indexing (6 patents)
Triggers for rapid re-crawl:
- Sitemap ping after publication
- URL Inspection Tool request
- IndexNow-style push notifications
- Breaking news signals (content velocity + query spike)
Mobile-First (5 patents)
Mobile-first indexing implications (active since 2019):
- Google primarily crawls and indexes the mobile version of a page
- Desktop content not on mobile version may not be indexed
- Responsive design: same HTML for both (ideal)
- Mobile UX signals: tap target sizes, text readability, viewport configuration
Additional Processing (16 patents)
| Signal | Patent | Handling |
|---|---|---|
| robots.txt | Multiple | Block/allow crawl access |
| noindex | Multiple | Exclude from index (honored after crawl) |
| nofollow | Multiple | Don't pass PageRank (signal still noted) |
| hreflang | Multiple | Alternate language/region versions |
| Structured data extraction | Multiple | Schema parsed during crawl, powers rich results |
Technical SEO Implications
- Ensure all important pages are crawlable — no robots.txt blocks, no noindex on key pages
- Submit accurate XML sitemap reflecting actual site structure with correct lastmod dates
- Implement canonical tags to prevent duplicate content signals
- Server-side render critical content — JS-dependent content is delayed in indexing
- Mobile-first design is baseline requirement (mobile-first indexing since 2019)
- Monitor crawl budget for large sites — consolidate thin/duplicate pages to preserve budget for important pages
- Use hreflang for international/multilingual content
- Keep URL structure clean and hierarchical (US9081861B2) — consistent, lowercase, no unnecessary parameters
Related Learning Modules
- Module 15: Crawling & Indexing — Full technical coverage
- Module 24: Domain & URL Structure — URL patent deep dive