Consultation
We respond within 15 minutes
4. Technical criteria: what AI crawlers scan
Technical SEO in 2026 has added a new dimension — in addition to the classic Googlebot, there is now an ecosystem of AI crawlers, each with its own crawling logic and its own requirements for content accessibility.
Crawlers and robots.txt
The current list of AI agents that must be considered in robots.txt:
GPTBot – OpenAI / ChatGPT Search
ClaudeBot – Anthropic
PerplexityBot – Perplexity
GoogleOther – Google AI (separate from Googlebot)
Applebot – Apple Intelligence
YouBot – You.com
A critically important point: many websites configured several years ago have lines in robots.txt such as Disallow: / for unknown agents or use a whitelist approach. In this case all AI crawlers are blocked by default — the site exists for Google, but is invisible to the entire AI ecosystem.
Checking and explicitly allowing each agent is the first technical step of an AI search audit.
JavaScript rendering — a critical point
Classic Googlebot has long been able to render JavaScript, although with a delay. AI crawlers are heterogeneous. Perplexity Bot and several other agents handle JS significantly worse or do not handle it at all. If page content is generated by client-side JavaScript — a significant portion of AI crawlers sees an empty page.
For Next.js developers this is a direct architectural requirement: SSR (Server-Side Rendering) or SSG (Static Site Generation) is not just about improving LCP — it is a requirement for content accessibility for AI crawlers. A site built on pure client-side React with content loaded via API after page load is technically invisible to part of the AI ecosystem.
Practical check: disable JavaScript in the browser and open the page. If the main content disappears — AI crawlers will have problems with it.
Structured Data as a direct feed for AI
Schema.org markup in 2026 serves a dual function: classic rich snippets in SERP plus a direct feed for entity recognition in AI systems. The difference is how AI uses this data — not for display, but for building semantic relationships between entities.
Priority markup types for AI visibility:
Organization / LocalBusiness — sameAs links to verified profiles (Wikipedia, Wikidata, LinkedIn, Crunchbase) build the entity graph of your brand. The more authoritative points confirm the existence of the entity — the higher the trust of the AI system.
Person — for content authors. Author schema with sameAs links to LinkedIn, Google Scholar, industry profiles. AI verifies author expertise through the graph, not through article text.
Article / BlogPosting — with explicit datePublished and dateModified. Content freshness is one of the key signals for Perplexity and ChatGPT Search. A page without an explicit publication date loses this signal entirely.
FAQPage — structured questions and answers are directly extracted by AI models as ready-made answer units. This is one of the most effective formats for appearing in AI citations.
Core Web Vitals: holistic scoring from March 2026
Until March 2026, Google evaluated CWV at the level of individual URLs. The standard practice was to optimize the top 50 landing pages and ignore the rest. With the March 2026 update, this strategy stopped working.
Google moved to domain-level aggregation: the performance of the entire domain forms a single weighted score. High-traffic pages have more weight, but slow pages anywhere on the site contribute a negative signal to the aggregate. If 30% of indexed URLs fail LCP — this drags down the entire domain, including optimized pages.
Practical consequence: audits must cover the entire domain, not a sample. The tool is CrUX data in Google Search Console at origin level, not URL level. Optimization must be done at the template level: fixing one blog template fixes thousands of URLs at once.
Current thresholds and reality:
Metric | Good | Market reality
LCP | < 2.5s | 62% of mobile pages pass
INP | < 200ms | 57% of sites pass — the most failing metric
CLS | < 0.1 | highest pass rate
For the Next.js stack, specific optimization points: priority prop on hero images (removes lazy loading from LCP element), font-display: swap in next/font, explicit width/height on all media elements for CLS, minimization of work in main thread for INP via Server Components.
llms.txt — new standard or marketing?
In 2024–2025 an initiative appeared — llms.txt — a file in the root of the site with a structured description of content specifically for AI agents, analogous to sitemap.xml. Google officially stated that it does not use llms.txt as a ranking signal. Perplexity and several other systems declare support.
Practical position: implementing llms.txt is minimal effort with potential upside for non-Google AI engines. Not a priority, but reasonable hygiene.
5. On-Page in AI context: from keyword density to entity coverage
This is perhaps the most radical shift in on-page optimization in the last ten years. Not evolution — but a paradigm shift. Keyword density, LSI keywords, exact match in headings — all of this has not died, but has moved to the background. At the forefront is a concept that can be formulated as follows: a page must comprehensively cover a topic as a semantic entity, not just contain target keywords.
From keywords to semantic entities
Google Knowledge Graph contains billions of entities — people, places, organizations, concepts, products — and relationships between them. When a user enters a query, Google increasingly interprets it through the lens of entities, not text strings. “Best coffee” is not just three words, it is a query with entities: coffee as a product, local user context, intent to buy or find a place.
A page that covers a topic as an entity explicitly connects its content with these entities — through terminology, through schema, through links to authoritative sources, through mentions of related concepts. AI models, when analyzing a page, build an internal entity graph and evaluate the completeness of topic coverage.
Practically this means: before writing content, you need to define not only keywords, but also the semantic field of the topic — which related entities, concepts, questions must be covered on the page. Tools like Google NLP API or similar allow checking how the algorithm interprets entities on existing pages.
Content structure as navigation for AI parsing
A language model, when extracting a citation, does not read a page like a human — it looks for extractable answer units. Page structure determines how easily the model can find and extract the needed fragment.
Several concrete principles:
Direct answer at the beginning of the paragraph. The journalistic inverted pyramid — first the main point, then details — works perfectly for AI extraction. A paragraph that starts with context and reaches the answer at the end is poorly extracted. A paragraph that gives a clear answer in the first sentence and then expands it works well.
H2/H3 as questions or clear statements. A heading like “Technical aspects” is bad. A heading like “How loading speed affects ranking” or “Three technical factors critical for AI crawlers” is good. The model uses the heading as a label for the content under it.
Definitions in explicit format. When a page provides a definition of a concept, it is better to structure it: bold term, colon, definition. This is a standard pattern that models recognize as a definition of an entity.
Tables for comparisons. Tabular format is one of the best for AI extraction of structured data. Comparison of tools, metrics, characteristics in a table is extracted significantly more effectively than the same data in text form.
Thin content vs comprehensive: a turning point
There is a common misconception that “long content ranks better.” This is not entirely correct. More precisely: comprehensive content ranks better than superficial content, and comprehensive content is by definition longer.
The difference between long and comprehensive content is critical. An article of 4000 words that repeats the same ideas with different wording is long superficial content. It does not gain advantage. An article of 2500 words that covers the topic from multiple angles, answers related questions, and contains original data is comprehensive content. It gains advantage.
The metric to use instead of word count: topic coverage score — the percentage of semantically related questions on the topic that the page answers. Tools like Surfer SEO, Clearscope, or manual analysis via People Also Ask approximate this metric.
Update frequency as a signal
AI systems, especially Perplexity and ChatGPT Search, actively take content freshness into account. But “freshness” in AI context is not just publication date, it is verifiable information relevance.
A page published in 2022 and not updated since then, even with good rankings and strong backlink profile, loses to a fresher page when selecting a source for citation in an AI answer to a current query. Especially if the topic is subject to change — technology, legislation, market data.
Practice of leading SEO teams in 2026: a revision cycle for key pages once per quarter. Not rewriting — but updating data, adding current examples, correcting outdated statements with explicit dateModified updates in schema. This signals to crawlers that the content is actively maintained.
Factual density vs informational filler
One non-obvious shift: AI models, when evaluating source quality, consider the ratio of factual information to text volume. Content with a high share of introductory phrases, transitions, repetitions has low factual density. Content where each paragraph carries concrete information has high density.
This does not mean you should write in telegraphic style. It means that template introductions like “In the modern world SEO plays an important role…”, repeated conclusions, and padding content actively harm positioning as an AI citation source.
Maryan Polyak
Consultation
We respond within 15 minutes
Q&A on SEO in the Age of AI Search
Section 1: Search Transformation and Interface
Question: What is the AI Overview layer (formerly SGE) in Google, and how is it connected to classic search results?
Answer: AI Overview is an artificial intelligence layer powered by the Gemini model that generates a synthesized response above the organic search results. It does not index the internet from scratch – it pulls data from the top 10 classic organic results. Thus, traditional SEO serves as the foundation: without strong positions in the top 10, appearing in the AI block is practically impossible.
Question: What is the “zero-click” phenomenon in the AI era, and how does it affect traffic?
Answer: This is a situation where a user gets a complete answer directly on the search results page and does not click through to a website. With the rollout of AI Overview, the click-through rate (CTR) for informational queries dropped by 40-60%. However, this is not the death of SEO for websites, but rather a shift in the user’s entry point: appearing in AI responses drives brand awareness and branded search growth.
Question: For which types of search queries does Google most frequently display AI responses, and for which does it rarely do so?
Answer: AI is triggered for informational queries (how, why, what is), comparisons (X vs Y), research topics, and complex multi-part questions. AI is rarely triggered for transactional queries (buy, price), navigational queries (searching for a specific website), branded queries, and local commercial queries (restaurant near me).
Section 2: The Ecosystem of Alternative Search Engines
Question: How does Perplexity’s ranking logic fundamentally differ from Google’s?
Answer: Perplexity does not depend on Google’s index and uses its own crawler (PerplexityBot). The system focuses on the freshness, relevance, and citation frequency of data. It gives clear priority to materials with precise, recent publication dates, academic sources, research papers, and original industry statistics.
Question: What specific nuance should be considered when optimizing for ChatGPT Search?
Answer: Microsoft Bing serves as the core index for ChatGPT Search, on top of which OpenAI applies its own re-ranking layer. To gain visibility in ChatGPT, it is critical to work with Bing Webmaster Tools, optimize metadata, and account for social signals (especially LinkedIn for B2B). It is also necessary to explicitly allow the GPTBot crawler in the robots.txt file.
Section 3: Technical SEO and Website Requirements
Question: What critical error in the robots.txt file can make a website invisible to AI search engines?
Answer: Using outdated directives like Disallow: / for unknown user-agents or relying on overly strict whitelisting. In 2026, you must explicitly grant access to AI crawlers such as GPTBot, ClaudeBot, PerplexityBot, GoogleOther, and Applebot.
Question: How does development architecture (specifically Next.js/React) affect AI bot crawling?
Answer: AI crawlers (like PerplexityBot) handle client-side JavaScript much worse than the classic Googlebot, or do not process it at all. Therefore, using SSR (Server-Side Rendering) or SSG (Static Site Generation) has become a strict technical requirement. If content is rendered only on the client side (CSR), the AI bot will see a blank page.
Question: How has Google’s approach to evaluating Core Web Vitals (CWV) changed since March 2026?
Answer: Google shifted from evaluating individual URLs to aggregating metrics at the entire domain level. Slow, unoptimized pages in any section of a website now negatively impact the overall domain score. Optimization must be carried out centrally at the template level (e.g., configuring image and font properties in Next.js).
Section 4: On-Page Content Optimization for AI
Question: What does the shift from “keywords” to “semantic entities” mean?
Answer: Algorithms no longer look for simple text matches. They map queries against the Knowledge Graph (billions of entities: brands, concepts, people) and evaluate how comprehensively and deeply a page covers the entire semantic field and related concepts, rather than just containing isolated keywords.
Question: What are the main rules for structuring text so an LLM can easily extract an answer from it?
Answer: Use the inverted pyramid principle: a clear, direct answer is provided in the very first sentence of a paragraph, followed by details. Format H2/H3 headings as specific questions or statements. Format definitions clearly using a colon (Term: definition). Use tables to compare characteristics.
Question: What is the “factual density” of content, and how does it affect AI ranking?
Answer: It is the ratio of useful, verifiable information to the total volume of text. AI models penalize informational fluff (wordiness, cookie-cutter intros like “In today’s world…”, and repetitions). Text must be highly concise, with every paragraph delivering new facts.
Section 5: Authority (EEAT) and Link Building
Question: How do algorithms technically measure EEAT parameters in 2026?
Answer: EEAT does not measure the text itself, but rather the authority of the entity (author or brand) within the Knowledge Graph. This is built on external, verifiable signals: mentions on authoritative resources, connections via sameAs markup, the entity’s presence in Wikipedia/Wikidata, and the author’s profiles on LinkedIn or Google Scholar.
Question: Why is using the sameAs property in Schema.org microdata so important?
Answer: The sameAs array of links in Organization or Person schemas serves as a technical verification link for search engines: it explicitly points out that the organization on the website is the exact same entity as the company profile on LinkedIn, Wikidata, or Crunchbase. This reinforces the entity’s weight in the Knowledge Graph.
Question: How has the role of backlinks and brand mentions (Digital PR) changed?
Answer: Links still pass classic PageRank (as confirmed by Google leaks). However, within the EEAT framework, implied links (unlinked brand name mentions on authoritative resources, like Forbes, without an active hyperlink) have gained massive importance. Google has learned to associate such mentions with the brand entity and use them as a trust signal.
Section 6: Strategic Conclusion
Question: Do you need to create separate SEO strategies for each AI search engine in 2026?
Answer: No, the most effective strategy is a single, multi-layered system. The foundational 80% of requirements (technical hygiene, speed, SSR, EEAT, deep and structured content) are equally important for all platforms. Individual engines require only fine-tuning: Bing Webmaster Tools for ChatGPT, focusing on data freshness for Perplexity, and Knowledge Graph optimization for Google.
Maryan Polyak
Consultation
We respond within 15 minutes



