This fragment will not appear in snippets.
``` For documents: the server header X‑Robots‑Tag can repeat directives like "noindex", "nosnippet", "noai" if needed. ## Files and bots: how to manage GPTBot, OAI‑SearchBot, PerplexityBot and YandexAdditionalBot (with robots.txt examples) The difference matters: - GPTBot — mainly for training models. Doesn’t directly affect appearance in answers. - OAI‑SearchBot — access for content used in ChatGPT products. - PerplexityBot — crawling for Perplexity answers. - YandexAdditionalBot — extra crawling for Yandex search and AI modes. robots.txt templates (adapt to your policy): Allow citation but limit training: ``` User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: YandexAdditionalBot Allow: / User-agent: GPTBot Disallow: / ``` Disallow all listed to a private area: ``` User-agent: OAI-SearchBot Disallow: /private/ User-agent: PerplexityBot Disallow: /private/ User-agent: YandexAdditionalBot Disallow: /private/ User-agent: GPTBot Disallow: /private/ ``` llms.txt (policy for LLMs). Place in the root. Example: ``` # Conditions for AI systems using content training: disallow search-citation: allow contact: legal@your-domain ``` Important: disallowing training is not the same as disallowing classic search indexing. Keep these policies separate. ## Schema.org and JSON‑LD: ready fragments (Article, FAQPage, HowTo, BreadcrumbList) Article — for expert materials: ```json { "@context": "https://schema.org", "@type": "Article", "headline": "Как выбрать ERP для производства", "datePublished": "2026-05-10", "dateModified": "2026-10-01", "author": {"@type": "Person", "name": "Имя Фамилия"}, "publisher": {"@type": "Organization", "name": "Компания"} } ``` FAQPage — for Q/A blocks: ```json { "@context": "https://schema.org", "@type": "FAQPage", "mainEntity": [ { "@type": "Question", "name": "Сколько длится внедрение?", "acceptedAnswer": {"@type": "Answer", "text": "Обычно 6–12 недель для пилота."} }, { "@type": "Question", "name": "Какие гарантии?", "acceptedAnswer": {"@type": "Answer", "text": "SLA и поэтапная приёмка работ."} } ] } ``` HowTo — for step‑by‑step instructions: ```json { "@context": "https://schema.org", "@type": "HowTo", "name": "Аудит страницы под AI‑выдачу", "step": [ {"@type": "HowToStep", "name": "Проверьте индексируемость", "text": "Статус 200, в карте сайта, Allow в robots."}, {"@type": "HowToStep", "name": "Добавьте FAQ", "text": "3–5 коротких ответов с разметкой."}, {"@type": "HowToStep", "name": "Опубликуйте вывод", "text": "TL;DR в начале на 400–700 знаков."} ] } ``` BreadcrumbList — for breadcrumbs and context: ```json { "@context": "https://schema.org", "@type": "BreadcrumbList", "itemListElement": [ {"@type": "ListItem", "position": 1, "name": "Блог"}, {"@type": "ListItem", "position": 2, "name": "AI‑оптимизация"}, {"@type": "ListItem", "position": 3, "name": "Гайд по GEO/AEO"} ] } ``` ## Content formats that neural networks cite (FAQ, guides, tables, conclusions) Proven elements that increase citation chances: - Short answer at the top: 400–700 characters, no fluff, with numbers and criteria. - FAQ of 3–6 self‑contained questions and answers. - HowTo with steps and time/effort estimates. - Criteria/comparison tables: help models extract facts. - E‑E‑A‑T: an expert author, date, experience, cases and verifiability.  ## Templates: page checklist and stepwise implementation (3/3) — table Checklist “page for AI answers” (mark done): | Этап | Действие | Готово? | |---|---|---| | 1. Доступ | Страница индексируема; Allow в robots; в карте сайта | [ ] | | 1. Доступ | 200 OK, Core Web Vitals в зелёной зоне | [ ] | | 2. Структура | Короткий ответ (TL;DR) на первом экране | [ ] | | 2. Структура | FAQ 3–6 пунктов + JSON‑LD FAQPage | [ ] | | 2. Структура | HowTo со шагами (если применимо) | [ ] | | 3. Доказательства | Автор/дата, кейс/цифры, ссылки на первоисточники | [ ] | | 3. Доказательства | Article/BreadcrumbList разметка | [ ] | | 3. Доказательства | Обновление не реже 1 раза в квартал | [ ] | Stepwise implementation 3/3: | Шаг | Что делаем | Ответственный | Срок | Статус | |---|---|---|---|---| | 1/3 Доступ | Robots, карта сайта, проверка статусов | SEO/Dev | Неделя 1 | [ ] | | 2/3 Структура | TL;DR, FAQ, HowTo, таблицы | Контент | Недели 2–3 | [ ] | | 3/3 Доказательства | Автор, кейсы, E‑E‑A‑T, JSON‑LD | Маркетинг | Неделя 4 | [ ] | ## How to collect and verify sources: requirements for E‑E‑A‑T - Experience: show practical experience — “how we did it”, before/after numbers, role names. - Expertise: author profile, relevant certifications and topic fit. - Authoritativeness: brand mentions, citations, participation in industry reviews, readable methodologies. - Trustworthiness: transparent policies, contacts, legal info, fresh update dates. E‑E‑A‑T checklist for the “Who we are” section: authors with titles, contacts, photos, link to editorial policy on the site, date of last content validation. ## Tools: IndexNow, Bing Webmaster, Search Console, LLM API providers - Google Search Console: add all domain variants, check coverage, upload sitemap, use performance reports for control clusters. - Bing Webmaster Tools: set up the site, use IndexNow — publish the key and send events on page create/update. - IndexNow: automate pings on publish/edit; for CMS — hook up webhooks. - Logs and monitoring: log visits from needed bots (OAI‑SearchBot, PerplexityBot, YandexAdditionalBot) and 4xx/5xx errors. - LLM API providers: use locally to generate drafts of FAQ/HowTo, then manually refine and structure into JSON. ## How to train/support models on your content — example process The goal is to make content easy to extract facts from. Recommended approach is “structure and prompt”, not mass fine‑tuning. Process: 1) Gather core intents: form question clusters and group them by page. 2) Content template: for each cluster — TL;DR, HowTo, FAQ, criteria table, mini‑case. 3) JSON‑LD and breadcrumbs: Article + FAQPage/HowTo + BreadcrumbList. 4) Bot policy: allow bots for citation, restrict training if needed. 5) Answer checks: weekly run control queries and log citations. Example prompts for content preparation (use on your materials): Prompt 1. TL;DR from an article - Task: squeeze 500–700 characters with criteria and numbers. - Response format: 2–3 sentences + bullet list of 3–5 items. Prompt 2. HowTo - Task: create a step‑by‑step instruction with time estimates. - Response format: JSON with keys step, name, time, text. Prompt 3. FAQ - Task: produce 3–6 self‑contained Q/A of varying length. - Constraints: avoid vague words, include at least one answer with numbers/ranges. Expected JSON format for HowTo (example): ```json { "howto": [ {"step": 1, "name": "Проверка индексации", "time": "5 мин", "text": "Проверьте статус 200 и наличие в карте сайта."}, {"step": 2, "name": "Добавить FAQ", "time": "10 мин", "text": "3–5 чётких Q/A с разметкой."} ] } ``` ## How to measure success: metrics and citations dashboard Key metrics: - AI‑Citations: number of citations by platform (Google AI Overviews/AI Mode, ChatGPT, Perplexity, Yandex). - AI‑Visibility: share of queries where you appeared in the AI answer from a selected set. - AI‑Traffic: sessions/clicks from AI sources (by UTM and referral channels). - Lead/Revenue impact: leads, conversions and deals attributed to AI sources. Examples of UTM for special landings: - utm_source=ai_overviews&utm_medium=referral&utm_campaign=geo - utm_source=chatgpt&utm_medium=answer&utm_campaign=howto - utm_source=perplexity&utm_medium=citation&utm_campaign=faq Dashboard template (weekly): | Дата | Платформа | Запрос | Позиция/покрытие | Цитирование | Слаг целевой | Сессии | Конверсии | |---|---|---|---|---|---|---|---| | 2026‑10‑01 | Google AI | «выбор ERP для SMB» | 1/AI‑ответ | Да | /blog/erp‑smb‑guide | 86 | 3 | | 2026‑10‑01 | ChatGPT | «сравнение CDN для B2B» | AI Mode | Да | /blog/cdn‑compare | 41 | 1 | | 2026‑10‑01 | Perplexity | «что такое GEO для сайта» | Ответ | Нет | /blog/geo‑intro | 0 | 0 | | 2026‑10‑01 | Яндекс | «аудит CRM интеграции» | Нейро | Да | /blog/crm‑audit | 21 | 1 | ## Frequently asked questions (FAQ) A site’s own FAQ block with Schema is above; here are short answers useful for GEO/AEO. - What should I do first so pages can be used in AI answers? Indexability, a clear short answer, FAQ/HowTo, up‑to‑date Schema.org and verifiability. Then control bots and monitor citations. - Do I need to allow GPTBot and OAI‑SearchBot in robots.txt for ChatGPT to use my content? Allow OAI‑SearchBot — it’s important for citation. GPTBot is about training; decide according to content policy. - Which content formats are cited by neural networks more often (FAQ, HowTo, tables)? FAQ, HowTo, criteria/comparison tables and a short conclusion at the top of the page. - [How do I measure whether a neural network cites my pages](/en/blog/promoting-in-ai-answers-how-to-influence-and-measure/) (control queries, UTM tags)? Maintain a list of control queries and a weekly dashboard: platform, query, citation fact, page slug, sessions and conversions. Use UTMs for special landing pages. - How do I prevent my content from being used to train models if I want to forbid that? Restrict GPTBot in robots.txt, use X‑Robots‑Tag and publish llms.txt with your policy. ## Practical part: ready files/snippets/robots.txt/JSON‑LD Mini copy‑paste kit (adapt to yourself): Robots (access + AI policy): ``` User-agent: * Allow: / Disallow: /admin/ Sitemap: /sitemap.xml User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: YandexAdditionalBot Allow: / User-agent: GPTBot Disallow: / ``` Headers for documents (example): ``` X-Robots-Tag: max-snippet:300 ``` llms.txt (in the root): ``` training: disallow search-citation: allow contact: legal@your-domain ``` JSON‑LD blocks (minimal): Article: ```json { "@context": "https://schema.org", "@type": "Article", "headline": "Название статьи", "author": {"@type": "Person", "name": "Автор"}, "datePublished": "2026-10-01", "dateModified": "2026-10-01" } ``` FAQPage: ```json { "@context": "https://schema.org", "@type": "FAQPage", "mainEntity": [ {"@type": "Question", "name": "Вопрос 1", "acceptedAnswer": {"@type": "Answer", "text": "Ответ."}} ] } ``` HowTo: ```json { "@context": "https://schema.org", "@type": "HowTo", "name": "Короткая инструкция", "step": [ {"@type": "HowToStep", "name": "Шаг 1", "text": "Действие."} ] } ``` BreadcrumbList: ```json { "@context": "https://schema.org", "@type": "BreadcrumbList", "itemListElement": [ {"@type": "ListItem", "position": 1, "name": "Главная"}, {"@type": "ListItem", "position": 2, "name": "Раздел"} ] } ``` Mini script to check a page (pseudocode): ``` page = fetch(page_url) assert status(page) == 200 assert in_sitemap(page) assert robots_allows(page) assert has_tldr(page, min_chars=400) assert has_faq(page, min_items=3) assert has_schema(page, types=["Article"]) ``` ## FAQ ### What should I do first so pages can be used in AI answers? Check accessibility and indexing: open robots.txt for the needed bots, correct canonicals, up‑to‑date sitemaps and Schema.org markup (Article/FAQPage/HowTo). Then make sure the page has a short explicit answer, date/author and links to primary sources. In Google and Bing panels set site properties and add control queries to tracking. ### Do I need to allow GPTBot and OAI‑SearchBot in robots.txt for ChatGPT to use my content? For appearing in ChatGPT Search and similar modes, access for OAI‑SearchBot is what matters. GPTBot is mainly for model training. Allow OAI‑SearchBot and optionally restrict GPTBot. Do not block regular search bots separately, otherwise you’ll lose basic indexing. ### Which content formats are cited by neural networks more often? Most often — FAQ with clear Q/A, HowTo with steps, short conclusions at the top, and tables with criteria and comparisons. Structure, verifiability and semantic precision matter. Strengthen E‑E‑A‑T: author, experience, date, sources. ### How can I measure whether a neural network cites my pages? Collect a list of control queries and weekly check Google AI Overviews/AI Mode, ChatGPT, Perplexity, Yandex (Neuro/Alice). Log citations, source domains and slugs of target pages. Add UTM tags for special landing pages and campaign labels in analytics. ### How do I prevent my content from being used to train models if I want to forbid that? In robots.txt disallow training bots (for example, GPTBot) and use X‑Robots‑Tag headers with directives for AI. Also publish an llms.txt with terms. Note: disallowing training is not the same as prohibiting classic search indexing.