AcademyWorkflowsMarketing

Build llms.txt files from Screaming Frog crawls

Marketing and SEO teams that want to give AI tools a curated index of their website.

Use it nowWant it running at your company?
  • Recommendedour rating
  • 2,516views on n8n

Use it now

Open inClaudeChatGPTGeminiPerplexity
Open on n8n.io ↗

Paste it straight onto an n8n canvas with Ctrl+V or Cmd+V.

Third party software. By copying, downloading or installing it you accept the Academy legal notice: you are responsible for reviewing its code and permissions before use.

generate-ai-ready-llms-txt-files-from-screaming-frog-website-crawls.json

Loading the file...

What it does

Generates an llms.txt file from a Screaming Frog crawl of a website. A form asks for the site name, a short description and the internal_html CSV export. The workflow builds the file, which you download from n8n. An optional text classifier can filter which URLs are included, and it is off by default.

Use cases

  1. 01Create a first llms.txt from an existing crawl
  2. 02Limit the file to specific sections of a site
  3. 03Add a node that uploads the file to a cloud drive

Connects

OpenAI Chat Model · Text Classifier

Questions about
Build llms.txt files from Screaming Frog crawls.

What is Build llms.txt files from Screaming Frog crawls used for?

Marketing and SEO teams that want to give AI tools a curated index of their website. Generates an llms.txt file from a Screaming Frog crawl of a website. A form asks for the site name, a short description and the internal_html CSV export. The workflow builds the file, which you download from n8n. An optional text classifier can filter which URLs are included, and it is off by default.

How do I import Build llms.txt files from Screaming Frog crawls into n8n?

Copy the workflow JSON from this page and paste it onto the n8n canvas with Ctrl+V or Cmd+V. Then connect your credentials in each node.

Is Build llms.txt files from Screaming Frog crawls safe to use?

It passed our automatic scan for credential theft, hidden instructions and risky install commands. Third party open source software. Sabemos AI does not maintain it. Check the code and permissions before you connect it to company data.

Related resources

Read the guide: AI marketing content systems ↗

  1. 001Brand GuidelinesMarketing and design teams who need materials styled consistently with Anthropic's brand.OfficialSkill
  2. 002HumanizerMarketing and communications teams use it to polish drafts that sound too much like AI before publishing.RecommendedSkill · ★ 55k
  3. 003Marketing Skills for AI AgentsMarketing teams and founders use it to give their AI agent a repeatable method for CRO, copy and SEO tasks.RecommendedSkill · ★ 54k
  4. 004AI videos with Seedance, posted to TikTok, YouTube, InstagramContent and social media teams that want to automate short video production and publishing across several platforms.RecommendedWorkflow · 215k views
  5. 005Multi-platform social media content with AISocial media managers and marketing teams who want drafts and publishing for several channels in one reviewed workflow.RecommendedWorkflow · 205k views
  6. 006AI video generation and multi-platform publishingContent creators, agencies and marketing teams who produce short-form video for several social channels.RecommendedWorkflow · 191k views