LLMS Optimizer (llms.txt)

LLMS Optimizer Plugin for Botble CMS

A powerful Botble CMS plugin that automatically generates an optimized llms.txt file for AI crawlers and Large Language Models (LLMs), following the official llms.txt specification.

🚀 Features

Core Features

• ✅ Automatic llms.txt Generation - Generates AI-friendly content index following official llms.txt standard
• ✅ Dynamic Content Detection - Automatically detects all slugable content types (Pages, Posts, Products, etc.)
• ✅ Smart Descriptions - Includes brief descriptions for each content item (excerpt, description, or content preview)
• ✅ Multiple Output Modes - Serve via dynamic route, static file, or both
• ✅ Intelligent Caching - Built-in caching system with configurable duration
• ✅ Auto Cache Invalidation - Automatically clears cache when content is created, updated, or deleted

Advanced Features

• 🎯 Fully Dynamic - No hardcoded content types - automatically supports any plugin with slugable models
• 🎯 Optional Section - Add less important links in a separate "Optional" section
• 🎯 Custom Site Description - Add detailed AI-friendly site description
• 🎯 Flexible Sorting - Sort content by newest first or alphabetically
• 🎯 Link Descriptions - Include brief descriptions with each link (following llms.txt spec)
• 🎯 Sitemap Integration - Automatic XML sitemap reference with markdown formatting
• 🎯 Token Estimation - Optional token count estimation for AI context planning

Developer Features

• 🔧 Extensible via Filters - Multiple filter hooks for customization
• 🔧 Clean Architecture - Follows Botble's best practices and patterns
• 🔧 No Dependencies - Works out of the box with core Botble features
• 🔧 Future-Proof - Automatically supports new plugins without code changes

📦 Installation

Via Botble Marketplace (Recommended)

1. Go to Admin Panel → Plugins → Add New
2. Search for "LLMS Optimizer"
3. Click Install
4. Click Activate
5. Navigate to Settings → LLMS Optimizer to configure

Manual Installation from GitHub

1. Download the latest release from GitHub Releases
2. Extract the ZIP file
3. Upload the llms-optimizer folder to platform/plugins/ directory
4. Go to Admin Panel → Plugins
5. Find "LLMS Optimizer" and click Activate
6. Navigate to Settings → LLMS Optimizer to configure

⚙️ Configuration

Navigate to Admin Panel → Settings → LLMS Optimizer to configure:

Content Types

• Enable/disable specific content types (Pages, Posts, Products, etc.)
• Automatically detects all available content types from installed plugins

Formatting Options

• Site Description: Custom AI-friendly description of your site
• Include Site Tagline: Show site tagline/description
• Include Item Descriptions: Add brief descriptions for each link
• Include Sitemap Reference: Link to XML sitemap
• Link Format: Markdown or plain URL format
• Sorting Order: Newest first or alphabetical
• Max Items Per Type: Limit items per content type (1-1000)

Optional Section

• Include Optional Section: Add "Optional" section for less important content
• Optional Links: Custom links in markdown format (one per line)

Performance & Caching

• Cache Duration: How long to cache generated content (in minutes)
• Output Mode:
  • Dynamic Route (default) - Serve via /llms.txt route
  • Static File - Generate public/llms.txt file
  • Both - Enable both methods

📖 Usage

Accessing llms.txt

Once configured, your llms.txt file is available at:

https://yoursite.com/llms.txt

Example Output

Generated by Shaqi LLM Optimizer v1.0

> The XML sitemap of this website can be found here: [sitemap.xml](https://yoursite.com/sitemap.xml)

# Your Site Name

> Your custom site description for AI systems

## Pages
- [About Us](https://yoursite.com/about): Learn more about our company and mission
- [Contact](https://yoursite.com/contact): Get in touch with our team

## Blog Posts
- [Latest News](https://yoursite.com/blog/latest-news): Read our latest updates and announcements

## Products
- [Product Name](https://yoursite.com/products/product-name): High-quality product description

## Optional
- [Changelog](https://yoursite.com/changelog)
- [Community Forum](https://yoursite.com/community)

---
Estimated tokens: ~450

Admin Actions

• Save Settings - Save configuration and clear cache
• Clear Cache - Manually clear cached content
• Regenerate - Force regenerate static file (if enabled)

🔌 Extensibility

Available Filters

// Modify final output
add_filter('llms_optimizer_output', function ($content) {
    return $content . "\n\nCustom footer text";
});

// Modify content types
add_filter('llms_optimizer_content_types', function ($types) {
    // Add custom content type
    $types['Custom Section'] = [
        ['title' => 'Custom Item', 'url' => 'https://...', 'description' => '...']
    ];
    return $types;
});

// Modify items for a specific type
add_filter('llms_optimizer_items', function ($items, $type) {
    // Customize items
    return $items;
}, 10, 2);

// Add content to Optional section
add_filter('llms_optimizer_optional_section', function ($lines) {
    $lines[] = '- [Custom Link](https://...)';
    return $lines;
});

🎯 Why llms.txt?

The llms.txt file is a standardized format (similar to robots.txt) that helps AI systems and Large Language Models understand your website's structure and content. It provides:

• Better AI Understanding - Helps AI systems index and understand your content
• Improved AI Responses - AI assistants can provide more accurate information about your site
• SEO for AI - Optimize your site for AI-powered search and discovery
• Future-Proof - Following an emerging standard adopted by major companies

Learn more at llmstxt.org

🤝 Support

• Issues: GitHub Issues
• Source Code: GitHub Repository

📝 License

This plugin is open-sourced software licensed under the MIT license.

👨‍💻 Author

Ishtiaq Ahmed

• GitHub: @rajaishtiaq6
• Email: [email protected]

🙏 Credits

• Built for Botble CMS
• Follows llms.txt specification
• Inspired by real-world implementations from Stripe, Cloudflare, and other leading companies

Made with ❤️ for the Botble CMS community