AI Web Directory is a website discovery platform and a custom content management system I developed to make finding and submitting useful websites simpler. It combines a conventional, human-moderated web directory with AI-generated listing metadata, domain-based navigation, automated screenshots, Ahrefs Domain Rating, and its own traffic analytics.
The main idea was to remove the repetitive work traditionally associated with web directories. A visitor should not have to write a title, description, SEO metadata, and tags just to suggest a website. At the same time, automated submissions should not mean publishing everything without review. AI Web Directory brings these two requirements together: AI prepares the information, while moderation determines what becomes public.
Website submission with only a domain name
The most distinctive feature is the submission process. There is no need to create a user account, register a profile, or fill out a long listing form. The visitor opens Submit website, enters a domain such as example.com, and clicks Analyze with AI.
The CMS normalizes the domain, checks whether it is already in the catalog, applies domain and content restrictions, and retrieves information from the website. It examines the page title, meta description, keywords, headings, declared language, and visible homepage text. The AI processing stage then creates an English-language directory entry: an informative title, a factual description, a relevant category, topic tags, an SEO title, and a meta description.
The visitor sees a preview comparing information extracted from the website with the AI-generated results. The generated fields are intentionally locked at this stage to keep the submission workflow consistent. One final click sends the listing to the moderation queue. Only the domain is required as input; the editorial metadata is prepared automatically. The submitted website is not published until it is approved.
This two-stage approach also works with websites originally written in other languages. The system detects the source website's language, but produces the public listing text in English. A language recorded for a website is therefore not the same thing as a translated version of the directory interface.
A structured catalog instead of a collection of random links
AI Web Directory uses a deliberately simple front end. The homepage introduces the project, displays the available categories, presents newly added websites, and provides links to popular tags. The design uses responsive cards, generous spacing, compact text previews, and a lightweight navigation structure.
At the time of the screenshots, the catalog was organized into 20 categories, including AI Tools, Business, Development, E-commerce, Education, Design, Finance, Cybersecurity, Marketing, Media, Productivity, Real Estate, SEO, Software, Technology, and Travel. Every listing has its own page with a description, domain, category, relevant tags, a favicon, a screenshot, a direct link, and suggestions for similar websites.
There are several ways to explore the collection. Categories are suitable for broad topics; tags connect websites by specific subjects; TLD pages group websites by domain extension; and language pages group them by the detected language of the original resource. Category, tag, TLD, and language views use paginated, SEO-friendly URLs. Public tag landing pages are created for tags shared by at least five approved websites, avoiding thousands of nearly empty index pages.
Human moderation and automated quality checks
Automation does not replace editorial control. Newly submitted websites enter a pending state and are reviewed in the administration area. I can approve, reject, edit, or delete an entry, adjust its category and metadata, change its website language, and manage its screenshot. Duplicate domains are detected before another listing is accepted.
The system checks domains against configurable restrictions and examines website content for disallowed topics. It also includes request rate limiting, temporary blocking after certain analysis failures, and protection against fetching private or reserved network addresses. These checks help reduce low-quality or abusive submissions and make automatic website analysis safer.
On approval, the CMS can initiate screenshot generation, request an Ahrefs Domain Rating, and submit the new listing URL through IndexNow. Those jobs are handled separately where appropriate, so a slow external service does not need to block the normal moderation workflow.
Bulk importing and AI-assisted administration
The public submission form is designed for one domain at a time, but maintaining a directory with thousands of websites requires a different workflow. I therefore built a separate bulk AI import system for administrators.
It accepts domain lists, normalizes addresses, identifies duplicates and restricted domains, fetches useful homepage signals, and creates individual preparation jobs. Up to 50 domains can be prepared in one operation. After reviewing the prepared jobs, the administrator can send selected entries to the OpenAI Batch API, synchronize the results, inspect generated metadata, and import successful entries. Failed jobs can be retried individually rather than restarting the entire batch.
The administrative tools also cover category management, AI-generated descriptions for categories and tags, article creation and editing, metadata updates, and search across existing directory entries. The editor can additionally distinguish standard listings from promoted entries and configure how links appear in the directory and in similar-website recommendations. Together, these features make AI Web Directory more than a public listing form: it is a purpose-built publishing and catalog-management system.
Automated screenshots, favicons, and Ahrefs Domain Rating
A dedicated screenshot service creates website previews using a Chromium-based browser workflow and converts captures to WebP. Screenshots can be generated when websites are approved and refreshed from the administration panel. The implementation also accounts for pages that are slow to load or protected by browser challenges, with fallback handling where a usable capture cannot be obtained.
Favicons provide quick visual identification throughout the directory. On individual listing pages, the larger screenshot gives visitors a preview before they decide whether to open the external website.
For domains where data is available, the directory also displays Ahrefs Domain Rating (DR) alongside the website information. The administration area can retrieve a DR when a listing is approved and update previously stored ratings in batches of up to 1,000 domains. The date of the rating check is retained so the displayed figure has an identifiable update history. This is an additional reference metric, not a rating calculated by the directory itself.
Visitor likes, rankings, and related websites
Every website has a like mechanism, allowing visitors to support resources they find useful. The system limits repeated voting, including a 24-hour restriction for the same voter and website. A dedicated Top websites section ranks listings by visitor likes.
Related-website suggestions offer another way to navigate the catalog. Rather than forcing visitors to return to the homepage after each listing, the directory recommends other resources associated with similar topics and categories. It makes discovery possible through several small, connected paths instead of a single search-results page.
Public statistics and an unusually diverse catalog
I also created a public statistics page so that the catalog's composition can be explored without access to the administration panel. It includes total approved websites, entries awaiting moderation, category and article counts, tag usage, and activity in the like system. Two complete interactive tag clouds show the distribution of domain extensions and website languages, with larger labels representing more common groups.
The screenshots taken on 25 October 2026 captured 4,605 websites, 20 categories, 14,972 unique tags, 43,682 tag assignments, and 1,484 tags occurring on at least five websites. The catalog also represented 203 domain zones and 102 website languages. These are historical snapshot figures, not hard-coded limits: the live statistics change as the directory develops.
The language distribution is especially interesting. Although English-language resources make up the largest part of the catalog, it also includes German, Ukrainian, Russian, French, Georgian, Turkish, Lithuanian, and many other languages. Similarly, its TLD data goes far beyond the largest generic extensions: it includes country-code, newer generic, and less common domain zones.
Built-in traffic analytics without relying solely on third-party dashboards
In addition to public catalog statistics, I developed a separate administrative analytics system. It records public page loads and groups them by page title and URL. The page-statistics report shows total loads alongside mobile, desktop, and bot traffic, including counts and percentages. Results can be sorted across the complete dataset and displayed with pagination.
The analytics tools go deeper than per-page counters. Separate administrative views examine referring websites, search engines and identifiable search queries, crawler activity, unrecognized bot user agents, country-level traffic, and source classifications. This is particularly useful for understanding how a web directory is discovered by people, search engines, AI crawlers, and automated systems.
I deliberately kept this analytical layer inside the CMS. It gives me direct access to the dimensions that matter for this project, rather than requiring a generic report to be adapted to the directory's structure.
Search-engine-friendly architecture
Directory websites can become difficult for search engines to navigate when thousands of pages are generated without a clear structure. AI Web Directory addresses this with human-readable routes, individual SEO metadata, canonical URLs, breadcrumb navigation, Open Graph and Twitter Card tags, and JSON-LD structured data.
The CMS provides a sitemap index with separate, paginated XML sitemaps for directory entries, articles, tags, domain zones, and website languages. Website sitemaps can also include associated screenshot images. It also exposes robots.txt and llms.txt. IndexNow integration can notify supported search engines when new approved listing pages are published. Tag-page visibility thresholds and consistent URL patterns help keep a large collection organized as it grows.
Technology stack and custom architecture
I built AI Web Directory as a custom PHP application, rather than implementing it on WordPress or adapting an off-the-shelf directory plugin. The backend uses PHP 8.5+ and MySQL, with database operations handled through PDO. Its MVC-style structure includes a front controller, a lightweight custom router, controllers, models, reusable services, and PHP-rendered templates.
The interface uses HTML, CSS, and JavaScript without depending on a large client-side application framework. A responsive card-based layout keeps the frontend easy to navigate on desktop and mobile devices. Composer is used for dependencies, including HTML Purifier for sanitizing HTML content.
Several independent services support the main application:
- AI integration: configurable adapters for OpenAI, Anthropic, and Gemini, with a separate OpenAI Batch API workflow for large imports.
- Website fetching: HTTP fetching, HTML parsing, text extraction, language detection, and browser-based fallback handling.
- Media generation: Chromium-driven screenshots, WebP conversion, and favicon retrieval.
- SEO and external data: Ahrefs DR integration, IndexNow notifications, XML sitemaps, canonical metadata, and structured data.
- Application security: administrative authentication, CSRF tokens, output escaping, rate limiting, duplicate checks, restricted-domain checks, and protections against server-side requests to private addresses.
- Internal analytics: page-level counters, device and bot classification, referrer analysis, search-query reporting, and country statistics.
This modular structure allows the public directory, administration panel, AI processing, screenshot generation, and traffic reporting to evolve independently without making the visitor-facing experience more complicated.
What this project demonstrates
AI Web Directory is an example of where AI makes a practical difference in a traditional web application. It does not need to dominate the interface or replace every editorial decision. Instead, it takes over repetitive tasks such as summarizing websites, generating tags, and preparing metadata, while leaving final publication in human hands.
For me, the project also serves as a working environment for testing directory architecture, large-scale content management, automated domain processing, search-engine discovery, and self-hosted analytics. The result is a growing, usable catalog supported by a CMS designed specifically around its workflow.
Live project: https://aiwebdir.com
Comments (0)