Silkpath FAQ

Silkpath is a free website crawler for SEO, AEO and GEO audits. Enter a website address and it visits every page, then reports technical problems page by page and lets you download the HTML of the whole site as a ZIP.

Silkpath was created, designed and built by its founder, Raj Meena. Raj Meena's portfolio is at raj-meena.netlify.app, and the LinkedIn profile is linkedin.com/in/rajmeena.

Status codes and redirects, indexability, canonicals, XML sitemaps, page titles, meta descriptions, headings, content and duplicates, internal and external links with broken links, images and alt text, structured data, Open Graph, hreflang, security headers, mobile viewport, and AEO and GEO signals. See all features.

AEO (answer engine optimization) makes pages easy for answer boxes and voice assistants to quote. GEO (generative engine optimization) makes sure AI tools such as ChatGPT, Perplexity, Gemini and AI Overviews can reach, read and cite a site. Read What is AEO and What is GEO.

Yes. The app is free to use. Crawling runs on a Silkpath crawler server, which can be hosted on a free plan.

The crawler server can be protected with a password, so only the people its owner trusts can start crawls. If you need access, contact Raj Meena on LinkedIn.

Up to 50,000 URLs per crawl by default, and you can set a lower limit per domain.

In the shared cloud storage of the crawler, so every device shows the same History. Without cloud storage, crawls stay in your browser. Deleted crawls wait 30 days in the Recycle Bin. Page files for the ZIP stay on the crawler server for 3 hours.

No. Silkpath reads each page's raw HTML, the way most crawlers and AI tools first see it. A site that builds its pages in the browser looks nearly empty, and Silkpath warns about it so you can check it with a rendering crawler.

Yes. "Respect robots.txt" is on by default for every crawl.

A website crawler is a program that follows the links of a site the way a search engine does, visits every page, and records what it finds: status codes, titles, headings, links, images and markup. Silkpath is a website crawler built for SEO, AEO and GEO audits.

llms.txt is a plain text file at the root of a website that lists the pages an AI tool should read first, with a short summary of the site. Silkpath checks whether a site has llms.txt and llms-full.txt, and whether robots.txt blocks AI crawlers such as GPTBot, ClaudeBot and PerplexityBot. See the llms.txt checker.

Open Silkpath, enter the website address, and press Start Crawl. When the crawl ends, open the report: the Overview shows a site health score, and each section (Crawl, On-page, Links and Media, Markup, AI Search) lists the pages that fail a check. Export any section as PDF, Excel or CSV. The User Guide has every step.

Silkpath runs in the browser and is free, with no install, and it includes AEO and GEO checks such as llms.txt, AI crawler access and FAQ schema. Screaming Frog is a mature desktop crawler with more settings and a paid licence for large crawls. Many people use both. Read the comparison.

PDF, Excel and CSV for any section, any tab or the whole report, share links for those files, and a ZIP of every page's HTML with optional CSS, JavaScript, images, fonts and media.

Yes. Schedule a single crawl for a set time from the arrow next to Start Crawl, or use Auto Re-crawl on a report to crawl the site again every few days (1 to 365), even while the app is closed.

Yes. In Chrome or Edge, use Install App in Settings or the install icon in the address bar. On Android use Install App or the browser menu. On iPhone and iPad, tap Share, then Add to Home Screen.

Yes. Turn on notifications in Settings, App and Notifications, and each crawl you start on that device sends a notification when it ends, even with Silkpath closed. On iPhone and iPad, add Silkpath to the Home Screen first (iOS 16.4 or later).

Usually the site's firewall refused the crawler (a 403 or a challenge page), robots.txt blocked it, or the site builds its links with JavaScript. Silkpath shows a message that names the reason, and the Crawl Log lists each refused page.