SEO REVIEW LAB / ROBOTS.TXT GENERATOR

Robots.txt Generator

Create a clear wildcard-crawler robots.txt draft from explicit allow and disallow paths, with an optional same-origin sitemap declaration.

Runs in this browser tabNo account requiredOne site and sitemap URL up to 500 characters; path lists up to 5,000 characters; every path must begin with / and contain no spaces or #.

Step 1 / Build the brief

Draft the crawler rules

5 fields

Enter one path pattern per line. This first release creates rules only for User-agent: * and does not inspect an existing file or test a crawler.

01Website originUse the public HTTP or HTTPS site URL that will host /robots.txt.Required02Default crawler accessAllow writes an empty Disallow line; block writes Disallow: / before any explicit Allow paths.Required03Disallow pathsEnter one path or supported pattern per line, each beginning with /.Optional04Allow pathsOptionally enter more specific allowed paths, one per line.Optional05Sitemap URLOptionally enter a complete sitemap URL on the same origin.Optional

Before generationThe optional sitemap must use the same origin as the site. Duplicate path lines are removed while their first order is retained.

Checking database publication status before enabling this workspace.

No crawl or upload

The entered brief is processed in this browser tab. These tools do not crawl the supplied URL, fetch remote images, or send the draft to Nirmion.

Transparent transformation

The workspace uses visible rules, length counts, URL parsing, and standards-shaped templates rather than hidden ranking scores or unverified AI claims.

Human approval required

A technically valid draft can still be inaccurate for a page, campaign, platform, or policy. Test the result and obtain the appropriate approval before release.

Plain-language guide

How the directive file is assembled

Nirmion validates the site and sitemap as HTTP or HTTPS URLs, requires the sitemap origin to match, trims non-empty path lines, removes exact duplicates, and rejects malformed paths. It writes a wildcard user-agent block, an empty Disallow for default access or Disallow: / for default blocking, followed by the supplied Disallow, Allow, and Sitemap lines. It does not fetch or merge a live robots.txt file.

01
Start with verified inputs
Map the public URL paths that crawlers may request. Do not use robots.txt to protect confidential data, and verify whether wildcard syntax is supported by each crawler you care about.
02
Understand the transformation
The browser validates line shape and same-origin sitemap placement, then writes an explicit directive sequence for the wildcard crawler group.
03
Test the destination
Review with developers and SEO owners, compare against the current production file, test relevant crawler interpretations, and publish only at the site root as /robots.txt.

Useful when

Practical jobs this tool supports

Draft a small site policy before the first production release.
Review a proposed block for search, staging-like, or private-area paths.
Add a same-origin sitemap declaration to a controlled file.

Release check

Simulate the consequences first

One broad path can stop crawling across many pages or resources. Check case, slashes, query patterns, assets, and crawler-specific support.

Every blocked path is intentional and belongs to this host.
Required CSS, JavaScript, images, and public content remain crawlable.
The sitemap URL resolves publicly and uses the same origin.
The new file is compared with production and tested before replacement.

Important boundary

Robots.txt is not access control

Google explains that robots.txt manages crawler access and is not a mechanism for keeping a web page out of search. Rules are voluntary for crawlers, blocked URLs may still be indexed from links, and sensitive content requires authentication or another real security control.

Read Google Search Central robots.txt guidance

Questions before publishing

What a careful user should know

Crawler control and information security solve different problems; the generated file must not blur that boundary.

Can this hide a confidential page?

No. Use authentication or another access-control mechanism. Robots rules are public and cannot enforce crawler behavior.

Does the tool check my existing robots.txt?

No. It makes no network request. Compare and merge the draft with your current production policy manually.

Why require a same-origin sitemap?

This bounded first release avoids silently advertising a different site. Advanced cross-site arrangements should be reviewed and authored directly.