SEO Tools
Optimize your website for search engines. Generate sitemaps, meta tags, and structured data.
Search engines judge what they can read. These SEO tools generate and validate robots.txt, XML sitemaps, schema markup, Open Graph previews, and canonical URLs - the plumbing that helps your pages get indexed correctly.
6 tools availableRobots.txt Generator
Control search engine crawler access to your site
Sitemap Generator
Create XML sitemaps for better indexing
Schema Markup Generator
Generate JSON-LD structured data for SEO
Open Graph Preview
Preview how links appear on social media
Canonical URL Builder
Build canonical URLs to prevent duplicate content
Structured Data Validator
Validate JSON-LD and preview rich snippets
The files search engines read, and the rule that decides which one wins
Indexing plumbing is three separate contracts: who may crawl, which URLs exist, and which of several near-identical URLs is the one. Get one wrong and the others cannot compensate - a sitemap full of URLs that robots.txt disallows is worse than no sitemap at all.
robots.txt is a group grammar, not a list of paths
RFC 9309 defines the unit as a group: "One or more user-agent lines that are followed by one or more rules. The group is terminated by a user-agent line or end of file." A rule therefore belongs to the group above it, and "The last group may have no rules, which means it implicitly allows everything." When two rules could apply, the tie is broken by length, not by order: "The most specific match found MUST be used. The most specific match is the match that has the most octets." Crawlers have to be prepared to parse a sizeable file, too - the parsing limit "MUST be at least 500 kibibytes" - so an oversize robots.txt is silently truncated by some clients rather than rejected.
Sitemaps and canonicals: what you assert, and what the engine decides
The sitemap protocol is precise about scale: "each Sitemap file that you provide must have no more than 50,000 URLs and must be no larger than 50MB (52,428,800 bytes)", with index files to group them. Its lastmod value "should be in W3C Datetime format" and is meant to be the date of last modification of the page - a value you can only produce honestly if your build derives it from real edits. Open Graph covers the other half of discovery: "The four required properties for every page are" og:title, og:type, og:image and og:url, where og:url is "The canonical URL of your object that will be used as its permanent ID in the graph".
And the canonical itself is a statement, not a command: RFC 6596 describes the link relation as one that "specifies the preferred IRI from resources with duplicative content". Two pages can each declare the other canonical; the engine still chooses.
| File or tag | What it controls | Rule that decides the outcome | Specification |
|---|---|---|---|
| robots.txt | which crawlers may fetch which paths | rules belong to the group above them; the most specific (longest) match wins | RFC 9309 |
| XML sitemap | URL discovery and lastmod hints | 50,000 URLs and 50MB per file; use an index to group files | sitemaps.org protocol |
| rel=canonical | which duplicate of a page is preferred | states a preference; it is not a directive | RFC 6596 |
| Open Graph tags | the title, image and URL used when the page is shared | four properties are required, and og:url is the permanent ID | ogp.me |
Write the rules with the robots.txt Generator, build the file set with the Sitemap Generator, check the markup with the Schema Markup Generator and Structured Data Validator, preview the shared card in OG Preview, and normalise the URL itself with the Canonical URL Builder.
Sources: RFC 9309, Robots Exclusion Protocol; sitemaps.org protocol; RFC 6596, the canonical link relation; The Open Graph protocol.