* **User-agent:** `*` applies rules to all web crawlers. You can specify specific bots (e.g., `Googlebot`).
* **Disallow:** Prevents crawlers from accessing specific files or directories. `Disallow: /` blocks the entire site.
* **Allow:** Creates exceptions to `Disallow` rules, allowing crawlers to access specific files or subdirectories within a disallowed directory.
* **Sitemap:** Informs crawlers about the location of your XML sitemap.
* **Syntax Checker:** This is a basic checker for common `robots.txt` syntax. It does not guarantee full compliance with all crawler interpretations.
* **Disclaimer:** Always test your `robots.txt` file using Google Search Console's `robots.txt` Tester before deploying it to your live site. Incorrect `robots.txt` can block search engines from crawling your entire website.