# AI search crawlers — explicitly allowed for GEO visibility User-agent: GPTBot User-agent: ClaudeBot User-agent: anthropic-ai User-agent: PerplexityBot User-agent: Applebot-Extended User-agent: Googlebot-Extended Allow: / # Private user pages Disallow: /profile Disallow: /privacy_acceptance Disallow: /saved_jobs Disallow: /companies/profile Disallow: /companies/profile/edit Disallow: /companies/job_offers/ Disallow: /admin/ Disallow: /flipper # Auth and registration pages (no search value) Disallow: /users/sign_in Disallow: /users/sign_up Disallow: /users/oauth_signup Disallow: /users/password Disallow: /users/confirmation Disallow: /users/unlock Disallow: /companies/sign_in Disallow: /companies/registration Disallow: /onboarding # Turbo Frame partials — server-side fragments, no standalone value Disallow: /job_offers/*/recommendations # Analytics beacon fired by the page JS while Google renders it; bots are already dropped server-side Disallow: /amplitude/ # Block common security scanning paths Disallow: /wp-* Disallow: /wordpress* Disallow: /backup* # Blog: disallow temporarily until the public blog is shipped Disallow: /blog # /job_offers?page= is a Turbo Frame cursor. Other listings still paginate in HTML. Disallow: /job_offers?page= Disallow: /job_offers?*&page= # Sort and locale variants — no indexing value, protect crawl budget Disallow: /*?sort_by= Disallow: /*&sort_by= Disallow: /*?locale= Disallow: /*&locale= # All other crawlers User-agent: * Allow: / # Private user pages Disallow: /profile Disallow: /privacy_acceptance Disallow: /saved_jobs Disallow: /companies/profile Disallow: /companies/profile/edit Disallow: /companies/job_offers/ Disallow: /admin/ Disallow: /flipper # Auth and registration pages (no search value) Disallow: /users/sign_in Disallow: /users/sign_up Disallow: /users/oauth_signup Disallow: /users/password Disallow: /users/confirmation Disallow: /users/unlock Disallow: /companies/sign_in Disallow: /companies/registration Disallow: /onboarding # Turbo Frame partials — server-side fragments, no standalone value Disallow: /job_offers/*/recommendations # Analytics beacon fired by the page JS while Google renders it; bots are already dropped server-side Disallow: /amplitude/ # Block common security scanning paths Disallow: /wp-* Disallow: /wordpress* Disallow: /backup* # Blog: disallow temporarily until the public blog is shipped Disallow: /blog # /job_offers?page= is a Turbo Frame cursor. Other listings still paginate in HTML. Disallow: /job_offers?page= Disallow: /job_offers?*&page= # Sort and locale variants — no indexing value, protect crawl budget Disallow: /*?sort_by= Disallow: /*&sort_by= Disallow: /*?locale= Disallow: /*&locale= Sitemap: https://www.takealuk.com/sitemap.xml