# CDP Institute — robots.txt
# Public content (News, Blog, Resource Library landing pages) is open to crawlers
# and answer-engine agents. Member-only, account, admin and API surfaces are
# disallowed. Gated downloads themselves stay behind login (see src/middleware.ts);
# the disallow rules below just keep crawlers out of those non-content paths.

User-agent: *
Allow: /
Disallow: /account/
Disallow: /downloads/
Disallow: /member-center/
Disallow: /welcome/
Disallow: /internal/
Disallow: /newsgen/
Allow: /_emdash/api/media/
Disallow: /_emdash/
Disallow: /api/
Disallow: /login/
Disallow: /register/

# Clean Markdown copies of content for AI agents / answer engines: append ".md" to
# any news, blog, resource, or vendor URL. Index of key + newest content: /llms.txt
Sitemap: https://www.cdpinstitute.org/sitemap.xml
Sitemap: https://www.cdpinstitute.org/sitemaps/google-news.xml