# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file # # To ban all spiders from the entire site uncomment the next two lines: # User-agent: * # Disallow: / Sitemap: https://www.mongodb.com/sitemap.xml User-agent: mauibot Disallow: /community/forums/ User-agent: semrushbot Disallow: /community/forums/ User-agent: ahrefsbot Disallow: /community/forums/ User-agent: blexbot Disallow: /community/forums/ User-agent: seo spider Disallow: /community/forums/ User-agent: bingbot Crawl-delay: 60 Disallow: /community/forums/auth/ Disallow: /community/forums/assets/browser-update*.js Disallow: /community/forums/email/ Disallow: /community/forums/session Disallow: /community/forums/session/ Disallow: /community/forums/admin Disallow: /community/forums/admin/ Disallow: /community/forums/user-api-key Disallow: /community/forums/user-api-key/ Disallow: /community/forums/*?api_key* Disallow: /community/forums/*?*api_key* User-agent: * Crawl-delay: 10 # Directories Disallow: /misc/ Disallow: /profiles/ Disallow: /u/ Disallow: /community/forums/auth/ Disallow: /community/forums/email/ Disallow: /community/forums/session Disallow: /community/forums/session/ Disallow: /community/forums/admin Disallow: /community/forums/admin/ Disallow: /community/forums/user-api-key Disallow: /community/forums/user-api-key/ Disallow: /community/forums/g/ Disallow: /community/forums/u/ Disallow: /community/forums/user_avatar/ # Files Disallow: /admin/* Disallow: /data/* Disallow: /comment/reply/ Disallow: /filter/tips/ Disallow: /node/add/ Disallow: /search/ Disallow: /user/register/ Disallow: /user/password/ Disallow: /user/login/ Disallow: /user/logout/ Disallow: /lp/p/* Disallow: /lp/s/* Disallow: /lp/d/* Disallow: /thank-you/* Disallow: /link/* Disallow: /community/forums/assets/browser-update*.js Disallow: /docs/*.epub Disallow: /docs/*.pdf # No access for quicktabs in the URL Disallow: /*?quicktabs_* Disallow: /*&quicktabs_* Disallow: /*?qt-view_* Disallow: /*&qt-view_* # No crawling field collections or taxonomy Disallow: /field-collection/* Disallow: /tags/* Disallow: /taxonomy/term/* # Paths (no clean URLs) Disallow: /?q=admin/ Disallow: /?q=comment/reply/ Disallow: /?q=filter/tips/ Disallow: /?q=node/add/ Disallow: /?q=search/ Disallow: /?q=user/password/ Disallow: /?q=user/register/ Disallow: /?q=user/login/ Disallow: /?q=user/logout/ # Specific pages Disallow: /commercialsupport/ Disallow: /partners/referral-federal Disallow: /customer-evaluation-downloads-development-versions Disallow: /community/forums/*?api_key* Disallow: /community/forums/*?*api_key* Disallow: /docs/meta/ Disallow: /docs/search/ Disallow: /docs/openapi/ # Prevent exposing future docs Disallow: /docs-qa* # Prevent exposing devhub UAT Disallow: /devcenter-uat*