As the internet grows increasingly cluttered with AI-generated noise and algorithmic bias, MevaSearch is on a mission to map the web differently. Central to this mission is our proprietary crawler: WebAtlasBot.
If you’ve noticed "WebAtlasBot" in your server logs and are wondering what it is, why it’s there, and if it’s safe, you’re in the right place.
What is WebAtlasBot?
WebAtlasBot is the official web-crawling software for MevaSearch. Its primary job is to discover and index high-quality, technical, and educational content to power our global database, the WebAtlas Index.
Unlike traditional crawlers that prioritize ad-heavy or viral content, WebAtlasBot is programmed to identify semantic value. We are building a knowledge base that resists manipulation and focuses on the "infrastructure of information."
Identification
When our bot visits your site, it identifies itself with the following User-Agent string:
WebAtlasBot/1.8 (+https://mevasearch.com/bot)
Is WebAtlasBot Safe?
Yes. WebAtlasBot is designed to be a "good citizen" of the internet. We prioritize the stability and security of your hosting environment.
We use adaptive crawling technology. By monitoring your server’s response times, WebAtlasBot automatically slows down its request rate to ensure your human visitors never experience a lag.
We strictly adhere to the established protocols of the web:
- Robots.txt: We follow all Disallow and Allow directives.
- Crawl-delay: We respect the Crawl-delay parameter to prevent server strain.
- Noindex/Canonical: We honor HTML metadata, ensuring we don't index private content or duplicate pages.
How to Control WebAtlasBot
We believe webmasters should have total control over their data. If you want to customize how we interact with your site, you can use standard robots.txt rules:
To limit access to specific folders:
User-agent: WebAtlasBot
Disallow: /private-folder/
To slow down the crawling speed:
User-agent: WebAtlasBot
Crawl-delay: 10
Why Allow WebAtlasBot to Index Your Site?
By allowing WebAtlasBot to crawl your pages, you are ensuring your content is represented in a filter-free, high-utility search engine.
MevaSearch users are often researchers, developers, and educators looking for deep knowledge. If your site provides high-quality technical or educational value, being part of the WebAtlas Index connects you directly with a highly engaged, professional audience.
Contact Us
We strive for a collaborative relationship with the webmaster community. If you have questions, feedback, or specific crawling requirements, please reach out to our team:
Email: bot[at]mevasearch.com
