MagnetmeBot is the official web crawler used by Magnet.me to discover and index career-related content. This allows us to keep our platform up to date with the latest vacancies, events, and employer information.
Magnet.me-web/XX.XX
(where XX.XX
is a version number)MagnetmeBot respects the robots.txt
protocol and supports the following directives:
User-agent: MagnetmeBot
Allow
Disallow
Crawl-delay
Note: Setting a shorter Crawl-delay
value will not guarantee a site is crawled more often, just faster.
MagnetmeBot is designed to crawl websites efficiently and respectfully. Here’s what it can and cannot do:
These best practices will help MagnetmeBot, but also search engines such as Google, understand and rank your content better:
/jobs/software-engineer
help the cralwer recognize relevant content.
If you have questions or experience unexpected behavior from MagnetmeBot, please get in touch: info@magnet.me
Magnet.me also operates an automated user agent for calling specific API endpoints. This agent does not crawl websites and does
not adhere to robots.txt
, since it only accesses known URLs.
It identifies itself with the following user agent:
Magnet.me API (+https://magnet.me/bot.html)