MagnetmeBot

MagnetmeBot is the official web crawler used by Magnet.me to discover and index career-related content. This allows us to keep our platform up to date with the latest vacancies, events, and employer information.

Bot Overview

Robots.txt

MagnetmeBot respects the robots.txt protocol and supports the following directives:

Note: Setting a shorter Crawl-delay value will not guarantee a site is crawled more often, just faster.

Capabilities of MagnetmeBot

MagnetmeBot is designed to crawl websites efficiently and respectfully. Here’s what it can and cannot do:

Optimizing for MagnetmeBot

These best practices will help MagnetmeBot, but also search engines such as Google, understand and rank your content better:

Contact

If you have questions or experience unexpected behavior from MagnetmeBot, please get in touch: info@magnet.me


Magnet.me API

Magnet.me also operates an automated user agent for calling specific API endpoints. This agent does not crawl websites and does not adhere to robots.txt, since it only accesses known URLs.

It identifies itself with the following user agent: Magnet.me API (+https://magnet.me/bot.html)