MagnetmeBot

MagnetmeBot is the official web crawler used by Magnet.me to discover and index career-related content. This allows us to keep our platform up to date with the latest vacancies, events, and employer information.

Bot Overview

• Name: MagnetmeBot
• Purpose: To discover and update career-related pages for display on Magnet.me
• Bot Type: Respectful, transparent crawler
• User Agent: Magnet.me-web/XX.XX (where XX.XX is a version number)
• IP addresses: https://magnet.me/bot-ips.json or https://magnet.me/bot-ips.txt

Robots.txt

MagnetmeBot respects the robots.txt protocol and supports the following directives:

• User-agent: MagnetmeBot
• Allow
• Disallow
• Crawl-delay

Note: Setting a shorter Crawl-delay value will not guarantee a site is crawled more often, just faster.

Capabilities of MagnetmeBot

MagnetmeBot is designed to crawl websites efficiently and respectfully. Here’s what it can and cannot do:

• It may load and view Javascript content, but this isn’t always guaranteed.
• It focuses on the main version of a page (the "canonical" version).
• It doesn’t fill out forms or click buttons — it only scrolls and follows links.
• It ignores ads and does not trigger analytics or tracking tools.
• It adjusts its crawl speed automatically to avoid overloading your website.
• If a page exists in multiple languages, it will choose a single preferred language to post (using "hreflangs") .
• Additional configuration options are available — please contact your account manager to enable them.

Optimizing for MagnetmeBot

These best practices will help MagnetmeBot, but also search engines such as Google, understand and rank your content better:

• Provide a robots.txt file: A proper robots.txt file helps MagnetmeBot find your sitemaps and ensures it crawls exactly at the speed you want it to.
• Provide a sitemap: Ensure your XML sitemap is accessible and includes all relevant pages.
• Use descriptive URLs: URLs like /jobs/software-engineer help the crawler recognize relevant content.
• Use Structured Data: Add JobPosting markup for vacancies and BusinessEvent markup for recruitment-related events.
• Specify the hiring organization: When your website has job openings for more than one organization, use hiringOrganization in the Structured Data to define which jobs may be posted, and for which organization.

Contact

If you have questions or experience unexpected behavior from MagnetmeBot, please get in touch: info@magnet.me

---

Magnet.me API

Magnet.me also operates an automated user agent for calling specific API endpoints. This agent does not crawl websites and does not adhere to robots.txt, since it only accesses known URLs.

It identifies itself with the following user agent: Magnet.me API (+https://magnet.me/bot.html)