Medior Data Engineer – Extraction & Ingestion (Python)

Turn raw market noise into Europe’s most trusted curated food‑industry market intelligence insights

A-INSIGHTS turns raw market signals into clear, actionable insights for decision-makers at names like McCain, FrieslandCampina and Ahold Delhaize. Our data pipelines are the first mile of that journey. As our next Medior Data Engineer, you’ll own a portfolio of connectors and ingestion services, designing resilient scrapers and API connectors that keep fresh, high-quality data flowing 24/7, while mentoring juniors and aligning engineering work with business goals.

What you will be doing

You are co-responsible for the front door of our central data platform

• Lead design & delivery of Python ingestion jobs (scrapers, API connectors) with robust error handling, idempotency and documented backfill strategies.

• Define the first steps of ELT: validation, normalization and schema mapping and agree on data contracts with downstream consumers.

• Own observability & reliability: define Service Level Indicators and Objectives, add logs/metrics/alerts, drive and track error budgets.

• Set standards & mentor: create connector templates/runbooks, lead code reviews, pair with juniors, and coach on clean code and debugging.

• Design for ethics & resilience: handle pagination, throttling and anti-bot challenges respectfully; follow legal guidelines; secure secrets/PII via Azure Key Vault and least privilege access.

• Embed AI enrichment (Azure OpenAI & Cognitive Services) to classify, summarize and deduplicate content; set acceptance criteria for model outputs.

Projects you’ll tackle in the first year

• Lead migration processes of legacy scrapers to Azure Functions with retry logic, exponential back-off and structured logs, plan phases, risks and rollbacks.

• Connector standardization: ship reusable templates, runbooks and SLIs/SLOs for ingestion, and guide the team to adopt them.

• Document-to-insight pipeline: evolve extraction/vectorization/summarization at scale; improve throughput and cost per document.

• AI-powered news crawler: raise precision/recall and define evaluation metrics for summarization/deduplication.

Ongoing work you will contribute to

• Build & maintain Python ingestion jobs and first-step ELT pipelines with clear ownership boundaries.
• Deploy and observe services through Azure DevOps and Docker; proactively optimize cost/performance in Azure.
• Keep our data platform reliable, scalable and secure; lead post-incident improvements.
• Define and follow data-engineering best practices for design, testing and documentation; guide their adoption.
• Build and maintain cloud infrastructure in Azure to support scalable and secure data ingestion processes.

Tech stack

Daily tools

• Python (including Pandas, Selenium, and internal SDKs)
• API integration (REST, GraphQL)
• Azure DevOps (Git version control, CI/CD pipelines)
• Docker & Azure Container Registry
• Azure Function Apps & Logic Apps

You’ll also touch

• Azure AI Services & AI Foundry
• SQL (SQL Server Management Studio / Azure Data Studio)
• Azure Key Vault
• Azure Storage Services (Blob, Table, Queue)
• Azure Monitor / Application Insights

Haven’t used every tool yet? If you’ve mastered similar tech and love learning, we’ll back you.

Your Talent

• 4–6 years of professional software engineering experience, with strong Python skills.
• Proven end-to-end ownership of ingestion projects (from design to production) or major refactors in production.
• Led design reviews, wrote technical docs/runbooks.
• Git power user branching, rebasing, PR reviews and CI checks are second nature.
• Strong on clean code, debugging, architectural trade-offs and performance tuning in Azure.
• Nice to have: web scraping at scale, Selenium, SQL optimization, event-driven/streaming (e.g., Kafka/Event Hubs), Pandas/NumPy, experience in mentoring junior team members

Who you'll be working with

Jan – Senior Data Engineer

Sander – Senior Analyst

Martijn – Head of Product & Tech

What we offer:

• €55,000–€65,000 annually, including 8% holiday allowance.
• Performance bonus up to 10% of gross salary.
• Hybrid work: 2–5 days/week in our Amsterdam HQ.
• Personal growth plan and budget for both hard & soft skills.
• A supportive team of experienced engineers (pair‑programming, architecture sparring).
• Regular company events, dinners, BBQs, boat trips, padel tournaments, because fun fuels focus.

About A-INSIGHTS

We enable the world’s biggest food companies like McCain, Royal FrieslandCampina, Ahold Delhaize, Puratos, Savencia and more to save time, deepen insights and seamlessly share knowledge. With a strong product, a clear mission and rapid growth ahead, there’s never been a better time to join us.

Our Amsterdam team is a vibrant mix of 15+ nationalities who blend curiosity with craft (and occasionally turn each other into works of art).

‍We’re serious about inclusion

Research shows some groups only apply when they meet every requirement. If you’re excited by this role but your experience doesn’t line up perfectly, please apply anyway—we’d love to hear your story.

Have Questions? For any inquiries or additional information, please contact Nofar Biran.

• Email: nofar.biran@a-insighst.eu
• WhatsApp: +31 (0)611532011