Web Crawling Engineer

Mistral AIParis, TX

Apply with Sonara

Automate your job search with Sonara.

Submit 10x as many applications with less effort than one manual application.¹

Reclaim your time by letting our AI handle the grunt work of job searching.

We continuously scan millions of openings to find your top matches.

Overview

Schedule

Full-time

Career level

Senior-level

Remote

Option for remote

Benefits

Health Insurance

Parental and Family Leave

Health & Wellness Programs

Job Description

About Mistral

At Mistral AI, we believe in the power of AI to simplify tasks, save time, and enhance learning and creativity. Our technology is designed to integrate seamlessly into daily working life.

We democratize AI through high-performance, optimized, open-source, and cutting-edge models, products, and solutions. Our comprehensive AI platform is designed to meet enterprise needs, whether on-premises or in cloud environments. Our offerings include le Chat, the AI assistant for life and work.

We are a dynamic, collaborative team passionate about AI and its potential to transform society.

Our diverse workforce thrives in competitive environments and is committed to driving innovation. Our teams are distributed between France, the USA, the UK, Germany, and Singapore. We are creative, low-ego, and team-spirited.

Join us to be part of a pioneering company shaping the future of AI. Together, we can make a meaningful impact. See more about our culture on https://mistral.ai/careers.

Role Summary

We are looking for a skilled and motivated Web Crawling Engineer to join our dynamic engineering team. The ideal candidate should have a solid background in distributed web crawling, scraping and data extraction, with experience using advanced tools and technologies to collect and process large-scale data from diverse web sources at large scale.

What you will do

As a Web crawling engineer, you will be responsible for:

Developing and maintaining web crawlers using Go to extract data from target websites.
Utilize headless browsing techniques, such as Chrome DevTools, to automate and optimize data collection processes.
Collaborate with cross-functional teams to identify, scrape, and integrate data from APIs and web pages to support business objectives.
Create and implement efficient parsing patterns using tokenizers, regular expressions, XPaths, and CSS selectors to ensure accurate data extraction.
Design and manage distributed job queues using technologies such as Redis, Aerospike and Kubernetes to handle large-scale distributed crawling and processing tasks.
Develop strategies to monitor and ensure data quality, accuracy, and integrity throughout the crawling and indexing process.
Continuously improve and optimize existing web crawling infrastructure to maximize efficiency and adapt to new challenges.

About you

Core programming and web technologies

Proficiency in Go (Golang)/Rust/Zig for building scalable and efficient web crawlers.
Deep understanding of TCP, UDP, TLS and HTTP/1.1,2,3 protocols and web communication.
Knowledge of HTML, CSS, and JavaScript for parsing and navigating web content.
Familiarity with cloud platforms (AWS, GCP), orchestration (Kubernetes, Nomad), and containerization (Docker) for deployment.

Data Structures & Algorithms

Mastery of queues, stacks, hash maps, and other data structures for efficient data handling.
Ability to design and optimize algorithms for large-scale web crawling.

Web Scraping & Data Acquisition

Hands-on experience with networking and web scraping libraries.
Understanding of how search engines work and best practices for web crawling optimization.

Databases & Data Storage

Experience with SQL and/or NoSQL databases (knowing Aerospike is a bonus) for storing and managing crawled data.
Familiarity with data warehousing and scalable storage solutions.

Distributed Systems & Big Data

Knowledge of distributed systems (e.g., Hadoop, Spark) for processing large datasets.

Bonus Skills (Nice-to-Have)

Experience with web archiving projects & tooling, open-source archiving is a big plus!
Experience applying Machine Learning to improve crawling efficiency or accuracy.
Experience with low-level networking programming and/or userspace TCP/IP stacks.

Hiring Process

Here is what you should expect:

Introduction call- 35 min
Hiring Manager Interview- 30 min
Live-coding Interview- 45 min
System Design Interview- 45 min
Deep dive interview (optional) - 60min
Culture-fit discussion- 30 min
Reference checks

By applying, you agree to our Applicant Privacy Policy.

Location & Remote

This role is primarily based in one of our European offices- Paris, France and London, UK. We will prioritize candidates who either reside there or are open to relocating. We strongly believe in the value of in-person collaboration to foster strong relationships and seamless communication within our team.

In certain specific situations, we will also consider remote candidates based in one of the countries listed in this job posting - currently France, UK, Germany, Belgium, Netherlands, Spain and Italy. In any case, we ask all new hires to visit our Paris HQ office:

for the first week of their onboarding (accommodation and travelling covered)
then at least 2 days per month

What we offer

Competitive salary and equity

️ Health insurance

Transportation allowance

Sport allowance

Meal vouchers

Private pension plan

Parental : Generous parental leave policy

Visa sponsorship

By applying, you agree to our Applicant Privacy Policy.

Automate your job search with Sonara.

Submit 10x as many applications with less effort than one manual application.

Apply with Sonara Apply manually

FAQs About Web Crawling Engineer Jobs at Mistral AI

What is the work location for this position at Mistral AI?

This job at Mistral AI is located in Paris, TX, according to the details provided by the employer. Some roles may also include multiple work locations depending on the requirement.

What pay range can candidates expect for this role at Mistral AI?

Employer has not shared pay details for this role.

What employment applies to this position at Mistral AI?

Mistral AI lists this role as a Full-time position.

What experience level is required for this role at Mistral AI?

Mistral AI is looking for a candidate with "Senior-level" experience level.

Does Mistral AI allow remote work for this role?

Yes, this position at Mistral AI supports remote work, giving candidates the flexibility to work outside the primary office location.

What benefits are offered by Mistral AI for this role?

Mistral AI offers following benefits: Health Insurance, Parental and Family Leave, and Health & Wellness Programs for this position. Actual benefits may vary depending on the employer's policies and employment terms.

What is the process to apply for this position at Mistral AI?

You can apply for this role at Mistral AI either through Sonara's automated application system, which helps you submit applications 10X faster with minimal effort, or by applying manually using the direct link on the job page.