Python Web Scraper

We are looking for an experienced Python developer to build a web scraper that extracts publicly available driver and reviewer profile data from BlaBlaCar.

Project Goal:

Build a structured database of approximately 50,000 unique driver profiles, including the profiles of users who reviewed them.

Data Collection Process

Step A – Route Search Pages:

From search result pages such as:

These pages list trips for given routes and dates.

Step B – Trip Pages:

Each trip links to a page such as:

Access to these pages is unrestricted.

Step C – Driver Profile Pages:

Each trip page contains a link to the driver profile page, for example:

and

From each driver profile page, I need structured data extraction (fields detailed in attached sample output).

Step D – Reviewer Profiles

From the driver's review page, extract:

List of 10 last reviewers

Reviewer profile URLs

Then visit each reviewer profile and extract the same structured profile information.

This means the scraper must:

1/Collect driver profiles

2/Collect reviewer profiles. If that step is deemed too complex/data intensive it can be dispensed of.

Important Notes:

No login credentials required.

CAPTCHAs may appear after multiple requests.

The scraper should handle:

Rate limiting

Captcha mitigation strategy (please propose approach)

Deduplication logic

Error handling and retry logic

The objective is to build a large, comprehensive dataset (~50,000 unique users).

Deliverables

Structured dataset (CSV or JSON) of 50,000 unique profiles (drivers + reviewers).

Clean, well-documented Python scraper code.

Instructions to run the scraper.

Contract duration of less than 1 month.

Mandatory skills:

Python, Data Scraping, Automation, Data Extraction, captchas