Backend Engineer

This role requires genuine, production-grade experience with web scraping — not hobbyist scripts, but robust systems that run daily against real-world sites with anti-bot protections, JavaScript rendering, session management and frequent structural changes.

 

DAY-TO-DAY RESPONSIBILITIES

Scraper Development (Primary — 60% of time)

•      Implement new scrapers for assigned competitor websites using the team's standard Scrapy-based framework, with Playwright or Selenium for JavaScript-heavy targets.

•      Reverse-engineer target websites by analysing browser network traffic in Chrome DevTools: identify XHR/fetch API calls, understand request headers, cookies and session flows, detect fingerprinting patterns.

•      Parse structured and semi-structured HTML using BeautifulSoup4 and lxml; write robust XPath and CSS selectors that are resilient to minor DOM changes.

•      Extract and normalise product data: price (including multi-currency), promotional mechanics, assortment (SKU name, brand, category), availability and URL.

•      Handle pagination: offset-based, cursor-based, infinite scroll, AJAX-loaded content and URL-pattern iteration.

•      Implement session management for scrapers requiring login flows, including form submission, CSRF token handling, and cookie persistence.

•      Identify and implement evasion strategies: randomised delays, user-agent rotation, request header spoofing, referer injection.

 

Platform Integration & Quality (Secondary — 30% of time)

•      Write unit tests for all parser functions using pytest; maintain >80% test coverage on new code.

•      Document each scraper: target URL patterns, data fields extracted, known failure modes, approximate run time and output volume.

•      Integrate scrapers into the orchestration layer (Airflow DAGs) according to team standards; configure scheduling, retries and alerting.

•      Monitor your scrapers in production: review daily success/failure dashboards, triage failures within agreed SLA, fix breakages promptly.

•      Participate in code reviews: review peers' scrapers for correctness, resilience and adherence to team coding standards.

 

Maintenance & Incident Response (10% of time)

•      Investigate and resolve scraper failures caused by site structure changes, increased anti-bot aggression, IP blocks or infrastructure issues.

•      Classify failure root causes and propose framework-level improvements to the Lead Scraping Engineer where patterns emerge.