Backend Engineer

This role requires genuine, production-grade experience with web scraping — not hobbyist scripts, but robust systems that run daily against real-world sites with anti-bot protections, JavaScript rendering, session management and frequent structural changes.

DAY-TO-DAY RESPONSIBILITIES

Scraper Development (Primary — 60% of time)

• Implement new scrapers for assigned competitor websites using the team's standard Scrapy-based framework, with Playwright or Selenium for JavaScript-heavy targets.
• Reverse-engineer target websites by analysing browser network traffic in Chrome DevTools: identify XHR/fetch API calls, understand request headers, cookies and session flows, detect fingerprinting patterns.
• Parse structured and semi-structured HTML using BeautifulSoup4 and lxml; write robust XPath and CSS selectors that are resilient to minor DOM changes.
• Extract and normalise product data: price (including multi-currency), promotional mechanics, assortment (SKU name, brand, category), availability and URL.
• Handle pagination: offset-based, cursor-based, infinite scroll, AJAX-loaded content and URL-pattern iteration.
• Implement session management for scrapers requiring login flows, including form submission, CSRF token handling, and cookie persistence.
• Identify and implement evasion strategies: randomised delays, user-agent rotation, request header spoofing, referer injection.

Platform Integration & Quality (Secondary — 30% of time)

• Write unit tests for all parser functions using pytest; maintain >80% test coverage on new code.
• Document each scraper: target URL patterns, data fields extracted, known failure modes, approximate run time and output volume.
• Integrate scrapers into the orchestration layer (Airflow DAGs) according to team standards; configure scheduling, retries and alerting.
• Monitor your scrapers in production: review daily success/failure dashboards, triage failures within agreed SLA, fix breakages promptly.
• Participate in code reviews: review peers' scrapers for correctness, resilience and adherence to team coding standards.

Maintenance & Incident Response (10% of time)

• Investigate and resolve scraper failures caused by site structure changes, increased anti-bot aggression, IP blocks or infrastructure issues.
• Classify failure root causes and propose framework-level improvements to the Lead Scraping Engineer where patterns emerge.