Python Scraping Developer
Our client is a Berlin-based, remote-first scale-up providing cutting‑edge market intelligence and software solutions to the automotive industry. As the company enters an exciting new phase of growth, they are looking for an experienced Python Scraping Developer to strengthen their international, high‑impact team.
If you thrive on tackling complex data extraction challenges, building highly scalable web crawlers, and ensuring large-scale scraping systems run flawlessly in production, this role is for you. You will be responsible for the entire lifecycle of our high-volume scraping pipelines, guaranteeing the data we collect is accurate, consistent, and delivered with speed.
Responsibilities
Design & Development: Develop, test, and deploy robust web scraping scripts and crawlers using advanced Python tools (Playwright, Selenium, Requests, BeautifulSoup, etc.).
Scalability: Architect and maintain asynchronous scraping systems capable of massive, large-scale data extraction.
Resilience: Implement, monitor, and optimize sophisticated anti-blocking strategies and proxy rotation to ensure high reliability and uptime.
Integration: Manage and automate data ingestion pipelines and seamless integrations with external REST APIs.
Operational Excellence: Debug, monitor, and continuously improve scraper performance, reliability, and data quality.
Collaboration: Partner with other engineers to enhance our core scraping infrastructure, tooling, logging, and monitoring systems.
DevOps Support: Assist with DevOps tasks, including Docker, CI/CD, and managing Linux environments.
Requirements
Core Experience: Proven, hands-on professional experience in high-volume web scraping and data extraction using Python.
Technical Depth: Solid understanding of HTML parsing, browser automation techniques, and asynchronous programming.
Frameworks: Proficiency with leading web scraping frameworks (e.g., Playwright, Scrapy, or Selenium).
Web Knowledge: Strong knowledge of REST APIs, HTTP protocols, and effective proxy management.
Database Skills: Familiarity with both SQL and NoSQL databases for efficient data storage and processing.
Infrastructure: Experience with Docker, Linux environments, and version control (Git).
Communication: Fluent in English (written and spoken).
Mindset: Self-driven, detail-oriented, and capable of taking full ownership of significant projects.
Nice to Haves (Bonus Points)
Experience with advanced async libraries (e.g., asyncio)
Understanding of data quality validation and pipeline monitoring tools.
What they offer
Impact & Ownership: A high degree of freedom and the opportunity to have a meaningful, measurable impact on a growing scale-up business.
Flexibility: A high degree of flexibility – our client is a remote-first company and actively support remote work.
Growth: A competitive compensation package and dedicated support for your personal & professional development (ongoing training & coaching).
Team & Atmosphere: A great work atmosphere within a small, talented, and international team.
Office (Optional): A modern office located on the campus of Wildau Tech University, easily accessible by public transport (just outside Berlin).