Description
About the Role:
We are looking for an experienced Python Developer with expertise in web scraping, automated portal interactions, and cloud-native deployment using AWS. The ideal candidate will have hands-on experience working with Playwright for browser automation, managing multi-factor authentication (MFA) flows, and deploying scalable scraping tasks via AWS Lambda and related services.
You will assist in architecting and building robust, secure, and scalable scraping solutions that interact with complex web applications and secured portals.
Responsibilities:
- Assist with the design and implementation of advanced scraping solutions using Python, Playwright, and AWS services.
- Automate interactions with JavaScript-heavy and authentication-secured websites, including handling MFA, CAPTCHAs, and session/token-based login flows.
- Architect scraping pipelines using serverless AWS components such as Lambda, Step Functions, S3, CloudWatch, and Secrets Manager.
- Build systems that scale to support high volumes of data extraction with fault tolerance, retries, and intelligent logging.
- Integrate and manage complex workflows across multiple portals, APIs, and data sources.
- Contribute to architectural decisions, tooling, and best practices.
Qualifications:
- 5+ years of experience in Python development, with a strong focus on automation and data extraction.
- Proven expertise in web scraping using tools like Playwright, Selenium, Scrapy, BeautifulSoup, and requests.
- In-depth experience handling multi-step authentication flows, including multi-factor authentication (MFA), CAPTCHA solving, and session/cookie management.
- Proficient in deploying and managing scraping workloads in AWS, particularly Lambda, S3, IAM, CloudWatch, and Secrets Manager.
- Experience with asynchronous programming, headless browsers, and JavaScript-rendered content.
- Solid understanding of web protocols (HTTP, HTTPS, cookies, headers), and ability to reverse-engineer network calls and authentication mechanisms.
- Comfortable working with APIs, JSON/XML, and data transformation.
Preferred Qualifications:
- Experience with CI/CD pipelines, Docker, and infrastructure as code (e.g., CloudFormation, Terraform).
- Familiarity with data pipeline orchestration tools such as Apache Airflow or AWS Step Functions.
- Prior experience scraping from secure or enterprise-level portals (e.g., financial, healthcare, legal).
- Background in data engineering or ETL workflows is a plus.
- Exposure to Python testing frameworks and writing unit/integration tests.
Education:
- Bachelor’s or Master’s degree in Computer Science, Software Engineering, or related field, or equivalent professional experience.
We believe the best ideas and innovation happen when we are together. Employees in this role will work in the office four days, with the flexibility to work remotely one day (Friday).
Benefits
iCapital offers a comprehensive benefits package that includes a total compensation program consisting of competitive salary, annual performance bonus, and equity for all full-time employees; healthcare with 100% employer-paid health and dental insurance; and generous paid time off (PTO).
For additional information on iCapital Network, please visit https://www.icapitalnetwork.com/about-us Twitter: @icapitalnetwork | LinkedIn: https://www.linkedin.com/company/icapital-network-inc
Similar jobs
About the Role: We are looking for an experienced Python Developer with expertise in web scraping, automated portal interactions, and cloud-native deployment using AWS. The ideal candidate will have hands-on experience w…
iCapital is looking to hire an experienced Python Developer with expertise in web scraping, automated portal interactions, and cloud-native deployment using AWS. The ideal candidate will have hands-on experience working…
Est. 165,000 USD
About the Role iCapital is looking to hire an experienced Python Developer with expertise in web scraping, automated portal interactions, and cloud-native deployment using AWS. The ideal candidate will have hands-on expe…
About the Role We are seeking an enthusiastic and motivated Python Developer to join our growing team. The ideal candidate will have foundational experience in Python programming and a passion for building scalable, high…
Est. 97,500 USD
About the Role iCapital is seeking an Associate Python Developer to join the growing Core Operations Technology team. This individual will have foundational experience in Python programming and a passion for building sca…
Est. 120,000 USD
About the Role iCapital is looking for a Software Developer in Test to join the Quality Assurance team. The ideal candidate brings practical knowledge of agile development methodologies and engineering best practices. As…
Est. 80,000 EUR
About the Role The Full Stack Engineer at iCapital will work end-to-end on new features, enhancements, and defect resolution on one or more of iCapital's software platforms. You will work closely with our Product Managem…
Est. 125,000 USD
About the Role iCapital is seeking a motivated and detail-oriented Associate Business Process Analyst, UiPath Developer to join our Intelligent Automation team. The ideal candidate will combine solid business analysis sk…
Est. 80,000 EUR
About the role: We are looking for a technically strong Senior Application Security practitioner ready to take on real ownership. You will work directly with the Head of AppSec, contributing to the Secure Design practice…
Est. 60,000 EUR
About the Role The Software Engineer at iCapital Network will work end-to-end on new features, enhancements, and defect resolution on one or more of iCapital's software platforms. You will work closely with our Product M…
Est. 60,000 EUR
About the Role The Software Engineer at iCapital Network will work end-to-end on new features, enhancements, and defect resolution on one or more of iCapital's software platforms. You will work closely with our Product M…
Est. 125,000 USD
About the Role iCapital is seeking a motivated and detail-oriented Associate Business Process Analyst, UiPath Developer to join our Intelligent Automation team. The ideal candidate will combine solid business analysis sk…
Est. 80,000 EUR
About the Role iCapital is looking for an Software Engineer. You will be involved in the development of key parts of the platform while scaling the backend team to meet the needs of a new international audience. You will…
Est. 80,000 EUR
About the Role iCapital is looking for a Full-Stack Engineer to join the iCapital Identity Platform team. This role will help define and deliver new features for one or more of iCapital's software platforms. You will wor…
Est. 160,000 USD
About the Role iCapital is looking to a hire a Vice President to join the Platform Infrastructure team. This team plays a critical role in keeping the production and development environments running smoothly and securely…
Est. 120,000 USD
About the Role iCapital is looking to hire a Software Developer in Test to work on the iCapital platform. This individual will bring practical knowledge of agile development methodologies and engineering best practices.…
Est. 115,000 USD
About the Role iCapital is looking to hire a Full Stack Engineer Assistant Vice President to join the Post-Investment Technology team. This role will work end-to-end on new features, enhancements, and defect resolution o…
Est. 150,000 USD
About the Role The Backend Engineer at iCapital will play an integral role in designing and implementing systems that power the iCapital platform. We are looking for engineers who can build APIs and applications that are…
Est. 55,000 EUR
About the Role We are seeking a mid‑level Scrum Master to support Agile delivery across technology teams focused on data, automation, and AI‑enabled solutions. This role partners closely with Product Owners, Engineering…
Est. 205,000 USD
About the Role The Platform Infrastructure team at iCapital plays a critical role in ensuring that both production and development environments operate smoothly, securely, and reliably. This role leverages advanced cloud…
Est. 115,000 USD
About the Role The Full Stack Engineer at iCapital will work end-to-end on new features, enhancements, and defect resolution on one or more of iCapital's software platforms. You will work closely with our Product Managem…
Est. 55,000 EUR
About the Role iCapital is looking for a talented Data Analyst– Platform Solutions to join the International Platform Solutions team in Lisbon. The role encompasses supporting multiple projects focused on technical imple…
Est. 60,000 EUR
About the Role As a Software Engineer on the Fund Accounting team, you'll have the opportunity to shape the platforms and systems that define how our operations run at scale. This is a full-stack, end-to-end engineering…
Est. 140,000 USD
About the Role The Full Stack Engineer at iCapital will work end-to-end on new features, enhancements, and defect resolution on one or more of iCapital's software platforms. This individual will work closely with our Pro…
Est. 117,500 USD
About the Role iCapital is looking to hire a Backend Engineer Associate or Assistant Vice President to join the Software Engineering team. This individual will play an integral role in designing and implementing systems…
Est. 60,000 EUR
About the Role iCapital is seeking a Technology Program Manager to partner closely with Engineering / Technology leadership to deliver complex, high-impact initiatives across the firm’s most critical technology prioritie…
Est. 65,000 USD
About the Role iCapital is looking to hire an Analyst to join its Platform Support team. The team provides service and technical support to its rapidly growing client base of financial advisors and individual investors,…
Est. 140,000 USD
About the Role iCapital is seeking a strategic, hands-on, and results-driven Vice President Program Manager to join the Product Operations team, directly supporting the Pre-Investment vertical. This role will lead the pl…
Est. 80,000 EUR
About the Role iCapital is looking for a Assistant Vice President, Cyber Cloud Security Engineer to join the Security Engineering team, within the Information Security organization. This role will establish and manage cl…
Job Title: Python Engineer About Us “Capco, a Wipro company, is a global technology and management consulting firm. Awarded with Consultancy of the year in the British Bank Award and has been ranked Top 100 Best Companie…