Description
About the Role:
We are looking for an experienced Python Developer with expertise in web scraping, automated portal interactions, and cloud-native deployment using AWS. The ideal candidate will have hands-on experience working with Playwright for browser automation, managing multi-factor authentication (MFA) flows, and deploying scalable scraping tasks via AWS Lambda and related services.
You will assist in architecting and building robust, secure, and scalable scraping solutions that interact with complex web applications and secured portals.
Responsibilities:
- Assist with the design and implementation of advanced scraping solutions using Python, Playwright, and AWS services.
- Automate interactions with JavaScript-heavy and authentication-secured websites, including handling MFA, CAPTCHAs, and session/token-based login flows.
- Architect scraping pipelines using serverless AWS components such as Lambda, Step Functions, S3, CloudWatch, and Secrets Manager.
- Build systems that scale to support high volumes of data extraction with fault tolerance, retries, and intelligent logging.
- Integrate and manage complex workflows across multiple portals, APIs, and data sources.
- Contribute to architectural decisions, tooling, and best practices.
Qualifications:
- 5+ years of experience in Python development, with a strong focus on automation and data extraction.
- Proven expertise in web scraping using tools like Playwright, Selenium, Scrapy, BeautifulSoup, and requests.
- In-depth experience handling multi-step authentication flows, including multi-factor authentication (MFA), CAPTCHA solving, and session/cookie management.
- Proficient in deploying and managing scraping workloads in AWS, particularly Lambda, S3, IAM, CloudWatch, and Secrets Manager.
- Experience with asynchronous programming, headless browsers, and JavaScript-rendered content.
- Solid understanding of web protocols (HTTP, HTTPS, cookies, headers), and ability to reverse-engineer network calls and authentication mechanisms.
- Comfortable working with APIs, JSON/XML, and data transformation.
Preferred Qualifications:
- Experience with CI/CD pipelines, Docker, and infrastructure as code (e.g., CloudFormation, Terraform).
- Familiarity with data pipeline orchestration tools such as Apache Airflow or AWS Step Functions.
- Prior experience scraping from secure or enterprise-level portals (e.g., financial, healthcare, legal).
- Background in data engineering or ETL workflows is a plus.
- Exposure to Python testing frameworks and writing unit/integration tests.
Education:
- Bachelor’s or Master’s degree in Computer Science, Software Engineering, or related field, or equivalent professional experience.
We believe the best ideas and innovation happen when we are together. Employees in this role will work in the office four days, with the flexibility to work remotely one day (Friday).
Benefits
iCapital offers a comprehensive benefits package that includes a total compensation program consisting of competitive salary, annual performance bonus, and equity for all full-time employees; healthcare with 100% employer-paid health and dental insurance; and generous paid time off (PTO).
For additional information on iCapital Network, please visit https://www.icapitalnetwork.com/about-us Twitter: @icapitalnetwork | LinkedIn: https://www.linkedin.com/company/icapital-network-inc
Similar jobs
Est. 55,000 EUR
About the Role: We are looking for an experienced Python Developer with expertise in web scraping, automated portal interactions, and cloud-native deployment using AWS. The ideal candidate will have hands-on experience w…
iCapital is looking to hire an experienced Python Developer with expertise in web scraping, automated portal interactions, and cloud-native deployment using AWS. The ideal candidate will have hands-on experience working…
Est. 165,000 USD
About the Role iCapital is looking to hire an experienced Python Developer with expertise in web scraping, automated portal interactions, and cloud-native deployment using AWS. The ideal candidate will have hands-on expe…
About the Role We are seeking an enthusiastic and motivated Python Developer to join our growing team. The ideal candidate will have foundational experience in Python programming and a passion for building scalable, high…
Est. 97,500 USD
About the Role iCapital is seeking an Associate Python Developer to join the growing Core Operations Technology team. This individual will have foundational experience in Python programming and a passion for building sca…
Est. 120,000 USD
About the Role iCapital is looking for a Software Developer in Test to join the Quality Assurance team. The ideal candidate brings practical knowledge of agile development methodologies and engineering best practices. As…
Est. 125,000 USD
About the Role iCapital is seeking a motivated and detail-oriented Associate Business Process Analyst, UiPath Developer to join our Intelligent Automation team. The ideal candidate will combine solid business analysis sk…
Est. 125,000 USD
About the Role iCapital is seeking a motivated and detail-oriented Associate Business Process Analyst, UiPath Developer to join our Intelligent Automation team. The ideal candidate will combine solid business analysis sk…
Est. 120,000 USD
About the Role iCapital is looking to hire a Software Developer in Test to work on the iCapital platform. This individual will bring practical knowledge of agile development methodologies and engineering best practices.…
Est. 80,000 EUR
About the Role The Full Stack Engineer at iCapital will work end-to-end on new features, enhancements, and defect resolution on one or more of iCapital's software platforms. You will work closely with our Product Managem…
Est. 115,000 USD
About the Role iCapital is looking to hire a Full Stack Engineer Assistant Vice President to join the Post-Investment Technology team. This role will work end-to-end on new features, enhancements, and defect resolution o…
Est. 160,000 USD
About the Role iCapital is looking to a hire a Vice President to join the Platform Infrastructure team. This team plays a critical role in keeping the production and development environments running smoothly and securely…
Est. 150,000 USD
About the Role The Backend Engineer at iCapital will play an integral role in designing and implementing systems that power the iCapital platform. We are looking for engineers who can build APIs and applications that are…
Est. 115,000 USD
About the Role The Full Stack Engineer at iCapital will work end-to-end on new features, enhancements, and defect resolution on one or more of iCapital's software platforms. You will work closely with our Product Managem…
Est. 117,500 USD
About the Role iCapital is looking to hire a Backend Engineer Associate or Assistant Vice President to join the Software Engineering team. This individual will play an integral role in designing and implementing systems…
Job Title: Python Engineer About Us “Capco, a Wipro company, is a global technology and management consulting firm. Awarded with Consultancy of the year in the British Bank Award and has been ranked Top 100 Best Companie…
Est. 100,000 USD
About the Role iCapital is looking for a Software Developer in Test to work on the iCapital platform. In this role, you’ll bring practical knowledge of agile development methodologies and engineering best practices. As a…
Est. 140,000 USD
About the Role The Full Stack Engineer at iCapital will work end-to-end on new features, enhancements, and defect resolution on one or more of iCapital's software platforms. This individual will work closely with our Pro…
Est. 65,000 USD
About the Role iCapital is looking to hire an Analyst to join its Platform Support team. The team provides service and technical support to its rapidly growing client base of financial advisors and individual investors,…
Est. 205,000 USD
About the Role The Platform Infrastructure team at iCapital plays a critical role in ensuring that both production and development environments operate smoothly, securely, and reliably. This role leverages advanced cloud…
Est. 140,000 USD
About the Role iCapital is seeking a strategic, hands-on, and results-driven Vice President Program Manager to join the Product Operations team, directly supporting the Pre-Investment vertical. This role will lead the pl…
Est. 110,000 USD
About the Role iCapital is seeking a Backend Software Engineer at the Associate or Assistant Vice President level. You will design, build, and operate services that power the iCapital platform. You will collaborate with…
Est. 150,000 USD
About the Role iCapital is seeking a Backend Software Engineer at Vice President level. You will design, build, and operate services that power the iCapital platform. You will collaborate with engineers, product managers…
Est. 145,000 USD
About the Role The Backend Engineer at iCapital will play an integral role in designing and implementing systems that power the iCapital platform. We are looking for engineers who can build APIs and applications that are…
Est. 80,000 EUR
About the role: We are looking for a technically strong Senior Application Security practitioner ready to take on real ownership. You will work directly with the Head of AppSec, contributing to the Secure Design practice…
Est. 60,000 EUR
About the Role The Software Engineer at iCapital Network will work end-to-end on new features, enhancements, and defect resolution on one or more of iCapital's software platforms. You will work closely with our Product M…
Est. 60,000 EUR
About the Role The Software Engineer at iCapital Network will work end-to-end on new features, enhancements, and defect resolution on one or more of iCapital's software platforms. You will work closely with our Product M…
Est. 145,000 USD
About the Role The Platform Infrastructure team at iCapital is responsible for designing, standardizing, and evolving the core infrastructure that powers our global technology stack. As Vice President, Platform Infrastru…
Job Title: Python Developer About Us “Capco, a Wipro company, is a global technology and management consulting firm. Awarded with Consultancy of the year in the British Bank Award and has been ranked Top 100 Best Compani…
Est. 150,000 USD
About the Role The Backend Engineer at iCapital will play an integral role in designing and implementing systems that power the iCapital platform. We are looking for engineers who can build APIs and applications that are…