Description
Overview:
We are looking for a Senior Data Engineer with deep expertise in Lakehouse architecture, real-time data streaming, cloud data infrastructure, and microservices development on Azure Kubernetes Service (AKS). You will play a central role in designing and delivering next-generation data pipelines, BI solutions, AI/ML platforms, streaming APIs, and scalable microservices that power Guidepoint's research and analytics products.
This is a high-impact, hands-on engineering role. You will work closely with data architects, data scientists, analysts, frontend engineers, QA, and DevOps teams to translate complex business requirements into scalable, reliable, and observable data systems.
This is a Hybrid role from our Pune office.
What You'll Do:
Data Engineering & Lakehouse
- Design, build, and maintain ETL pipelines, data ingestion workflows, and table schemas on Azure Databricks to support BI, analytics, and AI/ML use cases
- Architect and optimize the Lakehouse using Delta Lake on Databricks, ensuring reliability, performance, and cost efficiency
- Build and support data pipelines from business applications such as Salesforce, NetSuite, and other enterprise systems
- Develop and maintain Knowledge Graph models, entity relationship structures, and NLP-based insight pipelines
- Maintain data governance, data privacy standards, and compliance best practices throughout the data lifecycle
- Perform root cause analysis on data and processes to identify opportunities for improvement
- Collaborate with data architects, scientists, and business consumers to populate and optimize the data warehouse for reporting and analytics
Microservices & AKS Development
- Develop and support scalable web APIs and microservices using Python and Azure Platform Services
- Build new applications, services, and platforms; optimize existing solutions and refactor legacy components using modern, scalable architectures
- Design, implement, and deploy microservices on Azure Kubernetes Service (AKS) using Docker, Kubernetes, Helm, and Azure DevOps YAML pipelines
- Perform end-to-end deployments including infrastructure setup, configuration, and monitoring on AKS
- Decompose portions of legacy applications into modern microservices architecture
- Design and manage JSON payloads and payload contexts for inter-service communication
- Engage in database schema design and management, including updating tables and rows for large datasets
- Collaborate with cross-functional teams — Full-Stack, QA, DevOps, and Product — in agile SDLC processes
Real-Time Streaming & SSE
- Design and implement robust SSE (Server-Sent Events) endpoints using Python frameworks (FastAPI, Flask, Django) for real-time event delivery to web and mobile clients
- Build and maintain asynchronous backend services using asyncio, aiohttp, or similar libraries for non-blocking, high-concurrency streaming
- Architect streaming data pipelines integrating SSE with upstream message brokers — Kafka, Redis Pub/Sub, RabbitMQ
- Optimize connection lifecycle management: reconnection logic, heartbeat signals, event ID tracking, and graceful shutdowns
- Collaborate with frontend teams to define and evolve SSE event schemas and API contracts
- Implement observability across streaming services: distributed tracing, structured logging, and metrics using Prometheus, Datadog, or OpenTelemetry
Engineering Excellence
- Write comprehensive unit, integration, and load tests for all data, streaming, and microservices components
- Write and maintain robust CI/CD pipelines using Azure DevOps YAML pipelines
- Participate in architecture reviews, code reviews, and on-call rotations
- Maintain thorough technical documentation and mentor junior engineers on best practices in data engineering, Lakehouse architecture, streaming systems, and microservices
What You Have:
Required
- Bachelor's degree in Computer Science, Engineering, or a related field from an accredited university
- 7+ years of professional data engineering and/or backend software engineering experience
- Advanced SQL expertise across relational and NoSQL databases (SQL Server, Neo4j, Elasticsearch, Cosmos DB)
- Strong hands-on experience building and optimizing data pipelines on Azure Databricks
- In-depth knowledge of Delta Lake, Data Warehousing, and Lakehouse architecture
- Highly proficient in Spark, Python, and SQL
- Proven experience designing and deploying microservices on AKS using Docker, Kubernetes, and Helm
- Hands-on experience with Azure DevOps YAML pipelines for CI/CD automation
- Experience with SSE or real-time streaming — event stream formatting, retry logic, connection management
- Strong grasp of async Python: asyncio, async/await, event loops
- Experience with message brokers: Kafka, Redis Streams, RabbitMQ, or similar
- Proven track record of processing and extracting value from large, complex, and disconnected datasets
- Excellent stakeholder management and communication skills across global, cross-functional teams
- Proven leadership skills with a strategic mindset and passion for driving innovation
Nice to Have
- Experience with Fivetran for data integration
- Familiarity with BI tools such as Power BI
- Experience building and deploying ML and feature engineering pipelines using MLflow
- Knowledge of Knowledge Graph development (e.g., Neo4j) and NLP-based analytics
- Familiarity with cloud-based AI/ML services and Generative AI tools
- Experience working in a compliance-based environment (building and deploying compliant software throughout the SDLC)
- Familiarity with API gateway configuration for streaming (NGINX, Kong, Azure API Gateway)
What We Offer:
- Competitive compensation
- Employee medical coverage
- Central office location
- Entrepreneurial environment, autonomy, and fast decisions
- Casual work environment
About Guidepoint:
Guidepoint powers end-to-end research workflows for the world’s best research teams.
Backed by a global network of more than 2 million experts and over 1,600 employees, Guidepoint delivers real-time access to expertise, primary research, and actionable knowledge that help organizations make informed decisions. Through consultations, surveys, events, proprietary content, and AI-enabled tools, we support every stage of the decision-making process—from developing hypotheses and gathering insights to validating assumptions and building conviction for critical decisions.
At Guidepoint, our success relies on the diversity of our employees, experts, and clients, which enables us to foster meaningful connections and a broad range of perspectives. We are committed to creating an inclusive and welcoming environment where individuals of all backgrounds, identities, and experiences can contribute and succeed.
Similar jobs
Overview: We are building a modern internal platform that enables application, data, and AI teams to build, deploy, and scale software efficiently and securely. We are looking for an experienced Principal Platform Engine…
Join Our Global Recruitment Team as a Talent Acquisition Associate Our Recruitment Team & Our Values Guidepoint’s Recruitment team is an integral part of a company that prides itself on identifying the most qualified…
Est. 55,000 GBP
Overview: Guidepoint's Market Research Team is seeking a Senior Market Research Analyst based in London to help oversee quantitative client relationships, design and manage research projects, write and develop questionna…
Est. 35,000 EUR
Overview: The Client Service team connects Guidepoint’s clients with subject-matter experts to better inform their business decisions. They take the time to understand each client’s specific research needs and deliver th…
Est. 35,000 GBP
Overview: The Client Service team connects Guidepoint’s clients with subject-matter experts to better inform their business decisions. They take the time to understand each client’s specific research needs and deliver th…
Est. 35,000 EUR
Overview: The Client Service team connects Guidepoint’s clients with subject-matter experts to better inform their business decisions. They take the time to understand each client’s specific research needs and deliver th…
Est. 35,000 EUR
Overview: The Client Service team connects Guidepoint’s clients with subject-matter experts to better inform their business decisions. They take the time to understand each client’s specific research needs and deliver th…
Overview: The Client Service team connects Guidepoint’s clients with subject-matter experts to better inform their business decisions. They take the time to understand each client’s specific research needs and deliver th…
Est. 35,000 EUR
Overview: The Client Service team connects Guidepoint’s clients with subject-matter experts to better inform their business decisions. They take the time to understand each client’s specific research needs and deliver th…
Est. 35,000 EUR
Overview: The Client Service team connects Guidepoint’s clients with subject-matter experts to better inform their business decisions. They take the time to understand each client’s specific research needs and deliver th…
Est. 55,000 GBP
Overview: The Client Service team connects Guidepoint’s clients with subject-matter experts to better inform their business decisions. They take the time to understand each client’s specific research needs and deliver th…
Est. 35,000 EUR
Overview: The Client Service team connects Guidepoint’s clients with subject-matter experts to better inform their business decisions. They take the time to understand each client’s specific research needs and deliver th…
Est. 35,000 EUR
Overview: The Client Service team connects Guidepoint’s clients with subject-matter experts to better inform their business decisions. They take the time to understand each client’s specific research needs and deliver th…
Est. 35,000 EUR
Overview: The Client Service team connects Guidepoint’s clients with subject-matter experts to better inform their business decisions. They take the time to understand each client’s specific research needs and deliver th…
Est. 35,000 EUR
Overview: The Client Service team connects Guidepoint’s clients with subject-matter experts to better inform their business decisions. They take the time to understand each client’s specific research needs and deliver th…
Est. 35,000 EUR
Overview: The Client Service team connects Guidepoint’s clients with subject-matter experts to better inform their business decisions. They take the time to understand each client’s specific research needs and deliver th…
Est. 60,000 USD
Overview: Guidepoint’s Client Service team connects leading consultancies with the subject-matter experts they need to make informed business decisions. By understanding each client’s specific research needs and deliveri…
Est. 70,000 USD
Overview: Guidepoint’s Client Service team connects leading investment firms, consultancies, and corporations with the subject-matter experts they need to make informed business and investment decisions. By understanding…
Est. 70,000 USD
Overview: Guidepoint’s Client Service team connects leading investment firms, consultancies, and corporations with the subject-matter experts they need to make informed business and investment decisions. By understanding…
Est. 70,000 USD
Overview: Guidepoint’s Client Service team connects leading investment firms, consultancies, and corporations with the subject-matter experts they need to make informed business and investment decisions. By understanding…
Est. 70,000 USD
Overview: Guidepoint’s Client Service team connects leading investment firms, consultancies, and corporations with the subject-matter experts they need to make informed business and investment decisions. By understanding…
Est. 70,000 USD
Overview: Guidepoint’s Client Service team connects leading investment firms, consultancies, and corporations with the subject-matter experts they need to make informed business and investment decisions. By understanding…
Est. 70,000 USD
Overview: Guidepoint’s Client Service team connects leading investment firms, consultancies, and corporations with the subject-matter experts they need to make informed business and investment decisions. By understanding…
Est. 60,000 USD
Overview: Guidepoint’s Client Service team connects leading consultancies with the subject-matter experts they need to make informed business decisions. By understanding each client’s specific research needs and deliveri…
Est. 35,000 GBP
Overview: Guidepoint’s Business Development teams are passionate about expanding our reach with both new and existing clients. We support all of Guidepoint’s service offerings, helping to build relationships and communic…
Overview: Play a vital role in Guidepoint’s success. As an Associate on the Client Service team, you will understand your client’s unique business questions and identify the experts who can best provide the answers. You…
Overview Qsight is a high-growth division of Guidepoint focused on building data intelligence solutions for the healthcare sector. Qsight leverages proprietary datasets and rigorous analysis of alternative data sources t…
Overview: The Insights product is a new offering for Guidepoint's Institutional investment and corporate clients, that offers teleconferences, surveys, and in-person events. The teleconferences live in a transcript libra…
Overview The Insights product is an offering for Guidepoint's Institutional investment and corporate clients, now in its 9th year, that offers teleconferences, surveys, in-person events, along with a plethora of AI-facil…
Overview: The Strategic Research team is the backbone of Guidepoint’s success. The team is responsible for efficiently delivering Guidepoint’s services to our clients around the world. We work to understand each client’s…