Description
About Prophecy
The leader in AI-native data preparation and analysis, Prophecy is revolutionizing how the world's top enterprises turn data chaos into reliable insights. We introduce the AI-native data lifecycle (generate, refine, deploy) where our industry leading AI agents and humans work hand-in-hand in visual and document interfaces to analyze, transform and prepare data, to ship trusted insights at enterprise scale. To learn more, visit us on LinkedIn.
Position Summary
This is an exceptional opportunity to be a technical leader on the team that powers the heart of Prophecy: the Runtime, our orchestration and execution service. Every data workflow , whether generated by our AI data agents or built by a human in the visual editor , is planned, scheduled, and run at enterprise scale by Runtime, directly against the customer's own platform (Databricks, Snowflake, BigQuery, and more). When a business user asks an agent a question and sees trusted results seconds later, Runtime is what made it happen.
As a Staff Engineer, you will set the technical direction for this service: the distributed execution engine, the task coordinator, the multi-engine connectors, and the fast interactive previews that make the AI-native experience feel instant. You will own hard, ambiguous, high-leverage problems end-to-end , from architecture to production reliability , and raise the bar for the engineers around you.
We're seeking A players to work with A players. We're a high-growth company on a once-in-a-lifetime journey to revolutionize the way data is prepared and analyzed. Be a part of it.
The impact you will have
Own the engine behind the product. Architect and evolve the runtime service that turns agent- and human-authored data workflows into high-performance jobs that run reliably in production and interactively during authoring. This is deep-tech at the center of the platform.
Make it fast and make it scale. Design the distributed execution and task-coordination layer , scheduling, concurrency, resource management, streaming of sample data (interims) , so that pipelines run efficiently from a single interactive preview to large production workloads.
Run everywhere the customer's data lives. Extend execution across multiple engines and warehouses , Databricks, Snowflake, BigQuery, Spark, dbt, Airflow , with first-class connectors, query pushdown, and consistent behavior across clouds.
Power the AI data agents. Build the execution path that lets our agents generate, run, validate, and refine pipelines in a tight loop , the difference between a plausible answer and a trusted one.
Own reliability and observability. Drive production robustness: fault tolerance, graceful degradation, recovery and restartability, monitoring, and the SLAs that enterprise customers depend on.
Lead technically. Set architecture and standards, drive cross-team designs, and mentor engineers. Own features and initiatives end-to-end, as a founder would , from the requirement to delivery. We don't believe in hierarchy or years of experience for their own sake; we believe in ownership and impact.
What You’ll Need
- 5+ years of backend engineering experience building distributed systems or data-intensive services, with a track record of solving deep-tech problems
- Deep expertise in distributed systems fundamentals: scheduling, task coordination, concurrency, resource management, and fault tolerance will help you succeed.
- Strong proficiency in a systems language such as Go, Scala, or Java (our runtime is primarily Go and Python; functional-programming experience is a plus)
- Hands-on experience running data at scale , with Spark and/or cloud data warehouses (Databricks, Snowflake, BigQuery) and query execution
- A strong foundation in data structures, algorithms, and software design, with excellent analytical and debugging skills
- Experience owning the reliability, performance, and observability of a production service
- A demonstrated ability to lead technically , driving architecture, influencing across teams, and raising the quality bar for others
- Comfort working in short, agile, iterative development cycles
- Entrepreneurial spirit, priding yourself on getting things done with a high level of quality
- Strong desire to work in a fast-moving startup environment with high ownership and attention to detail
- You have been a top performer your entire career and hold yourself to very high standards
- B Tech or above degree in CS or equivalent
Preferred Qualifications
- Experience integrating with multiple execution backends or building multi-cloud data connectors
- You have built platforms or services that other engineers and products build on top of
- Familiarity with AI/agent execution paths , running and validating LLM- or agent-generated code
- Experience building an execution engine, orchestrator, or query engine (e.g. Airflow, Spark internals, query planners/optimizers such as Apache Calcite, or dbt)
- You have shipped software over multiple releases, including systems where you drove the delivery of the first version
- Ability to assess new technologies and make pragmatic choices that guide us toward a long-term vision
What You’ll Have At Prophecy
- Great company culture.
- Competitive compensation.
- Fair and Open Equity awards for everyone.
- Amazing work.
- Private medical insurance.
- Manage your own growth and career.
- Experience building a transformative product.
- High learning opportunity.
- End-to-end ownership of your project.
Similar jobs
Est. 300,000 USD
About Prophecy The leader in AI-native data preparation and analysis, Prophecy is revolutionizing how the world’s top enterprises turn data chaos into reliable insights. We introduce the AI-native data lifecycle (generat…
Est. 140,000 USD
About Prophecy Prophecy is building the next generation AI-powered data prep and analysis platform. Our platform enables business analysts and data teams to transform raw data into reliable, production-ready datasets and…
Est. 85,000 USD
About Prophecy Prophecy is building the next generation AI-powered data prep and analysis platform. Our platform enables business analysts and data teams to transform raw data into reliable, production-ready datasets and…
Est. 140,000 USD
About Prophecy Prophecy is building the next generation of AI-powered Data Prep and Analytics. Our platform enables business analysts and data teams to transform raw data into reliable, production-ready datasets and insi…
Est. 157,000 USD
About Workato Workato delivers enterprise infrastructure for the agentic era, redefining iPaaS and helping enterprises unify data, applications, processes, and AI into a single, governed platform. A leader in Enterprise…
Est. 80,000 USD
About Prophecy Prophecy is building the next generation AI-powered data prep and analysis platform. Our platform enables business analysts and data teams to transform raw data into reliable, production-ready datasets and…
Est. 144,000 USD
About Workato Workato delivers enterprise infrastructure for the agentic era, redefining iPaaS and helping enterprises unify data, applications, processes, and AI into a single, governed platform. A leader in Enterprise…
At Prodigal, we are building AI Agents for loan servicing and collections. Founded in 2018 by IITB alumni, our journey began when our founders Shantanu and Sangram faced the antiquated the $13B+ lending/collections indus…
Emergent builds autonomous coding agents that replace traditional software development by generating, testing, and deploying production applications directly from plain-language intent. Our systems run in production at g…
Est. 140,000 USD
About Prophecy Prophecy is the world’s most advanced data integration platform, designed to make complex data work simple and powerful. Designed natively for modern cloud data platforms, Prophecy uniquely serves the need…
At Prodigal, we are building AI Agents for loan servicing and collections. Founded in 2018 by IITB alumni, our journey began when our founders Shantanu and Sangram faced the antiquated the $13B+ lending/collections indus…
At Prodigal, we are building AI Agents for loan servicing and collections. Founded in 2018 by IITB alumni, our journey began when our founders Shantanu and Sangram faced the antiquated the $13B+ lending/collections indus…
Est. 150,000 GBP
Director, Forward Deployed Engineering At Scale AI, we're not just building AI tools—we're pioneering the next era of enterprise AI. The world's largest enterprises are racing to deploy Generative AI into the core of the…
At Prodigal, we are building AI Agents for loan servicing and collections. Founded in 2018 by IITB alumni, our journey began when our founders Shantanu and Sangram faced the antiquated the $13B+ lending/collections indus…
Emergent builds autonomous coding agents that replace traditional software development by generating, testing, and deploying production applications directly from plain-language intent. Our systems run in production at g…
Est. 150,000 USD
Arionkoder is an AI consulting firm helping companies build, embed, and own AI systems that drive real business outcomes. We combine Product Development, Artificial Intelligence, and Team Augmentation to craft digital pr…
Est. 144,000 USD
Emergent builds autonomous coding agents that replace traditional software development by generating, testing, and deploying production applications directly from plain-language intent. Our systems run in production at g…
Est. 90,000 EUR
Application and Interview Impersonation Notice: Impersonating another individual when applying for employment, and/or participating in an interview process to assist another individual in obtaining employment, with Preci…
Est. 120,000 GBP
Application and Interview Impersonation Notice: Impersonating another individual when applying for employment, and/or participating in an interview process to assist another individual in obtaining employment, with Preci…
Application and Interview Impersonation Notice: Impersonating another individual when applying for employment, and/or participating in an interview process to assist another individual in obtaining employment, with Preci…
Est. 120,000 EUR
Application and Interview Impersonation Notice: Impersonating another individual when applying for employment, and/or participating in an interview process to assist another individual in obtaining employment, with Preci…
Emergent builds autonomous coding agents that replace traditional software development by generating, testing, and deploying production applications directly from plain-language intent. Our systems run in production at g…
Est. 205,000 USD
Gradial is the marketing operations system of work that helps marketers and creatives move from idea to execution faster. Our platform orchestrates across martech stacks, workflows, and people to automate marketing execu…
Est. 120,000 USD
About CodeRoad CodeRoad provides end-to-end software development services, helping businesses scale with ideal infrastructure solutions. From staff augmentation to dedicated IT teams and general software engineering, our…
Company Overview Robots & Pencils is an applied AI engineering firm building the next frontier of business architecture. We design and ship AI co-workers that integrate into enterprise operations and deliver measurab…
About Workato Workato delivers enterprise infrastructure for the agentic era, redefining iPaaS and helping enterprises unify data, applications, processes, and AI into a single, governed platform. A leader in Enterprise…
Est. 233,000 USD
About Amperity—Why This, Why Now Most customer data is a lie by omission. Every consumer brand wants to know its customers. Unfortunately, almost none actually do. Data is scattered across a dozen systems, half of it con…
Est. 120,000 USD
About CodeRoad CodeRoad provides end-to-end software development services, helping businesses scale with ideal infrastructure solutions. From staff augmentation to dedicated IT teams and general software engineering, our…
At Prodigal, we are building AI Agents for loan servicing and collections. Founded in 2018 by IITB alumni, our journey began when our founders Shantanu and Sangram faced the antiquated the $13B+ lending/collections indus…
Robots & Pencils is an applied AI engineering firm building the next frontier of business architecture. We design and ship AI co-workers that integrate into enterprise operations and deliver measurable results for ou…