Lead Product Data Scientist, Agents

At Klaviyo, we value the unique backgrounds, experiences and perspectives each Klaviyo (we call ourselves Klaviyos) brings to our workplace each and every day. We believe everyone deserves a fair shot at success and appreciate the experiences each person brings beyond the traditional job requirements. If you’re a close but not exact match with the description, we hope you’ll still consider applying. Want to learn more about life at Klaviyo? Visit klaviyo.com/careers to see how we empower creators to own their own destiny. Lead Product Data Scientist, Agents About the Team Klaviyo's Product Data Science team plays a critical role in shaping how we understand, measure, and improve product and business performance. We partner closely with Product, Engineering, Finance, and Go-to-Market teams to turn complex data into actionable insights that inform product strategy, customer experience, and long-term growth. As Klaviyo continues to expand its AI-powered capabilities, intelligent agents are becoming central to how businesses engage with their customers. Composer, Klaviyo's AI marketing agent, helps marketers analyze, build, and execute their marketing strategies, while Customer Agent enables brands to engage shoppers and answer questions across channels including chat, email, SMS, and WhatsApp. These products are evolving rapidly, creating new opportunities and challenges in understanding agent performance, measuring customer value, and connecting AI-driven experiences to meaningful business outcomes. Our team takes an AI-first approach to data science, leveraging emerging technologies and agentic tools to accelerate analytical workflows, improve decision-making, and focus our efforts on the complex questions that matter most to customers and the business. About the Role We're looking for a Lead Product Data Scientist, Agents to own and advance Klaviyo's measurement and data science strategy across Composer and Customer Agent, combining deep technical expertise with strong product and business judgment to influence some of our most important AI investments. This is a highly visible, hands-on technical leadership role at the intersection of data science, AI product development, and business strategy. You'll define how we measure the adoption, engagement, quality, cost, and business impact of our AI agents, establishing the analytical foundations that help teams understand what creates meaningful customer value. This isn't just about building dashboards or tracking traditional product metrics. We're looking for someone who can tackle emerging measurement challenges, develop rigorous approaches to evaluating agent performance, and determine whether AI-powered experiences are genuinely improving customer outcomes. You'll combine experimentation, causal inference, and advanced analytics to help leaders make informed decisions in an environment where established measurement frameworks are still evolving. You'll partner closely with Product, Engineering, Finance, and executive leadership to shape product strategy, evaluate pricing and monetization opportunities, and translate complex analytical findings into actionable recommendations. You'll also help define the technical direction of agent analytics at Klaviyo, mentor other data scientists, and establish the shared metrics, methodologies, and data foundations that enable teams to make confident, data-driven decisions. How You'll Make a Difference • Own agent measurement end to end across Composer and Customer Agent, developing comprehensive measurement frameworks that connect adoption, engagement, agent performance, customer value, and revenue impact. • Define what successful agent performance looks like by partnering with Product and Engineering to establish meaningful measures of agent effectiveness, task completion, customer outcomes, failure rates, and the trade-offs between quality, cost, and business value. • Build scalable agent evaluation frameworks by combining conversation data, agent traces, tool-call outcomes, human evaluation, and LLM-based classification to assess quality, identify failure patterns, and understand how changes to models and agent behavior influence product performance. • Measure causal impact and incremental value through experimentation and advanced statistical methods, including quasi-experimental approaches such as difference-in-differences, matched comparisons, staged rollouts, and synthetic controls, to determine how AI agents influence customer behavior, retention, and revenue. • Connect agent performance to business strategy by partnering with Finance and Product teams to evaluate usage-based pricing, packaging, credit economics, model costs, conversion, and the financial implications of product investments. • Establish trusted metrics and analytical standards by defining consistent measurement methodologies, governed data models, and evaluation practices that enable teams to make informed decisions as AI products and underlying technologies evolve. • Lead with an AI-first approach to analytics, leveraging coding agents, LLMs, and emerging tools to accelerate SQL development, data modeling, dashboard creation, conversation classification, and analytical workflows while maintaining rigorous validation and quality standards. • Build scalable, self-service analytics capabilities that empower product managers and cross-functional stakeholders to independently explore agent performance, evaluate customer outcomes, and identify opportunities for improvement. • Partner with Engineering and Analytics Engineering to develop reliable instrumentation, data pipelines, and dbt models that capture agent interactions, support robust evaluation, and maintain data quality as new capabilities are introduced. • Lead the technical direction of agent analytics, reviewing analytical approaches, mentoring other data scientists, establishing best practices, and raising the bar for statistical rigor, measurement