Kyle Kirwan
kyle-kirwan
Thought leadership
-
September 26, 2024

From Data Analyst To Founder | Bigeye Co-Founder Kyle Kirwan on Leadership in Tech

min read

Kyle Kirwan
Get Data Insights Delivered
Join hundreds of data professionals who subscribe to the Data Leaders Digest for actionable insights and expert advice.
Stay Informed
Sign up for the Data Leaders Digest and get the latest trends, insights, and strategies in data management delivered straight to your inbox.
Get the Best of Data Leadership
Subscribe to the Data Leaders Digest for exclusive content on data reliability, observability, and leadership from top industry experts.

Get the Best of Data Leadership

Subscribe to the Data Leaders Digest for exclusive content on data reliability, observability, and leadership from top industry experts.

Stay Informed

Sign up for the Data Leaders Digest and get the latest trends, insights, and strategies in data management delivered straight to your inbox.

Get Data Insights Delivered

Join hundreds of data professionals who subscribe to the Data Leaders Digest for actionable insights and expert advice.

In this podcast episode, Kyle Kirwan, co-founder of Bigeye, walks Ben Book of GigaOm through his journey from a data analyst at Uber to launching a cutting-edge data observability company. With a background in industrial engineering, Kyle’s unconventional path gave him a fresh perspective on data science and product management, which has shaped how he navigates leadership in the tech space.

Kyle shares stories from his time at Uber, where he worked on scaling a global data platform. The lessons learned—managing data quality, lineage, and observability in a fast-growing enterprise—formed the foundation of Bigeye’s mission today.

Key takeaways:

  • The power of data in driving strategic business decisions and innovation at scale.
  • How to build trust in your data and make it central to high-impact areas of the business.
  • Tackling scaling challenges by focusing on operational efficiency and data reliability.
  • Why enterprises need data observability, particularly when balancing legacy systems with cloud technologies.

By the end, you’ll walk away with actionable insights into how data-driven leadership and smart decision-making can help enterprises scale their data systems and improve business outcomes.

Watch the full interview here:

share this episode
Resource
Monthly cost ($)
Number of resources
Time (months)
Total cost ($)
Software/Data engineer
$15,000
3
12
$540,000
Data analyst
$12,000
2
6
$144,000
Business analyst
$10,000
1
3
$30,000
Data/product manager
$20,000
2
6
$240,000
Total cost
$954,000
Role
Goals
Common needs
Data engineers
Overall data flow. Data is fresh and operating at full volume. Jobs are always running, so data outages don't impact downstream systems.
Freshness + volume
Monitoring
Schema change detection
Lineage monitoring
Data scientists
Specific datasets in great detail. Looking for outliers, duplication, and other—sometimes subtle—issues that could affect their analysis or machine learning models.
Freshness monitoringCompleteness monitoringDuplicate detectionOutlier detectionDistribution shift detectionDimensional slicing and dicing
Analytics engineers
Rapidly testing the changes they’re making within the data model. Move fast and not break things—without spending hours writing tons of pipeline tests.
Lineage monitoringETL blue/green testing
Business intelligence analysts
The business impact of data. Understand where they should spend their time digging in, and when they have a red herring caused by a data pipeline problem.
Integration with analytics toolsAnomaly detectionCustom business metricsDimensional slicing and dicing
Other stakeholders
Data reliability. Customers and stakeholders don’t want data issues to bog them down, delay deadlines, or provide inaccurate information.
Integration with analytics toolsReporting and insights
about the author

Kyle Kirwan

Chief Product Officer, Bigeye

Kyle Kirwan is Co-Founder and Chief Strategy Officer of Bigeye, where he leads strategic partnerships, prototype development, and other zero-to-one projects.

Kyle’s journey to founding Bigeye began at Uber, where he helped scale the company’s experimentation and data platforms during a period of hypergrowth. As a product leader and former founding data scientist on Uber’s experimentation platform, he worked on standardizing metrics across thousands of A/B tests that shaped rider, driver, and pricing experiences for millions of users.

It was at Uber that Kyle met Egor Gryaznov. Shortly after Egor joined, he launched Uber’s first SQL bootcamp. Kyle signed up partly out of curiosity, and partly to make sure the new guy actually knew his stuff. They quickly bonded over giving each other increasingly complex SQL challenges to solve.

As Uber’s data ecosystem grew to hundreds of petabytes and thousands of weekly users, Kyle saw a pattern emerge: testing the data pipelines was valuable but didn’t scale. His team experimented with using machine learning models on the daily data profiles of tables in the data lake to see if anomalies could be identified without manually writing data quality checks. This technique would later be termed data observability.

In 2019, Kyle and Egor co-founded Bigeye to use the lessons learned at Uber to transform data management in the enterprise. Today Bigeye serves some of the world’s largest organizations and ensures their data is trustworthy, and that their enterprise AI initiatives are grounded in that trusted data.

about the author

about the author

Kyle Kirwan is Co-Founder and Chief Strategy Officer of Bigeye, where he leads strategic partnerships, prototype development, and other zero-to-one projects.

Kyle’s journey to founding Bigeye began at Uber, where he helped scale the company’s experimentation and data platforms during a period of hypergrowth. As a product leader and former founding data scientist on Uber’s experimentation platform, he worked on standardizing metrics across thousands of A/B tests that shaped rider, driver, and pricing experiences for millions of users.

It was at Uber that Kyle met Egor Gryaznov. Shortly after Egor joined, he launched Uber’s first SQL bootcamp. Kyle signed up partly out of curiosity, and partly to make sure the new guy actually knew his stuff. They quickly bonded over giving each other increasingly complex SQL challenges to solve.

As Uber’s data ecosystem grew to hundreds of petabytes and thousands of weekly users, Kyle saw a pattern emerge: testing the data pipelines was valuable but didn’t scale. His team experimented with using machine learning models on the daily data profiles of tables in the data lake to see if anomalies could be identified without manually writing data quality checks. This technique would later be termed data observability.

In 2019, Kyle and Egor co-founded Bigeye to use the lessons learned at Uber to transform data management in the enterprise. Today Bigeye serves some of the world’s largest organizations and ensures their data is trustworthy, and that their enterprise AI initiatives are grounded in that trusted data.

Get the Best of Data Leadership

Subscribe to the Data Leaders Digest for exclusive content on data reliability, observability, and leadership from top industry experts.

Stay Informed

Sign up for the Data Leaders Digest and get the latest trends, insights, and strategies in data management delivered straight to your inbox.

Get Data Insights Delivered

Join hundreds of data professionals who subscribe to the Data Leaders Digest for actionable insights and expert advice.

Join the Bigeye Newsletter

1x per month. Get the latest in data observability right in your inbox.