June 5, 2024

Announcing End-to-End Enterprise Lineage for Hybrid Data Environments

Column-level lineage support for ETL platforms, from legacy to Snowflake.

Adrianna Vidal

Today, we launched an exciting new feature- column-level lineage support for ETL platforms, including the availability of Informatica Powercenter lineage, making data observability within hybrid data environments possible. 

This new feature allows customers with on-premises legacy data sources to use Bigeye’s data observability platform to map the lineage of their entire data analytics pipeline—even as data moves between on-premises sources and their cloud data warehouse—delivering comprehensive data quality insights to data consumers.

Addressing Lineage Challenges

Enterprise organizations often operate in hybrid environments that include both cloud and on-premises data sources. This creates a challenge in tracking data lineage, as data flows through various systems and transformations. 

Without comprehensive lineage coverage, enterprises face several risks:

  1. Data Quality Issues: Incomplete lineage can lead to undetected data quality issues, causing inaccurate analytics and decision-making.
  2. Compliance Challenges: Regulatory compliance often requires tracking of data. Gaps in lineage make it difficult to ensure compliance.
  3. Operational Inefficiencies: Manually troubleshooting and tracking down data issues can quickly become time-consuming and resource-intensive.
  4. Lack of Trust in Analytics: When business users cannot trace the origins of data in their dashboards, it undermines their trust in the analytics provided.

Speaking on our new features at Snowflake Summit, Bigeye CEO Kyle Kirwan said, “Many enterprises we meet really struggle with lineage. Unfortunately there are a number of platforms out there that claim to do lineage but the level of actual automation is pretty limited, which creates gaps in the lineage graph. Our new Lineage Plus connectors for ETL platforms close a key gap, keeping the lineage trace unbroken between source transactional databases and data lakes or warehouses.”

Industry-Leading Coverage

Bigeye is the first data observability company to offer end-to-end, column-level coverage that spans from cloud data warehouses to on-premises data sources and extends through to analytics dashboards. This is achieved through the integration of over 50 Bigeye Lineage Plus Connectors, which support transactional databases, data lakes, ETL platforms, analytics tools, and more.

Enhanced Data Monitoring

With the complete hybrid data pipeline mapped, business users and data analysts can deploy targeted monitoring on specific tables and columns. This ensures that every dependency powering their analytics dashboards is covered. When data issues arise, Bigeye’s Scorecard feature summarizes and presents these issues either alongside the dashboards or directly within the product.

The benefits of Bigeye's new column-level lineage support for ETL platforms include:

  • Enhanced data quality insights: Comprehensive lineage helps identify and resolve data quality issues efficiently.
  • Regulatory compliance: Detailed lineage supports compliance by documenting data movement and transformations.
  • Improved operational efficiency: Clear lineage maps reduce troubleshooting time and improve operational efficiency.
  • Increased trust in data: Business users gain confidence in analytics with traceable data.

Join The Webinar: End to End Data Observability for the Enterprise

In our upcoming session, we'll explore how Bigeye provides comprehensive data observability solutions for enterprises, bridging the gap between modern data platforms like Snowflake and legacy systems. Learn how Bigeye's innovative approach to data observability can help you ensure the reliability, performance, and integrity of your data pipelines across the entire data ecosystem.

Whether you're dealing with modern cloud platforms or traditional on-premise systems, this webinar is for you. Don't miss out on this opportunity to discover how Bigeye can transform your data observability practices and drive better business outcomes.

Register for our webinar on June 11th to learn more!

share this episode
Monthly cost ($)
Number of resources
Time (months)
Total cost ($)
Software/Data engineer
Data analyst
Business analyst
Data/product manager
Total cost
Common needs
Data engineers
Overall data flow. Data is fresh and operating at full volume. Jobs are always running, so data outages don't impact downstream systems.
Freshness + volume
Schema change detection
Lineage monitoring
Data scientists
Specific datasets in great detail. Looking for outliers, duplication, and other—sometimes subtle—issues that could affect their analysis or machine learning models.
Freshness monitoringCompleteness monitoringDuplicate detectionOutlier detectionDistribution shift detectionDimensional slicing and dicing
Analytics engineers
Rapidly testing the changes they’re making within the data model. Move fast and not break things—without spending hours writing tons of pipeline tests.
Lineage monitoringETL blue/green testing
Business intelligence analysts
The business impact of data. Understand where they should spend their time digging in, and when they have a red herring caused by a data pipeline problem.
Integration with analytics toolsAnomaly detectionCustom business metricsDimensional slicing and dicing
Other stakeholders
Data reliability. Customers and stakeholders don’t want data issues to bog them down, delay deadlines, or provide inaccurate information.
Integration with analytics toolsReporting and insights

Join the Bigeye Newsletter

1x per month. Get the latest in data observability right in your inbox.