Announcing Bigconfig and developer tools for enterprise data monitoring as code

We’re thrilled to announce the general availability of Bigconfig, a powerful new way for Bigeye customers to define data monitoring as code and automatically apply it across their enterprise data warehouse.

Bigconfig is the first monitoring-as-code solution to support enterprise-scale data observability.  It was designed in partnership with data engineering teams who use Bigeye on hundreds to thousands of tables and need to integrate data observability into their existing version control and code review processes. Bigconfig includes declarative formatting, dynamic data tagging, and reusable monitoring definitions so customers can define what they want to monitor and how they want to monitor it with just a few lines of human-readable YAML.

As a team of engineers, we love that Bigeye gives us the option to create version-controlled data monitoring as code with an elegant, ‘Terraform-like’ solution. With Bigconfig, we use a  simple YAML file to define data monitoring rules and then let Bigeye automatically apply them across our entire data warehouse, including new tables that come online.

– Simon Dong, Udacity, Senior manager, Data engineering

With Bigconfig, data observability is:

• Dynamic and automated: Deploy metrics automatically on new data that matches your specifications with dynamic tagging
• Flexible: Specify and customize your monitoring with full granular control or apply any of Bigeye’s 60+ metrics out of the box
• Version controlled: Manage your data observability from a central location and track changes with version-controlled audit logs
• Fast and easy: Use optimized, human-readable YAML so there’s less code to write and no new language to learn
• Integrated: Manage your entire Bigeye operation without leaving the command line

☝️ See a Bigconfig YAML template ☝️

More monitoring, less configuration code

As we developed Bigconfig, we strived to ensure “as code” never became “too much code.” Other monitoring as code solutions require the user to specify each table to monitor and which checks to apply for each, making them impractical and unsuitable for enterprise applications. Bigconfig uses human-readable YAML so there’s no domain-specific language to learn, intuitive tagging with simple wildcards so you can specify and group tables to monitor across your warehouse, and the ability to customize and save metrics so you can apply them globally or directly to specific groups of tables.

Deeply customizable data observability with developer tools

In addition to Bigconfig, Bigeye customers now have access to Bigeye-CLI—an ergonomic command-line interface—and a robust set of developer tools so they can integrate data observability into any stack or workflow, no matter how custom. Bigeye-CLI offers access to Bigeye’s full automated monitoring, anomaly detection, and data diff’ing capabilities without leaving the command line. Adding outlier detection to thousands of tables, integrating blue-green testing into your data model’s CI/CD pipeline, or validating the migration of thousands of tables into a new data warehouse can all be automated in a few simple commands.

Complete control in an ergonomic API

Data engineers can also choose to interact with Bigeye directly through a friendly and familiar REST API, making it easy to extend Bigeye into a custom workflow. Are you using a custom ETL orchestrator and want to build a circuit-breaker pattern? No problem. Want to pull Bigeye monitoring data into a third-party tool like Datadog for additional analysis? The Bigeye REST API makes it fast and easy.

Deploy data observability right from your Airflow DAGs

For data teams orchestrating dbt jobs with Airflow, Bigeye offers several simple yet powerful Airflow operators to configure and deploy Bigeye quality checks right from an Airflow DAG.  

Python SDK for data science notebooks

Data Science teams are responsible for the performance of their models but they are often blind to the quality of the data coming into them. Bigeye provides a Python SDK that makes it easy to deploy Bigeye monitoring directly from a data science notebook like Jupyter or Hex for integrated data validation and confidence.

Ready to get started? Register for a live overview or request a demo to see Bigconfig in action.