Keeping third-party data fresh

SignalFire uses Bigeye to monitor millions of data points and fuel their investment strategies with reliable data.

About SignalFire: A data-driven venture capital firm

SignalFire is a venture capital firm built from the ground up with a data-driven investing strategy. The firm leverages data from millions of data points to make smart investment decisions. That data also powers Beacon Talent, SignalFire’s innovative AI-based recruiting platform that tracks the world’s top engineers, data scientists, product managers, designers, and business leaders.

Challenge: Where third party data can’t go stale

Much of the data SignalFire leverages comes from external sources. The data lands in a wide range of formats, often unstructured, and has varying degrees of quality. The data engineering team uses this wide variety of data to build Beacon Talent. Because those products are core to SignalFire’s strategy, the firm maintains a high bar for data quality. Any loss of data, degradation, or deviation from the norm must be resolved.

Before implementing Bigeye, the data engineering team ran ad-hoc queries to check on potential issues with the data. But at the scale that SignalFire operates, the data engineering and data science teams needed tools to allow them to trust all of the data, not just some of it.

Solution: Keeping data fresh with Bigeye’s freshness monitoring

With Bigeye pointed at their MySQL database, SignalFire’s data engineers have an easy way to continuously monitor their huge volumes of data. Bigeye is tailor-built to make monitoring easy at scale, enabling them to monitor the millions of data sets they ingest.

Tony Ho, SignalFire’s Director of Engineering, was able to deploy Bigeye’s built-in freshness tracking metrics on the third-party data his team depends on. Bigeye’s Autothresholds automatically detect and set alerting thresholds for him without needing to configure the expected update timings for each data source.

In the first week of implementation, Bigeye caught an issue that SignalFire said would have normally taken days to notice.

Result: 2 million+ data points validated automatically