Driving marketing performance with reliable testing data

Learn how JustAnswer uses Bigeye to ensure optimal marketing results and drive customer acquisition and revenue performance for their business.

Connecting people to on-demand expert advice

JustAnswer is the world’s largest Q&A website that connects people to real, qualified experts. JustAnswer allows anyone to get immediate expert advice on topics ranging from legal matters and healthcare, electronics and software, and even tutoring and home repairs—all for a low monthly fee.

The JustAnswer website is a powerful sales and marketing tool and the marketing teams are constantly running A/B tests across the site to improve customer experience, increase acquisition, and optimize revenue. As such, it’s critical to the JustAnswer business that tests are always accurate and reliable.

Challenge

JustAnswer has multiple teams running concurrent A/B tests across various aspects of the website—sometimes dozens at once. This creates the potential for tests to encounter a variety of issues that can compromise results integrity and force the team to scrap the experiment entirely. These include website updates interference, overlapping tests, unexpected data infrastructure issues, and data quality problems.

Before Bigeye, each team would monitor the health of their tests using a custom dashboard developed internally. This required the test owners to continually check the dashboard for issues, with a high likelihood they wouldn’t find them until after the test was already compromised. This solution was also inflexible and difficult for business users to understand. In addition, new metrics had to be created manually in SQL, and the data model driving the dashboard was complex and prone to breaking.

This resulted in lost investment from compromised tests, too much engineering time spent building and fixing test monitoring dashboards, frustrated business users, and difficulty meeting JustAnswer’s website optimization goals.

Solution

After evaluating several data observability solutions, JustAnswer selected Bigeye to help monitor their testing data. They appreciated Bigeye’s flexible anomaly detection features, the ability to easily create custom business-logic checks and group them into collections, and the fact that Bigeye supported on-prem data sources like SQL Server in addition to cloud data warehouses. Now, with Bigeye, the JustAnswer marketing teams no longer need to manually monitor tests from a dashboard. Instead, the data engineering team has set up collections of predefined data checks in Bigeye that get deployed automatically when a new test launches.

This means if one team decides to run a pricing test, they simply create the experiment and Bigeye will automatically apply the correct pre-established data checks based on the experiment ID. As soon as the test starts, Bigeye will monitor for over 30 issue types—including dramatic price anomalies, unexpected changes in traffic patterns between different versions of the experiment, and data infrastructure issues that could cause the experiment to fail. If Bigeye detects an issue, it sends an immediate alert to Microsoft Teams so the analyst can react before the entire test is compromised.

In addition, the JustAnswer engineering team uses the Bigeye REST API to automate monitoring without needing to log into the Bigeye UI. This includes automatically deploying Bigeye monitoring on new tests, adjusting Bigeye alert thresholds based on traffic patterns, and automatically disabling Bigeye if the experiment is stopped early to eliminate potential false alerts.

Finally, the data quality liaison for each product team uses Bigeye’s fundamental data quality checks to monitor for data freshness and volume to ensure the underlying data is correct and up-to-date at all times.

Results

With Bigeye, the JustAnswer data engineering team has eliminated the need to develop and maintain their own quality monitoring tool, saving an estimated 320 hours in development and maintenance time alone.

In addition, they’ve empowered over 40 business users to monitor their own data with simple, ML-driven business-logic checks. Now, Bigeye automatically monitors over 50 experiments a month, eliminating the need for each team to spend half a day setting up monitoring. In total, business users now spend 100 hours less per month on data quality and are able to invest that time into feature development and improving business performance.