The Challenge: Modernizing Data to Support Growth

A leading US residential solar company was scaling rapidly to meet surging demand for clean, sustainable energy. To ensure that they could continue to provide exceptional customer service as they grew, they needed a better way to manage growing volumes of data across installation operations, provider performance, customer operations, and sales. 

Their legacy Oracle data stack required IT and data team support for almost every data request, draining time and resources. The resulting backlog led some employees to create their own data tools, which led to conflicting data sets and silos across the organization. To keep up with their growth, they needed a scalable infrastructure, centralized data, and reporting that was flexible and readily available.

Our Solution: Cloud Migration without Business Disruption

Our team worked with internal stakeholders to identify their use cases, and advised on and implemented a migration to Google Cloud’s analytics platform—including Looker and BigQuery. We then worked with their data engineering teams to streamline their data warehouse migration and modernize their analytics without business disruption.

Rather than build complicated data pipelines with complex ETL processes, we loaded most data directly into BigQuery without transformation. We leveraged the power of BigQuery and Cloud Dataflow to transform approximately 20 percent of the data available in BigQuery. But a majority of data transformation occurred at query time through a combination of Looker’s Git-versioned data modeling layer, LookML, and the BigQuery query engine. This allowed the company to avoid complicated, brittle, and expensive ETL processes, and simplified the data pipeline.

How Did It Work Out?

Since our client’s migration from their on-premises legacy Oracle data stack to a modern cloud environment, they’ve created infrastructure and business-wide efficiencies to help them meet the growing demand for solar power. They’ve reduced ETL complexity, enabled fast queries, and made data accessible and trusted throughout the organization.

Google BigQuery and Looker together resulted in significant efficiency gains, reducing the overall data lifecycle by at least 50%. With 100% of their data now migrated from on-premise to the Cloud, they are making data-driven decisions to best serve their customers, as well as meet their growth metrics. And by leveraging LookML, Looker’s modeling layer, to unify metric definitions throughout the company, everyone can now be confident that they’re using the same metrics when they use data to guide strategy and decision making. 

To power organization-wide access to their data, the client implemented a hub-and-spoke model for self-service analytics. At the center, they have the core BI team that creates a single source of truth, and then provides data and dashboards across every level and department via curated BigQuery tables and LookML schema. The governed hub accounts for approximately 60% of queries across the organization, with the remainder being satisfied with models maintained by analysts who work in functional spokes like marketing, customer operations, or project operations.

Background

The heads of Finance and Sales of these major broadcasters wanted to better forecast revenue from directly sold digital ad campaigns in order to answer key business questions:

1. How does revenue pace, and are the sales teams on target to achieve revenue goals?
2. How does that revenue pacing differ by quarter and client?
3. Is there a way to identify potential problems in time to address them?

To answer these questions, these clients wanted an easy and more accurate way to understand how revenue accumulates on the books, and to compare revenue year on year to assess whether the teams are meeting, overperforming, or underperforming planned goals.

Advertising and traffic patterns aren’t always in alignment. Sometimes, for reasons that have little to do with the business, digital traffic patterns can be high, but selling low, or vice versa. 

The organizations wanted to examine their historical data, or track records, so they could identify recognizable patterns in how revenue collects on their books. In essence, the client wanted to plot booking curves for revenue.

Going deeper, individual brands, such as the sports or entertainment sections of the site, experience their own booking curves, and understanding those variances would enable the sales team to proactively spot issues that need addressing.

Challenge: Messy Painful Process

There were a number of challenges creating hurdles on the path to achieving the clients’ goals.

• Extracting and normalizing historical data was extremely difficult. At its heart, plotting a booking curve is a data science exercise, which means that all data inputs need to be standard. But the historical data had significant variations in categories over time. There was no authoritative naming of categories or time period.
• All existing historical analyses were built in Excel workbooks. That meant extracting data was far from straightforward. And the analysts continuously evolved their analysis, which meant there were a lot of natural variations within these Excel workbooks.
• Booking curves were complex. A booking curve is the intersection of the capabilities of the sales staff, combined with the buying habits of each client. A site with numerous categories — general news, entertainment, sports, health, etc. — will have a unique booking curve per section, making them a complex exercise in data science.
• Data disparity was a big challenge. To get a full picture of the revenue, the head of sales needed to tap into a wide range of data sources, including its Salesforce.com and Operative systems. But sales professionals are experts in the dynamics of the market, not data science.
• Data capture and data warehouse approaches needed to change. Because this was a new initiative, the data wasn’t stored in a way that would facilitate this type of exercise. The clients needed to change the way dates were stored, but lacked the skills or time to do so.  The project required fluency in more robust databases such as Teradata, Snowflake, as well as BI tools, such Tableau, Microstrategy, or Looker.
• Real-time data was difficult to achieve. Although the teams used spreadsheets to gain some insights, the process wasn’t scalable, and required significant manual input. That meant it was difficult to get insights on demand updated in real time.

No future or historical views were available. The spreadsheets didn’t allow a level of drilling, either in future or historical views, that the sales team needed to track current performance on the booking curve.

Solution

The clients asked us to develop a model as quickly as possible, and we began creating an authoritative way to represent categories and time periods. Next, we pulled and normalized the historical data from the Excel workbooks, Salesforce.com, and Operative, and imported it into Python which is well suited for normalizing disparate datasets and crunching data. 

Creating the booking curves was more of a data science exercise. Using linear regression, we plotted how sales accumulated on the books over time. This data exercise revealed that each quarter begins with some portion of its available inventory already sold. As the year progresses, the percentage of pre-sold inventory increases, so that by Q4, up to 90% of the revenue the network would receive for the quarter was already booked. Looking back over several years, we were able to build annual and quarterly booking curves – insight that has implications for both revenue and inventory forecasting. Since the curves change across sales teams, properties, and time periods, we also needed to smooth out agreed upon anomalies or variations in the signal, but still retain the general patterns. 

Our team also built numerous features into the model, such as what-if scenarios to help the head of sales to answer questions about the impact of potential market events on revenue, and alerts that trigger if revenue isn’t tracking as anticipated.

To make it easy for the head of sales to spot trends quickly and easily, we fed the data into Tableau, which created user-friendly dashboards, available from any PC or laptop. The dashboards are continuously updated in real time, so accurate information is always available.

Results More Timely And Accurate, Every Step Of The Way

• Timely and Accurate Insights. The heads of Sales now have more timely and accurate insights into sales efforts at every step along the way. The booking curves eliminate any surprises by making it easy to see how sales are trending compared to prior quarters.
• Far More Accurate Forecasting of Revenue Results. With the booking curves in place, sales management is able to forecast revenue results with much higher degrees of accuracy, and with a lot less effort on their part.
• Early Warning. The solution provides an early warning signal for the heads of sales and finance teams, that earnings may not be what they anticipated. This signal allows them to take corrective actions, and advise the appropriate C-level executives.
• What-If Scenarios. The heads of sales can easily test what-if scenarios and predict their implications on revenue.
• More Time to Focus on Sales. The dashboards have eliminated the countless hours spent entering data into spreadsheets, which means sales executives have more time to focus on strategic sales initiatives.

Background

A leading financial services website sells inventory via a direct sales model and programmatically in open auctions. To better forecast its revenue, it needed to understand how revenue tracked, in the past, present, and future.

Like all publishers, the company’s revenue doesn’t accumulate in a linear fashion; it accumulates episodically. In the beginning of the first quarter, its sales teams are selling into the first quarter, but by March, they’re selling into Q2 or even Q3. This made it difficult to accurately forecast quarterly revenue. Additionally, the company wanted to know the implications on its programmatic revenue if conditions didn’t occur as expected on the direct side of the house.

Challenge

The company had several challenges to overcome to achieve its revenue forecasting goals:

• Data disparity. Client and revenue data was generated and housed in multiple systems, including Salesforce.com and Operative on the direct sales side, and PubMatic, Rubicon Project and AdEx on the programmatic side. All the data needed to be retrieved and normalized into a single program.
• Lack of granular programmatic data. Programmatic data was only available on a weekly level, and the client had little information on the actual pricing received.
• Difficulty getting real-time data. The data analysis team created spreadsheets to gain some insights, a process that wasn’t scalable and required significant manual input. That meant it was difficult to get insights on demand or updated in real time.

No future or historical views. The spreadsheets didn’t allow a level of drilling, either in a future or an historical view, which meant the team couldn’t gain any significant insights.

Help Us Understand Your Challenges

Analyzing your data platform needs starts with a conversation. Contact us now to get the conversation started.Contact Us

Solutions

We went to work designing booking curves for the client, so they could forecast both their direct sales and programmatic revenue, track changes that occur over time, and forecast the impact of unexpected events in direct sales revenue on programmatic earnings, and vice versa.

For the booking curves to be useful, we needed to design models that tracked revenue year-over-year, as well as quarter-over-quarter. Additionally, booking curves can vary from client to client, even for the same order.

We began by pulling historical data from Salesforce.com and Operative into Python, which is well suited for normalizing disparate datasets and for crunching data. Next we pulled data from PubMatic, Rubicon Project, and AdEx using APIs.

Although the client had an abundance of data for its directly sold revenue, its programmatic data was much less granular. Creating a booking curve for the programmatic side of the house required a minimum of week-level views of prior volumes, as well as an analysis of the actual pricing involved and the price settings. We had to make smart assumptions and build a collaborative model that assessed:

• What is that strategy?
• How short or long-term is that strategy?
• What is it driven by?

To help the data analysis team share the insights throughout the company, we fed the data model into Snowflake which was accessed via Tableau, and created custom dashboards to display the booking curves.

How Did it Turn Out?

• Booking curves for accurate forecasting. The company now has accurate booking curves to forecast its direct and programmatic revenue.
• What-if scenarios. Built-in scenarios allow the company to predict implications of direct sales activities on programmatic revenue and vice versa.
• Early warning. The solution provides an early warning signal that earnings may not be what they anticipated. This signal allows them to take corrective actions, and advise the appropriate C-level executives.