Building a BigQuery Data Warehouse for a disruptive employment agency

Client background

This client is an NL based employment agency operating in different European countries. With +1,000 people employed and a +5 million candidate database, this client generates a yearly revenue of above EUR 400 million.

The challenge: The importance of data and proper analysis

As you can imagine for such a company, Data & Analytics is a key business functionality. Making the right matches based on data matters more than anything. The challenge for this client was to improve their reporting capabilities and integrate with their new CRM / HRM solution (AFAS).

The organization has various transactional systems, hosted both on-premise and on multiple cloud platforms. The disparate data sources require IT involvement to integrate this data into comprehensive reporting. Due to this, self service BI was not possible. Moreover, reporting on the operational sources caused issues in performance, and any change in one of the back-end systems might impact reporting. The siloed nature of the source data also made it difficult to ensure the quality of reporting, as data cleaning on a transactional system is troublesome.

As the customer had already decided to have these issues solved by implementing a Data Warehouse in Google BigQuery, they decided to team up with Devoteam because of our expertise in Google techniques and our recognition as Google Premier Partner.

The solution: A cutting-edge Data Warehouse using Google BigQuery

The solution implemented with Devoteam consists of a Data Warehouse using Google BigQuery, monitored by Stackdriver and access controlled via Google Cloud IAM.

Architectural & Design phase

In the Architectural & Design phase the several Data Warehouse Google Cloud components were selected plus integrated and a data model is designed which takes the specific characteristics of BigQuery and the renewed customer demands for reporting into account. Resulting in self service BI and integrating the new CRM / HRM solution.

The ETL Process

The ETL process was constructed using a variety of Google tools, such as Cloud Functions, App Engine, Cloud Storage and Dataflow. To orchestrate the developed ETL jobs Cloud Composer was used.

A Cloud based Data Warehouse resulting in many advantages

The introduction of one single cloud based Data Warehouse simplified and speeded up the reporting process, pertaining low TCO due to the usage of low cost cloud storage (and compute). By bringing all data together reporting was made easier, self service BI and collaboration is now possible and the quality of the data has improved. At the same time, protection of the data has increased as access can be managed on a more granular, role based level.

Infrastructure as code

In order to consistently roll out the solution over multiple environments we introduced Infrastructure as code. Because of this, the entire environment can be re-created in a matter of seconds by utilizing Terraform and automation using Google Cloud APIs.

Devoteam takes care of all aspects of digital transformation

A DevOps team, working with an Agile Scrum methodology, was introduced to cater for end-to-end platform and support implementations. Devoteam ensured full ownership of the project, by taking care of the Project Management, Analysis & Design, Architecture, Environment Setup, Solution Implementation, Release & Deployment and Testing of the solution.

We help you become a data driven company

Data and its value are growing exponentially in the 21st century. Major IT challenges are mostly related to data (and its massive growth). These challenges often include questions such as: How do I get a grip on data? Where is my data stored? Which (advanced) analytics possibilities are there to optimize its use? How reliable or trustworthy is my data?

Devoteam understands these challenges like no other and helps companies to become a high-quality data driven organization.

Data & Analytics solutions