Developed GCP cloud infrastructure for a US-based performance marketing & branding company
Overview
The objective of a client to automate data flows from external sources and migrate client’s data analytics product to the GCP environment
- To ensure up-to-date data is available in the GCP data warehouse environment
- To automate and establish data pipeline and data transformations in the GCP warehouse environment (Big Query).
- Ensure that a single data table is compatible and matches all conditions across all of the data products in the company
Solution
Google Big Query: Leverage Funnel.io’s built-in functionality to export data directly to BigQuery.
- Landing Table: Design a landing table in BigQuery to store raw data imported from Funnel.io every hour, maintaining the schema provided by Funnel.io.
Google Cloud Scheduler: Set up a Google Cloud Scheduler to execute hourly exports, ensuring up-to-date data is always available for analysis.
dbt Project in GitHub: Store your dbt project, including models and transformation logic, in a dedicated GitHub repository. This facilitates version control, collaboration, and easy tracking of changes.
Incremental Models: Implement incremental models within dbt to process only the new or updated data received in each hourly run. This optimizes performance and avoids redundant processing.
Google Cloud Looker:- the tool to help in the visualisations of the output results
Impact
Enhanced Data Science automation structure with Accelerated Model Development and Enhanced Feature Engineering
Improved data quality for dashboarding with timely insights & deeper data exploration with improved data-driven decision making
