Netezza to BigQuery Migration and ETL Optimization
RetailBusiness Impacts
73%
reduction in query cost with optimized scripts
70%
improvement in performance with serially optimized queries on BigQuery
90%
improvement in performance with optimized query orchestration
Customer Key Facts
- Country : USA
- Industry : Retail
Problem Context
The client is a multinational lifestyle retail corporation with a diverse brand portfolio. They wanted a specific subset of data to be transferred from Netezza to BQ, followed by the optimization of complex Netezza jobs on BigQuery. The client wanted to know if customers were classified as “New,” “Active,” “Re-Activated,” or “Unmatched” at different levels, including DAY, WEEK, MONTH, QUARTER, and YEAR.
Challenges
- Gigabytes of data stored in on-prem Netezza boxes in the form of tables
- Long run times (3 hours+) for business critical queries (80+ queries)
Technologies Used
Google BigQuery
Google Cloud Composer
Comparing the performance of running queries within Netezza and BigQuery
Solution
Quantiphi helped the client to compare the performance of running queries within Netezza and BigQuery by translating and optimizing existing Netezza jobs on BigQuery specific to the NAR (new active re-active) use case.
Results
34% reduction in the total number of queries by combining smaller query actions with larger ones
