Snowflake has become a popular choice for businesses that work with large volumes of data. Its decoupled and elastic architecture allows data teams to sidestep the limitations of traditional data warehouses with fully-managed cloud infrastructure.
However, Snowflake’s usage-based pricing model can become costly if left unchecked, and keeping it under control requires ongoing optimization - both in terms of Snowflake’s internal configurations, and in the tools that run ELT or BI workloads using Snowflake’s compute engine.
We’ve recently helped a SaaS company achieve dramatic savings while improving the performance of their Snowflake data warehouse. Read on for the details.
The Challenge
GridMatrix uses big data analytics to provide insights into road usage and transit patterns. The company is heavily reliant on Snowflake for its business intelligence, with Sisense providing the BI and reporting layer.
Due to large volumes of Salesforce raw data being stored in Snowflake and unoptimized data ingestion, they were consuming Snowflake credits at an unsustainable rate. This was also slowing down some of the dashboards that were reading the data from Snowflake.
The Solution
As part of a data warehouse and BI project, QBeeQ worked with the GridMatrix team to implement the following changes:
Replacing the raw data files staging method with more efficient data storage.
Redesigned the live Sisense model to better utilize Snowflake resources.
Replaced the Snowpipe data ingestion pipelines with Snowflake Tasks, which are less dependant on staging tables and more compute-efficient.
Enabling multi-warehouse mode, monitoring warehouse node usage and encouraging Snowflake users to use lower warehouse tiers where possible. In many cases warehouse size went from XL/Med to Small, and in some instances even XS.
The Results
The technical strategies employed by the QBeeQ team, in close collaboration with GridMatrix, ensured optimal performance and a simple data model. QBeeQ’s team was able to improve many aspects of the way Snowflake was configured and used, resulting in improved business outcomes.
~40% improvement in dashboard performance
40% reduction in Snowflake credits consumption and cost