The Role of Database Administrators in Snowflake: Optimization, Cost Control, and More
The digital landscape of today presents a multitude of data handling platforms and processes, requiring diverse expertise. It's become clear that understanding the division of roles is critical, particularly the distinction between Data Engineers and Database Administrators (DBAs).
Data Engineers are experts in data PROCESSES, skilled at cleansing and transforming data into an efficient and flexible analytics system. In contrast, DBAs are experts in specific data PRODUCTS, focusing on improving the performance and efficiency of interactions with the data.
I've engaged in numerous discussions about the role of DBAs in the cloud world, specifically with analytical systems like Snowflake. Some dialogues have been quite concerning, such as a conversation with a Databricks Solutions Architect who confessed to not considering performance while architecting an analytics system.
In the evolving data landscape, expecting a single role to be an expert in all areas - from storage, transactional databases (SQL, NoSQL), analytical systems (warehouses, lakes, lakehouses), streaming, ETL/ELT/Transformation, Integration, to visualizations - is a tall order.
The image below, although not exhaustive, gives an idea of the complexity and diversity of today's data world. If the thought of having more than 20 tabs open in your browser sends you into a spin, it might be best to ignore it.
In the context of Snowflake, the DBA role gravitates more towards optimization tasks directly linked to cost control and administrative duties. These may include refreshing environments, handling security, and ensuring data retention policies meet SLAs.
Managing costs in an analytical system like Snowflake can be a challenge, and without proper optimization, these expenses can surge. It's not unusual for our clients to save between 30-60% on their Snowflake expenditure after implementing our optimization recommendations. For an in-depth view of one such engagement, see my blog post from May 16.
A recurring theme in my discussions with data engineers revolves around tuning queries in Snowflake. There's a common misconception that "you don’t need to tune Snowflake, it tunes itself". Yes, Snowflake is exceptionally fast, so much so that it can give the illusion of self-tuning. But the reality is, SQL remains SQL. Poorly written SQL queries can still consume more time and, consequently, increase costs. This notion is not just my perspective but a fact reiterated in Snowflake training sessions and by Snowflake leadership during earnings calls.
To illustrate, consider these two queries, both designed to return the same dataset – all the data in these tables for June 2022. However, while one uses a range search on date, the other applies a function to search for data from the 6th month of 2022.
In summary, while Snowflake's speed and performance are undeniable, proper optimization and effective query writing remain crucial to managing costs and ensuring efficiency. The role of a DBA in Snowflake, thus, is far from obsolete – it's a role that's pivoted towards fine-tuning the system for peak performance and cost-effectiveness.
Contact Moser Consulting to learn more about DBA’s and Snowflake.
