Executive summary
Many companies skipped normal technology upgrade cycles as they struggled to manage the volatility caused by supply chain disruptions and citizen lock downs during the Global Pandemic. Companies must now prepare for the post-recession rebound in order to capitalize on the upcoming growth cycle. However, they are now challenged by an increased cost of capital and a rapidly changing pace of technology, which increases the degree of difficulty in choosing the right modernization path for their data estate. Choosing a Databricks-centric data architecture gives companies the lowest total cost of ownership and the highest return on investment available in the market. Coupled with the highest performance benefits, companies should modernize to Databricks now to capitalize on the next wave of economic growth.
Historical context
When an economy is healthy, companies have the luxury to prioritize experimentation and embrace new technology because business growth makes the failure of any risky venture. In a down economy, companies need to be much more judicious with their deployment of funds because of the uncertainty of the return of capital and the potential negative impacts on business operations. This economic volatility increases the degree of difficulty for companies trying to embrace new technology.
Prior to the Global Pandemic, the world experienced a period of significant economic growth as it recovered from the global recession of 2008. Powered by historically low interest rates and high consumer purchasing, many companies took this opportunity to modernize their technology platforms as cloud hyper-scalers like AWS and Azure became generally available and the case could be made to move away from traditional data centers. Companies were able to make an easy business case to move to the cloud.
The Global Pandemic introduced extreme volatility across every industry causing many companies to delay upgrading and modernizing their data estate infrastructure. The shutdown of global supply chains and the restriction of consumer movement caused exponential increases in transportation costs, the shut-down of global tourism, and the halt of shopping as we know it, which resulted in the greatest world economic decline since the 1930s. The result was that companies have had significant difficulty in outlaying capital to modernize their data estates over the past four years choosing instead to either conserve cash or cover operational losses.
The case for modernizing… now
Many economists expect a significant recovery to occur starting in Q3 of 2024. Companies need to get ready now for that recovery so they can capture the upcoming wave of growth instead of sitting on the sidelines waiting for the next cycle. The upside for companies that skipped prior upgrade cycles – they can save considerable cost and time by consolidating legacy workloads, eliminating specialty applications, and centralizing legacy data sources around a modern data estate architecture. Companies operating data estates based on an architecture over four years old have the opportunity to make wholesale upgrades rapidly and inexpensively.
We encourage companies to upgrade their data estates during downcycles. These periods of lower transaction volumes help minimize the impact on business operations and increase the chance of adoption of new work paradigms. Three major requirements for preparation of a flexible and scalable architecture during a downturn are 1) vendor consolidation, 2) specialty vendor elimination, and 3) investment in strategic scale partners. Each of these provides the basis for an effective Total Cost of Ownership (TCO) business case that supports upgrading even in times of higher-than-normal cost of capital.
Modernizing your data estate during downturns offers compelling advantages driven by cost benefits of increased vendor competition and operational benefits of lower business volumes. By consolidating vendors, organizations streamline their procurement processes, reduce complexity, and lower costs through bulk purchasing and negotiated agreements. Eliminating specialty vendors allows for a more cohesive and integrated ecosystem, simplifying management and enhancing interoperability. Lastly, investing in strategic scale partners not only ensures access to cutting-edge technologies and expertise but also fosters deeper partnerships that can drive innovation, agility, and long-term value creation. Overall, these strategic moves optimize resource allocation, minimize risks, and position companies for sustained success in today’s dynamic business landscape.
Modernizing with Databricks
Companies have a number of data intelligence technology partners to choose from when deciding how to architect their modern data estate. We prefer a Databricks-oriented architecture for a number of reasons. Databricks offers scalability and elasticity, allowing users to adjust computing resources based on actual workloads, eliminating overprovisioning, and minimizing idle resource costs. Databricks also integrates seamlessly with cost-effective cloud data lakes and provides a unified platform for data engineering, data science, and machine learning. This consolidation of tools and capabilities reduces licensing costs and fosters collaboration, enhancing productivity and reducing project timelines.
Additionally, Databricks supports advanced analytics and machine learning, contributing to cost savings through improved decision-making and automation. Robust data governance and security features help organizations avoid costly data breaches and regulatory fines, while integration with data warehousing solutions enhances the overall data analytics ecosystem. Overall, Databricks empowers organizations to control costs, improve efficiency, and derive more value from their data investments, making it a compelling choice for modern data platforms.
The paramount consideration for companies lies in the intricate transparency and governance mechanisms inherent within Databricks, pivotal factors that directly influence the total cost of ownership (TCO), particularly at scale. In contrast, alternative solutions like Snowflake incur higher initial costs and exhibit exponential cost escalation as the platform expands. Notably, Snowflake proves to be substantially pricier, with ETL and BI workloads costing on average five times more, and up to ten times more for complex workloads. Moreover, these alternatives operate opaquely, functioning as “black boxes,” thereby limiting Fin Ops and Centers of Excellence’s insight and oversight into cost structures and performance benchmarks. Implementing alternative solutions often necessitates extensive engineering efforts to develop custom integrations and connectors to legacy systems, essential for achieving the comprehensive experience inherently offered by Databricks. Consequently, the fluctuating TCO calculations over time significantly impede capital allocation in other critical areas of business operations.
Modernization benefits
Data democratization for analytics
Databricks has invested heavily in R&D over the past two years and is releasing advanced features like Delta Sharing and moving into GenBI that extends access to organizations data outside of Databricks. This extensibility reduces the need for multiple specialized tools and resources, ultimately saving on software licensing and infrastructure costs.
Integration of legacy systems
Databricks simplifies data integrations within a data estate by providing a unified platform with support for various data sources, built-in connectors, ETL capabilities, streaming data processing, machine learning integrations, and collaborative tools. It seamlessly connects to data lakes, data warehouses, relational databases, and streaming data sources, enabling organizations to ingest, transform, and analyze data from diverse systems.
Data governance and security
Databricks Unity Catalog offers security and compliance features to support data governance and adherence to organizational policies. Robust built-in security features and compliance capabilities ensure data security and regulatory compliance as data breaches and non-compliance can lead to severe financial and reputational consequences.
Practical example
The challenge
Navigating the complexities of a significantly depreciated data infrastructure poses a substantial hurdle, especially when embarking on a comprehensive modernization endeavor aimed at enhancing performance and optimizing the total cost of ownership.
The opportunity
Transitioning to a Databricks-centric architecture presented a pivotal opportunity for the customer to not only streamline operations but also achieve significant cost savings. By embracing this modern approach, the customer successfully slashed their total cost of ownership by more than 50%. This transformative shift involved strategically capitalizing on the modernization build-out expenses, effectively reducing operational expenditures from approximately $4 million to a lean $1.8 million. Leveraging the Databricks-centric architecture yielded a remarkable 4x price performance savings, underscoring the immense value derived from this forward-looking investment. Moreover, by spreading out the costs over a five-year period through amortization, the customer not only realized immediate benefits but also ensured sustained financial viability and resilience in the face of evolving technological landscapes.
Data migration TCO customer example
Current state
$4m
Annual OpEx
$0k
CapEx
Future state
$1.8m
Annual OpEx
$1m
CapEx
Total savings
$2.2m
(55%)
Annual OpEx
Data sources retired or reduced
Teradata
Hadoop
Snowflake
Conclusion
Companies need to take advantage of the upcoming economic recovery or else they will miss another economic growth cycle which will prove detrimental to the short-term growth of their business and the long-term growth of their industries. A Databricks-powered architecture offers the most compelling value proposition for the TCO of a data platform due to its cloud-native, scalable, and managed approach. Databricks empowers companies to achieve a low TCO by optimizing resource utilization, streamlining data workflows, and providing cost-effective, cloud-native solutions, ultimately delivering a robust and cost-efficient data platform. Every company should modernize now.
