The 2024 Databricks Data & AI Summit was the largest in history and the event has come a long way since we first attended in 2016. Databricks announced some impressive features and has made significant advancements in its Mosaic investment improving the cost and speed of training AI models exponentially. This year, they announced the acquisition of Tabular, which was a competitive M&A target by a number of players in the industry and gives Databricks the majority of the contributors to Apache Iceberg to its already talented team.
The Keynote presentations and the breakout sessions were impressive, but we learned most of our insights from the Executive Briefing Centers and peer-to-peer sessions that we hosted with our customers. We try to get as close as we can to the customer experience in practice because it’s easy to become enamored with new technology regardless of its practicality. Our customers tell us what’s actually useful and usable.
Governance is critical:
Most of our customers are at least considering Unity Catalog and many ask us why Databricks is making UC mandatory in order to unlock capabilities like GenAI and Serverless. Unity Catalog is less about the catalog itself and more about way to enforce metadata management maturity so that companies’ data can be considered trustworthy and bias can be prevented. UC is a great way to ensure that your organization is ready for those use cases. But no matter your choice of tech, prioritizing governance and enforcing it against the organization is essential to realizing the value out of any data use case.
Business Outcomes are More Important Than Data:
Data leadership must prioritize owning and driving business outcomes in order for new technology to be adopted. Business units hold the majority of the power and budget within an enterprise. Enterprise data organizations that fail to prioritize the needs of these business units risk becoming marginalized, especially if they attempt to impede the business units’ progress. It is crucial for data teams to align their goals with those of the business to maintain relevance and influence. Centralized technology strategies create shadow IT if the central IT org doesn’t move at least as fast as the business. Focus on improving business decision-making through tech.
Monitoring is Mandatory:
The biggest feedback we received in our sessions was that leaders couldn’t trust platforms like Databricks and Snowflake because they lacked end-to-end visibility into who and how the platform is being used. Consumption-based platforms terrify financial leaders, and one of the biggest barriers for customers to use the amazing features that Databricks offers is visibility and control over the platform. Databricks does an amazing job at providing transparency within its platform, but the feedback we got from customers is that their organizations are either structured in silos or they don’t have the cycles to get end-to-end visibility.
Leadership:
Ryan Neal
President & CEO
Overall, the 2024 Databricks Data & AI Summit was impressive and showcased that the innovation in our industry is higher than ever. It’s an arms race, and Databricks is maturing as an organization. They’re balancing pushing the bleeding edge of tech use cases and tech capabilities with the needs of the customer. They have invested heavily in the usability of the platform, and we are looking forward to the next year in data intelligence!
Gary Nakanelua
Managing Director of Technology
The biggest takeaway from the summit is Databricks’ commitment to setting the global standard for an organization’s data estate. They are achieving this by showcasing their expertise through open-source projects like ML Flow, Delta Sharing, Apache Spark, Unity Catalog (which was open sourced at DAIS 2024), and Delta Lake. This demonstrated expertise is continually integrated into their Data Intelligence platform, simplifying the complexities involved in maintaining a standardized data estate. This approach not only enhances efficiency but also reinforces Databricks’ leadership in the data and AI space.
Melissa Benton
Chief of Staff
Attending the Data and AI Summit in person was an inspiring opportunity, with 17,000 attendees underscoring the immense impact Databricks has among top global enterprises. Winning the Data and AI Governance Partner of the Year award among such prestigious partners is a testament to our commitment to excellence and customer success. The summit showcased the incredible innovation and collaboration within the data and AI community, reinforcing the value of being at the forefront of these advancements. Hearing Blueprint’s name recognized and respected among thousands of industry professionals was the ultimate validation of our hard work and dedication.
Heather Glynn
Managing Director of Solutions Delivery
My top takeaway from the 2024 Data and AI Summit is Databricks’ commitment to making data and AI accessible to everyone, demonstrated by open-sourcing Unity Catalog and launching Mosaic AI. These tools help organizations use data more effectively and securely, which is essential for powerful AI. For a systems integrator like us, leveraging these advanced tools allows us to streamline data management and enhance AI solutions’ accuracy and efficiency, providing our clients with more insightful analytics and data-driven strategies. Overall, Databricks’ innovations enable us to deliver even better outcomes for our clients.
Suzy Dolbow
Managing Director of Marketing
One thing that stood out to me is that despite the tremendous advancements in technology and the growing prevalence of AI, Databricks has built one of the best partner ecosystems. They seem to have partners who support every facet of the platform’s use cases, ensuring customers receive holistic support. This support ranges from leveraging the Databricks platform to collaborating with solutions integrators and ISV partners for their data and AI journeys. Being named the 2024 Data and AI Governance Partner of the Year was particularly special for Blueprint. About a third of the tech talks we heard in sessions emphasized that effective governance is essential for customers looking to dive into GenAI use cases.
Solutions Delivery:
Nick Carpender
Solutions Architect
This year’s DAIS was filled with engaging discussions with users and vendors about their journey with Databricks. A common theme in these conversations was the hurdles encountered during the maturity phase of their Databricks setup. Many found that their early deployments, often departmental or proof-of-concept in scope, rapidly expanded to serve broader enterprise needs due to the effectiveness of Databricks’ features and capabilities. However, this growth frequently outpaced planning for Azure infrastructure, network architecture, and medallion schema design, leading to underutilized features and suboptimal configurations. As a result, organizations are now navigating a phase of migration and standardization, aiming to address these oversights and fully harness Databricks’ potential.
Cori Hendon
Director of Solutions Delivery
There remains a significant interest in AI and its implementation on the Databricks platform. While data quality and governance may not be as thrilling, they are essential in today’s data landscape. It’s also evident that even the most technically impressive presentations could have benefited from the expertise of a Blueprint Builder, highlighting the exceptional strength and capabilities of our team in this area. This reaffirmed my confidence in our proficiency and the value we bring to the table.
Kristian Foster
Data Engineer Development Manager
My biggest takeaway from DAIS was our company’s position in the market for Unity Catalog and governance functionalities. Throughout the sessions and discussions, a recurring theme was the prevalence of manual methods within other systems. Our advantage lies in our ability to streamline and automate these processes effectively, offering solutions that not only enhance efficiency but also set a higher standard for operational excellence. This distinction underscores our commitment to innovation and proactive adaptation within our industry, positioning us prominently ahead in leveraging technology to drive forward-thinking solutions and sustainable growth.
Customer Success:
Michael Hallak
Director of Product
Databricks’ continual commitment to open source with the announcement of open Unity Catalog is a big cultural differentiator for the for the community. I also loved seeing so many people who I’ve worked with over the past 3 years, and getting feedback about how easy and valuable our Unity Catalog assessment capabilities are to run and use. I loved how real it was to watch users get hands on keyboard with our software in person; it’s surreal when we work in a digital environment and gratifying to see their positive feedback.
Marc Antunes
Account Executive
I had a lot of fun meeting with so many clients and Bricksters. The overarching message I heard was the same: clients are excited about their data & AI journey, and they’re excited to partner with Blueprint on that journey. In particular…good data governance continues to be a priority for existing Databricks customers and new. For existing clients, the name of the game is migrating to Unity Catalog, which many clients still find challenging. In any case, clients are excited about genAI use cases that will be enabled as a consequence of a well-governed data estate.
Chris Reid
Account Executive
The insights gained from the Data & AI Summit highlighted the significant challenges and immense value of migrating to Unity Catalog. It was apparent that Unity Catalog plays a pivotal role in driving clients’ business value and managing total cost of ownership (TCO), positioning it as a central component of the Databricks roadmap. This alignment underscores the strategic importance of integrating robust data management and governance solutions. Overall, the summit reinforced the critical need for innovative approaches to harness the full potential of data and AI.
