AI stack attack: Navigating the generative tech maze

In mere months, the generative AI technology stack has undergone a striking metamorphosis. Menlo Ventures’ January 2024 market map depicted a tidy four-layer framework. By late May, Sapphire Ventures’ visualization exploded into a labyrinth of more than 200 companies spread across multiple categories. This rapid expansion lays bare the breakneck pace of innovation—and the mounting challenges facing IT decision-makers.

Technical considerations collide with a minefield of strategic concerns. Data privacy looms large, as does the specter of impending AI regulations. Talent shortages add another wrinkle, forcing companies to balance in-house development against outsourced expertise. Meanwhile, the pressure to innovate clashes with the imperative to control costs.

In this high-stakes game of technological Tetris, adaptability emerges as the ultimate trump card. Today’s state-of-the-art solution may be rendered obsolete by tomorrow’s breakthrough. IT decision-makers must craft a vision flexible enough to evolve alongside this dynamic landscape, all while delivering tangible value to their organizations.

Credit: Sapphire Ventures

The push towards end-to-end solutions

As enterprises grapple with the complexities of generative AI, many are gravitating towards comprehensive, end-to-end solutions. This shift reflects a desire to simplify AI infrastructure and streamline operations in an increasingly convoluted tech landscape.

When faced with the challenge of integrating generative AI across its vast ecosystem, Intuit stood at a crossroads. The company could have tasked its thousands of developers to build AI experiences using existing platform capabilities. Instead, it chose a more ambitious path: creating GenOS, a comprehensive generative AI operating system.

This decision, as Ashok Srivastava, Intuit’s Chief Data Officer, explains, was driven by a desire to accelerate innovation while maintaining consistency. “We’re going to build a layer that abstracts away the complexity of the platform so that you can build specific generative AI experiences fast.” 

This approach, Srivastava argues, allows for rapid scaling and operational efficiency. It’s a stark contrast to the alternative of having individual teams build bespoke solutions, which he warns could lead to “high complexity, low velocity and tech debt.”

Similarly, Databricks has recently expanded its AI deployment capabilities, introducing new features that aim to simplify the model serving process. The company’s Model Serving and Feature Serving tools represent a push towards a more integrated AI infrastructure.

These new offerings allow data scientists to deploy models with reduced engineering support, potentially streamlining the path from development to production. Marvelous MLOps author Maria Vechtomova notes the industry-wide need for such simplification: “Machine learning teams should aim to simplify the architecture and minimize the amount of tools they use.”

Databricks’ platform now supports various serving architectures, including batch prediction, real-time synchronous serving, and asynchronous tasks. This range of options caters to different use cases, from e-commerce recommendations to fraud detection.

Craig Wiley, Databricks’ Senior Director of Product for AI/ML, describes the company’s goal as providing “a truly complete end-to-end data and AI stack.” While ambitious, this statement aligns with the broader industry trend towards more comprehensive AI solutions.

However, not all industry players advocate for a single-vendor approach. Red Hat’s Steven Huels, General Manager of the AI Business Unit, offers a contrasting perspective: “There’s no one vendor that you get it all from anymore.” Red Hat instead focuses on complementary solutions that can integrate with a variety of existing systems.

The push towards end-to-end solutions marks a maturation of the generative AI landscape. As the technology becomes more established, enterprises are looking beyond piecemeal approaches to find ways to scale their AI initiatives efficiently and effectively.

Data quality and governance take center stage

As generative AI applications proliferate in enterprise settings, data quality and governance have surged to the forefront of concerns. The effectiveness and reliability of AI models hinge on the quality of their training data, making robust data management critical.

This focus on data extends beyond just preparation. Governance—ensuring data is used ethically, securely and in compliance with regulations—has become a top priority. “I think you’re going to start to see a big push on the governance side,” predicts Red Hat’s Huels. He anticipates this trend will accelerate as AI systems increasingly influence critical business decisions.

Databricks has built governance into the core of its platform. Wiley described it as “one continuous lineage system and one continuous governance system all the way from your data ingestion, all the way through your generative AI prompts and responses.”

The rise of semantic layers and data fabrics

As quality data sources become more important, semantic layers and data fabrics are gaining prominence. These technologies form the backbone of a more intelligent, flexible data infrastructure. They enable AI systems to better comprehend and leverage enterprise data, opening doors to new possibilities.

Illumex, a startup in this space, has developed what its CEO Inna Tokarev Sela dubs a “semantic data fabric.” “The data fabric has a texture,” she explains. “This texture is created automatically, not in a pre-built manner.” Such an approach paves the way for more dynamic, context-aware data interactions. It could significantly boost AI system capabilities.

Larger enterprises are taking note. Intuit, for instance, has embraced a product-oriented approach to data management. “We think about data as a product that must meet certain very high standards,” says Srivastava. These standards span quality, performance, and operations.

This shift towards semantic layers and data fabrics signals a new era in data infrastructure. It promises to enhance AI systems’ ability to understand and use enterprise data effectively. New capabilities and use cases may emerge as a result.

Yet, implementing these technologies is no small feat. It demands substantial investment in both technology and expertise. Organizations must carefully consider how these new layers will mesh with their existing data infrastructure and AI initiatives.

Specialized solutions in a consolidated landscape

The AI market is witnessing an interesting paradox. While end-to-end platforms are on the rise, specialized solutions addressing specific aspects of the AI stack continue to emerge. These niche offerings often tackle complex challenges that broader platforms may overlook.

Illumex stands out with its focus on creating a generative semantic fabric. Tokarev Sela said, “We create a category of solutions which doesn’t exist yet.” Their approach aims to bridge the gap between data and business logic, addressing a key pain point in AI implementations.

These specialized solutions aren’t necessarily competing with the consolidation trend. Often, they complement broader platforms, filling gaps or enhancing specific capabilities. Many end-to-end solution providers are forging partnerships with specialized firms or acquiring them outright to bolster their offerings.

The persistent emergence of specialized solutions indicates that innovation in addressing specific AI challenges remains vibrant. This trend persists even as the market consolidates around a few major platforms. For IT decision-makers, the task is clear: carefully evaluate where specialized tools might offer significant advantages over more generalized solutions.

Balancing open-source and proprietary solutions

The generative AI landscape continues to see a dynamic interplay between open-source and proprietary solutions. Enterprises must carefully navigate this terrain, weighing the benefits and drawbacks of each approach.

Red Hat, a longtime leader in enterprise open-source solutions, recently revealed its entry into the generative AI space. The company’s Red Hat Enterprise Linux (RHEL) AI offering aims to democratize access to large language models while maintaining a commitment to open-source principles.

RHEL AI combines several key components, as Tushar Katarki, Senior Director of Product Management for OpenShift Core Platform, explains: “We are introducing both English language models for now, as well as code models. So obviously, we think both are needed in this AI world.” This approach includes the Granite family of open source-licensed LLMs [large language models], InstructLab for model alignment and a bootable image of RHEL with popular AI libraries.

However, open-source solutions often require significant in-house expertise to implement and maintain effectively. This can be a challenge for organizations facing talent shortages or those looking to move quickly.

Proprietary solutions, on the other hand, often provide more integrated and supported experiences. Databricks, while supporting open-source models, has focused on creating a cohesive ecosystem around its proprietary platform. “If our customers want to use models, for example, that we don’t have access to, we actually govern those models for them,” explains Wiley, referring to their ability to integrate and manage various AI models within their system.

The ideal balance between open-source and proprietary solutions will vary depending on an organization’s specific needs, resources and risk tolerance. As the AI landscape evolves, the ability to effectively integrate and manage both types of solutions may become a key competitive advantage.

Integration with existing enterprise systems

A critical challenge for many enterprises adopting generative AI is integrating these new capabilities with existing systems and processes. This integration is essential for deriving real business value from AI investments.

Successful integration often depends on having a solid foundation of data and processing capabilities. “Do you have a real-time system? Do you have stream processing? Do you have batch processing capabilities?” asks Intuit’s Srivastava. These underlying systems form the backbone upon which advanced AI capabilities can be built.

For many organizations, the challenge lies in connecting AI systems with diverse and often siloed data sources. Illumex has focused on this problem, developing solutions that can work with existing data infrastructures. “We can actually connect to the data where it is. We don’t need them to move that data,” explains Tokarev Sela. This approach allows enterprises to leverage their existing data assets without requiring extensive restructuring.

Integration challenges extend beyond just data connectivity. Organizations must also consider how AI will interact with existing business processes and decision-making frameworks. Intuit’s approach of building a comprehensive GenOS system demonstrates one way of tackling this challenge, creating a unified platform that can interface with various business functions.

Security integration is another crucial consideration. As AI systems often deal with sensitive data and make important decisions, they must be incorporated into existing security frameworks and comply with organizational policies and regulatory requirements.

The radical future of generative computing

As we’ve explored the rapidly evolving generative AI tech stack, from end-to-end solutions to specialized tools, from data fabrics to governance frameworks, it’s clear that we’re witnessing a transformative moment in enterprise technology. Yet, even these sweeping changes may only be the beginning.

Andrej Karpathy, a prominent figure in AI research, recently painted a picture of an even more radical future. He envisions a “100% Fully Software 2.0 computer” where a single neural network replaces all classical software. In this paradigm, device inputs like audio, video and touch would feed directly into the neural net, with outputs displayed as audio/video on speakers and screens.

This concept pushes beyond our current understanding of operating systems, frameworks and even the distinctions between different types of software. It suggests a future where the boundaries between applications blur and the entire computing experience is mediated by a unified AI system.

While such a vision may seem distant, it underscores the potential for generative AI to reshape not just individual applications or business processes, but the fundamental nature of computing itself. 

The choices made today in building AI infrastructure will lay the groundwork for future innovations. Flexibility, scalability and a willingness to embrace paradigm shifts will be crucial. Whether we’re talking about end-to-end platforms, specialized AI tools, or the potential for AI-driven computing environments, the key to success lies in cultivating adaptability.

Learn more about navigating the tech maze at VentureBeat Transform this week in San Francisco.