Sovereign AI as an Infrastructure Issue: Why Open Source Decides on Sovereignty

1 June 2026

4 min read

Artificial Intelligence is increasingly becoming a strategic infrastructure for businesses. In conversation with Alexander Hendorf, AI consultant and open-source expert, one thing becomes clear: to use AI confidently, you need more than just models and tools. Control, operational capability, and technical know-how are what really matter.

Key Takeaways

Confident AI is an infrastructure issue: What matters is not where the models are located, but whether companies understand and control their architecture.
Open source shifts responsibility in-house: Models and frameworks are available, but the difference lies in operational and integration expertise.
Smaller on-premise setups often outperform hyperscalers: In the financial sector, two consumer GPUs for a clearly defined use case delivered faster, cheaper, and with more control than a cloud solution.
AI agents demand order: Poorly documented APIs and sprawling tool landscapes become bottlenecks once agents trigger processes autonomously.

Confident AI as infrastructure, not a model question

What is confident AI? Confident AI refers to a company’s ability to independently build, operate, and integrate AI systems into its own value-creation processes without becoming technologically, regulatorily, or commercially dependent on individual platform providers. At its core, the concept combines data control, operational capability, and internal expertise into an infrastructure challenge.

Many companies are currently debating confident AI, European models, and the use of open source. Yet the discussion is no longer limited to individual applications or chatbots. AI is increasingly embedded in business processes, data platforms, and cloud architectures. That is precisely why the topic is evolving into an infrastructure question.

Even at professional events like PyCon DE and PyData, this shift is evident. The focus is no longer solely on models and frameworks, but on topics such as AI agents, API standards, data architectures, and software engineering.

Cloud or on-premise is the wrong question

The discussion around confident AI is often oversimplified. Frequently, it seems companies must choose between cloud and on-premise as a fundamental decision. According to Hendorf, that’s too narrow a view.

What truly matters is whether companies can understand and control their infrastructure—regardless of where it runs. Relying exclusively on hyperscalers and SaaS providers quickly leads to complex dependencies. At the same time, local operation only makes sense if the necessary expertise exists in-house.

“The software isn’t the asset,” says Hendorf. “The real challenge is the ability to independently build, integrate, and securely operate systems.”

“Not every use case needs a Porsche—sometimes a kick scooter will do.”
Alexander Hendorf, AI consultant and open-source expert

Open source is changing the playing field. Models, frameworks, and infrastructure components are widely available today. Companies can adapt systems, run them locally, and exert greater control. Precisely this increases responsibility within their own walls.

Why smaller solutions often make more sense

Close-up: hands of an IT admin at an on-premise server with two consumer GPUs — On-premise instead of hyperscaler: two consumer GPUs suffice for clearly defined use cases.

How relevant this question has become is shown by a real-world example from the financial sector. An asset manager wanted to train its own AI models and initially considered a cloud solution. In the highly regulated environment, however, governance and compliance processes would have triggered months of coordination.

After analyzing the actual use case, the decision went in a different direction. Instead of a large generic LLM, a much smaller on-premise solution emerged with its own server and two Nvidia consumer GPUs in a secured network.

The result: faster implementation, lower costs, and significantly more control over data and infrastructure.

For Hendorf, the example highlights a fundamental issue with many AI projects. Companies often aim for maximum scalability even when their requirements are far more specialized: “Not every use case needs a Porsche; sometimes a kick scooter is enough.” Smaller models can be more efficient, cheaper, and easier to control for clearly defined tasks than large universal systems.

AI agents intensify the pressure for order

With the rise of AI agents, this trend is accelerating. Agents access data, use APIs, and automate processes. To do so, they require structured technical environments.

Historically grown silo solutions, poorly documented interfaces, and complex tool landscapes are becoming an increasing problem. According to Hendorf, AI agents perform significantly better with standardized APIs and consistent data structures.

This forces companies to reassess their infrastructure. Open source can help because systems become more transparent and flexible. Without clean architecture and solid software engineering, however, new complexity can quickly arise.

Security doesn’t come automatically

Hendorf also sees many misconceptions around security and data protection. Open source isn’t automatically secure. At the same time, a local infrastructure isn’t inherently riskier than complex cloud landscapes with opaque data flows.

For sensitive data, secured internal networks can offer advantages. Ultimately, access control, governance, and technical architecture remain decisive.

This shift also changes the role of IT and security teams. They must enable innovation while ensuring companies retain control over data, models, and processes.

Infrastructure becomes a competitive advantage

The debate around open-source AI is therefore far more than a model question. Companies must decide how dependent they want to be on external platforms and which technical expertise they need to build in-house.

Cloud platforms and proprietary models will continue to play an important role. At the same time, the importance of in-house operational capability is growing. This, Hendorf argues, is the real foundation for digital sovereignty.

Or, to put it another way: to leverage AI strategically in the long term, you need not only access to models but control over the infrastructure behind them.

Frequently Asked Questions

What sets sovereign AI apart from traditional cloud-based AI?

With sovereign AI, the company retains control over models, data, and operations. Location alone doesn’t determine sovereignty: even a cloud solution can be sovereign if configuration, key management, and governance fall under your own responsibility.

Why is a small on-premises setup often sufficient?

Many use cases don’t require a generic large-scale model. Specialized, smaller models run reliably on just a few consumer GPUs and often deliver better, more transparent results for specific tasks than universal LLMs.

What role do AI agents play in infrastructure strategy?

AI agents actively access data and systems. They only operate reliably when interfaces are well-documented and data models are consistent—making architecture and API hygiene a strategic priority.