Sovereign AI as an Infrastructure Issue: Why Open Source Decides on Sovereignty

1 June 2026

7 min. reading time

Artificial intelligence is becoming a strategic infrastructure issue for companies. In conversation with Alexander Hendorf, AI consultant and open-source expert, it becomes clear: if you want to use AI sovereignly, you need to understand, operate and control what happens within your own system. Open source is not an afterthought here, but a prerequisite for control, operational capability and long-term competitiveness.

Key takeaways

Sovereign AI is architectural work. What matters is not just the model, but the ability to control data flows, operations and transition paths.
Open source shifts responsibility. Models, frameworks and infrastructure are available, but companies must be able to assess quality, security and operations themselves.
AI agents expose technical debt. Without clean APIs, documented data models and in-house test environments, every model change becomes a shot in the dark.

The EU package bets on open source – yet the bottleneck lies elsewhere

On 3 June 2026, the European Commission presented its European Tech Sovereignty Package: a package comprising Chips Act 2.0, a Cloud and AI Development Act and its own open-source strategy. The planned Cloud and AI Development Act is set to define graduated sovereignty levels for sensitive workloads and expand European data-centre capacity. The open-source strategy explicitly treats open software as a lever to reduce dependence on third countries in cloud, AI and cybersecurity.

For Hendorf, the direction is right, but the focus is incomplete. Capacity alone does not create control. What counts is whether companies and public authorities can also understand, operate and audit the systems running on that infrastructure.

“Europe’s bottleneck is not merely models or data centres. Europe’s bottleneck is the ability to build AI systems under control, operate them productively, assess them securely and adapt them to real-world use cases – locally, vendor-neutrally and on the basis of open source. Only open models and open software let you truly penetrate, audit, adapt and run AI systems in-house. Proprietary black boxes cannot be controlled sovereignly.”

Alexander Hendorf, AI expert and head of the Open Source working group at the German AI Association, in his statement on the EU Tech Sovereignty Package

AI sovereignty starts with the infrastructure

What is AI sovereignty? AI sovereignty describes a company’s ability to independently select, integrate, operate, and audit AI systems. This includes control over data flows, models, infrastructure, quality, security, and migration paths. What matters is not the model’s name, but the operational capability behind it.

Many businesses are currently debating sovereign AI, European models like Mistral, and the use of open source. Yet the conversation has long since moved beyond individual applications or chatbots. AI is now deeply embedded in business processes, data platforms, and cloud architectures: from contract analysis and customer communication to internal knowledge search.

Even at leading AI conferences such as PyCon DE and PyData, the focus has shifted. Today’s discussions center not only on models and frameworks, but also on AI agents, API standards, data architectures, and software engineering. That’s precisely why the question is shifting from model selection to operational capability.

Cloud or on-premise is the wrong question

The debate around sovereign AI is often framed too narrowly. Many companies feel pressured to choose between cloud and on-premise as if it were a binary decision. According to Hendorf, that’s missing the point.

The real issue is whether companies truly understand and control their infrastructure – regardless of where it runs. Relying exclusively on hyperscalers and SaaS providers can quickly lead to dependencies: model versions change without notice, pricing and quotas are adjusted unilaterally, and data flows end up in regions that compliance teams can’t approve. At the same time, local operation only makes sense if the necessary expertise exists in-house.

“The software isn’t the asset – it’s everywhere available. Even the hyperscalers run on open source and actively promote it,” says Hendorf. “The real asset is operational capability: the ability to understand a system, set it up yourself, and switch between cloud and your own hardware when needed. That skill set is what’s missing in many organizations.”

What operational capability looks like in practice can be summed up by a term that’s standard in engineering circles but rarely heard in boardrooms: the Harness. It refers to an in-house test and evaluation environment where every AI model is systematically tested against your own use cases, data samples, and quality criteria before going live.

“The Harness is the real asset,” says Hendorf. “It’s the safeguard for model changes and upgrades. Without it, every switch becomes a leap of faith. You only find out weeks later whether the new model still does what the old one did. With it, switching becomes a routine technical decision.”

This gap is one of the biggest vulnerabilities when companies want to change providers or update a model. Without your own Harness, you’re forced to trust what release notes promise – and hope it works in production.

Open source is changing the playing field

Models, frameworks and infrastructure components are more widely available today than ever before: from open language models like Llama or Mistral to inference servers such as vLLM and Ollama, and vector databases for in-house knowledge search. This shifts the AI debate away from the narrow question of who has access to the most powerful model. What becomes more important is who can understand, adapt and reliably operate AI systems.

The gap to proprietary leaders is shrinking. Hendorf points to assessments from the PyCon community indicating that commercial models are only months ahead of open alternatives rather than years. For companies this is a meaningful shift: they can increasingly configure, run locally, integrate into their own processes and exert stronger control over AI systems. This new freedom, however, also increases responsibility within their own organisation.

Because open source does not automatically remove vendor lock-in; it merely relocates the requirements. To leverage open AI, businesses need technical know-how, clear architectural decisions, governance and the ability to realistically assess quality, security and operating costs.

This brings a question to the forefront that often gets lost in the hype around European or American models: what criteria should companies use to select AI systems at all? Decisive is not merely the origin of a model, but whether it matches the specific use case, existing data assets, regulatory obligations and in-house operational capability.

“Which of today’s widely available models a company adopts is decided not by origin, but by application,” says Hendorf. “Models from China, for example from DeepSeek or Qwen, are technically world-class, openly available and in many tasks equal to or ahead of Western open-source models. Whether their training data is politically filtered is not a decision criterion for the vast majority of enterprise use cases such as contract analysis, knowledge search or classification. Every model carries bias. The relevant question is not where it comes from, but whether you can control it in your own application.”

Why smaller solutions are often the smarter choice

How pertinent this question has become is illustrated by a real-world example from the financial sector. An asset manager wanted to train its own AI models and initially considered a cloud solution. In the highly regulated environment, governance and compliance processes would have taken twelve to twenty-four months. During that period many initiatives are shelved before ever going live.

After analysing the actual use case, the decision went another way: instead of a large generic LLM, the team built a much smaller on-premise solution with its own server and two Nvidia consumer GPUs inside a secured network. According to Hendorf and his team, they brought it into production in four weeks.

The result: the total solution cost less than six months of cloud operation. At the same time, staff could start working with the system immediately instead of waiting for protracted approvals.

For Hendorf, the example highlights a core problem of many AI projects. Companies often aim for maximum scalability even when their requirements are far more specialised: “Not every use case needs a Porsche; sometimes a kick scooter is enough.” Smaller models with only a few billion parameters can be more efficient, cheaper and easier to control for clearly defined tasks than large universal systems. And they run on hardware the company already owns.

AI agents intensify pressure on governance

This trend is accelerating with the rise of AI agents. Agents access data, leverage APIs, and automate workflows, but they require structured technical environments to do so.

Historically grown silos, poorly documented interfaces, and convoluted tool ecosystems are becoming liabilities. According to Hendorf, AI agents perform far better with standardized APIs and consistent data structures. New open standards such as the Model Context Protocol assume that underlying systems are cleanly documented and accessible. Where this foundation is missing, agents don’t fail because of the model – they fail because of the house architecture.

That forces companies to reassess their infrastructure. Open source can help by making systems more transparent and flexible, yet without sound architecture and solid software engineering, new complexity emerges quickly.

Security doesn’t come for free

Hendorf also sees widespread misconceptions around security and data privacy. Open source is not automatically secure. Downloading a model from a public hub without vetting its provenance imports the same supply-chain risk as any other software dependency. At the same time, a local infrastructure is not inherently riskier than sprawling cloud landscapes with opaque data flows.

For sensitive data, well-secured internal networks can offer advantages. Ultimately, access control, governance, and technical architecture remain decisive. That includes the ability to trace, if necessary, which inputs a model processed and when.

AI is therefore reshaping the role of IT and security teams. They must enable innovation while ensuring the company retains control over data, models, and processes.

Infrastructure becomes a competitive edge

The open-source AI debate is about far more than model choice. Companies must decide how dependent they want to be on external platforms and which technical and organizational knowledge they need to build in-house.

Cloud platforms and proprietary models will continue to play an important role. Yet the growing premium is on operational self-sufficiency. Hendorf sees this as the real foundation for digital sovereignty.

The regulatory backdrop adds urgency. The EU AI Act classifies certain AI applications – such as those in HR, credit scoring, or critical infrastructure – as high-risk and demands traceable data flows, documented model decisions, and auditable operations.

Put simply: to use AI strategically over the long term, you need not only access to models but control over the infrastructure behind them.

Three questions before every new AI initiative

Who decides on the model: us or a vendor whose product roadmap doesn’t align with our quarterly cycles?
Which part of our value chain would grind to a halt if that vendor changed its terms tomorrow?
Are our data, interfaces, and processes documented well enough for an agent or a new model to leverage them at all?

If you can’t answer these questions within a day, you don’t have a model problem – you have an architecture problem. That’s where the real work on sovereign AI begins.

Frequently Asked Questions

What distinguishes sovereign AI from traditional cloud-based AI?

Sovereign AI means a company retains control over models, data flows, quality, and operations. Location alone isn’t the deciding factor. Even a cloud solution can be sovereign if architecture, governance, and migration paths are mastered.

Why is a proprietary harness so important?

A harness tests models against your specific use cases and quality criteria. Without this testing environment, every model change carries risk, as deviations often only surface once the system is live.

Why do smaller on-premises setups often suffice?

Many enterprise use cases don’t require a generic large model. Specialized smaller models can be cheaper, faster, and easier to control for clearly defined tasks.

Sovereign AI as Infrastructure: Open Source Decides Sovereignty