Open Source AI Is Eating the Stack (And That's Mostly Good News)

Mozilla just dropped their State of Open Source AI report and it's worth reading even if you skim it. Not because it tells you what's best. Because it shows you how fractured and fast-moving this space really is, which matters a lot if you're building products that depend on AI components staying stable.

I've been paying attention to where teams are actually placing their bets right now. Here's what I'm seeing across the projects I touch.

The Licensing Problem Is Real and Most Teams Are Ignoring It

When people say "open source AI" they usually mean one of three things: open weights, open training data, or open source code. These almost never show up together in the same model. LLaMA is a good example. Meta releases the weights but not the training data. Mistral releases weights but has commercial restrictions baked into certain versions. Falcon has a different license again.

For consumer apps this might not matter much. You call an API, you pay, you move on. But if you're in healthcare, fintech, or any regulated space, your legal team will eventually ask you exactly what model you're running, under what license, and whether any part of that model was trained on data that creates liability. Open source doesn't automatically mean free to use commercially. It sometimes means the opposite of what you assumed.

My practice now: whenever a model gets introduced into a project, I document the license, the version, and what parts of our system it touches. Boring but it saves pain later.

On-Device Inference Is Finally Practical for Narrow Tasks

This is the shift I'm most excited about. A year ago running any meaningful model on-device meant either a terrible UX or a bloated app binary. Today that's genuinely changing.

Apple's Core ML and Google's LiteRT (formerly TensorFlow Lite) have both matured. Phi-3 Mini from Microsoft runs on device across Android and iOS with acceptable latency for things like classification, intent detection, and short text summarization. Gemma 2B fits in memory on most current flagships.

What this unlocks practically: you can do a first pass of user intent or content classification without a network call. For something like a chat feature or a search filter, that means instant feedback instead of a spinner. For healthcare apps where sending user input to a cloud endpoint creates compliance questions, on-device inference is sometimes the only viable path.

The catch is still the same as it always was. You need to be honest about the task. On-device models are good at narrow, well-defined jobs. They struggle with reasoning chains, long context, or anything requiring up-to-date world knowledge. Use them for triage and fast feedback. Route complex tasks to a cloud model when you can afford the latency and the cost.

The Tooling Gap Is Closing But Deployment Still Needs Work

Mozilla's report is pretty clear that the inference and serving tooling around open models has improved dramatically. Ollama, llama.cpp, vLLM, and a handful of others have made self-hosting much more accessible than it was 18 months ago. You can run a capable model on a single GPU for experiments without burning a lot of money.

But production deployment is still where teams hit walls. The self-hosted path means you own monitoring, scaling, fallback logic, and version management. I've seen teams spin up a proof of concept on Ollama in a weekend and then spend three months trying to make it production-stable. The latency spikes under real load. The context window fills up in unexpected ways. The model drifts in behavior between versions.

If you're evaluating open models for production, treat it like adopting any critical infrastructure dependency. You need observability from day one. You need a rollback plan. You need to benchmark not just accuracy but tail latency, because the p99 is usually what breaks your UX.

One pattern I've found useful: run open models behind the same abstraction layer you'd use for any LLM provider. Keep the model selection swappable. If Mistral 7B starts misbehaving, you want to be able to route to a different model or a cloud fallback without touching product code.

What to Actually Evaluate When You're Picking a Model

The Mozilla report highlights something that I think builders underweight: benchmark performance does not predict product performance. A model that scores well on MMLU or similar academic benchmarks might be genuinely terrible at the specific thing your app needs.

Here's the evaluation process I use now:

1. Define the task sharply. Not "help users with support questions" but "classify a support message into one of 12 predefined categories with no hallucinated categories."

2. Build a test set from real production data. Synthetic prompts will lie to you. Real user inputs have typos, ambiguity, and edge cases that matter.

3. Measure what breaks the UX. For most mobile apps that means latency over 2 seconds, a refusal when the user input is harmless, or a confident wrong answer. Weight those failure modes heavily.

4. Test at the context lengths you'll actually use. Many models degrade significantly when you push them toward their context limit. If your feature involves chat history or long documents, test with those actual lengths.

5. Run evals before and after any model version change. Model providers update weights. Behavior changes. You want to catch regressions before your users do.

The open source space is moving faster than any team can fully track. The Mozilla report is useful because it gives you a framework for thinking about what you're actually choosing when you pick a model, not just a leaderboard ranking. That framing matters when you're building something that has to work reliably at 3am when nobody's watching.

Luma Commons works with product teams navigating exactly these decisions, from model selection to production deployment architecture.