LLM selection
OpenAI, Claude or open source?
The right model depends on the actual use case. Quality, privacy, latency, cost, tool use and the operating model need to be evaluated together.
The most important rule: do not choose by brand name
A generic “winner” is not very useful in software projects. Models perform differently depending on the task, language, output format, tool use and operating conditions. Selection should therefore be based on representative test cases from the real process.
Which criteria matter?
- Quality for the specific task type
- Structured outputs and tool calling
- Context length and document handling
- Latency and availability
- Cost per request or processing step
- Data processing and contractual requirements
- Options for fallbacks or provider changes
Why your own data matters more than benchmarks
Public benchmarks only approximate your actual process. Representative tests with real inputs, expected outcomes and explicit quality criteria provide a stronger basis for selection.
Open source is also an operating decision
Self-hosted models can provide more control over data flows and infrastructure. At the same time, they create responsibilities for hardware, updates, scaling, access control and monitoring. The full operating model needs to be considered, not just the model licence.
Plan provider portability realistically
An abstraction layer can make switching easier. Models are still rarely completely interchangeable: prompting, structured output behaviour, tool calling, limits and failure modes differ. Critical functions should be tested with multiple candidates before promising a multi-provider strategy.
Decision matrix
| Question | Why it matters |
|---|---|
| Which task is being evaluated? | Quality is task-dependent. |
| Which data is processed? | It affects privacy and the operating model. |
| Which latency is acceptable? | It affects UX and architecture. |
| What is the request volume? | It drives ongoing cost. |
| Which tools or APIs must the model control? | It affects model and provider selection. |
