Share it with your senior tech friends and colleagues
Reading Time: 4minutes
With over 300,000+ LLMs out there, how would you choose the right model for your Agentic AI application?
Ok, in reality, we don’t have to choose from 300,000+, since most models are just replicas with no unique features to offer, but 30+ models will definitely make your head spin?
And no, a larger model might not be useful for your case
And benchmarks would not help you either
Because benchmarks don’t consider your preferences and constraints.
When people talk about choosing models, they mainly talk about
OpenSource vs CloseSource
Small vs Large
Provider?
But that’s not really the decision you’re making.
You’re actually balancing
Cost
Confidential Information
Control
Connection (Internet)
Resource Constraint
Reasoning
Customisation
Caching (Built-in)
Let’s decode these parameters one by one
We have created an LLM Recommender. Just add your preference/ constraint and it will suggest the right LLM.
As a starting point, proprietary models are good, as they are easy to set up and have token-based pricing.
Once your workload becomes large and predictable, self-hosting an open-source model can become economically attractive.
So, for a rough estimate (not a thumb rule), if you are spending less than $1000 per month, then you could opt for proprietary models.
Data Confidentiality
With a self-hosted open-source model, you can keep inference data within infrastructure that you control.
With proprietary APIs, you are dependent on provider’s privacy and enterprise controls.
So, if keeping inference data within infrastructure you control is non-negotiable, self-hosted open-weight models are a better fit.
Control over deployment location
This is a very under-discussed parameter.
With Proprietary LLMs, you don’t have control over where the model is deployed, but with open source LLMs, you can choose the location.
How does this matter, you ask?
If your target audience is in the UK, deploying the model closer to them can reduce latency and may also help with data residency, compliance and sovereignty requirements.
Connection (Internet)
If your application must operate without internet access, your model needs to run locally. You don’t have much choice left.
And hence many hosted proprietary APIs are immediately eliminated from consideration.
You have to go with the model that can be deployed locally and can run in offline mode.
Resource Constraints
If you have compute constraints where you do not have enough RAM/ VRAM at your disposal, then you have to opt for smaller models or quantized versions of larger LLMs
Reasoning
We don’t have to use LLMs everywhere and for every task.
Bigger LLMs are generally better at complex reasoning tasks.
However, some tasks do not require complex reasoning. In that case, a smaller LLM may be sufficient and can reduce cost.
There are 2 more parameters that are not deal-breakers currently, but it’s good to know about them as well.
Customization
Remember fine-tuning?
Well, with large context windows, context engineering and harness engineering, fine tuning a model has very limited use cases.
But if you still have to fine-tune the model, check which models support it.
Open-weight models generally give you more control over the fine-tuning process, but several proprietary providers also offer fine-tuning.
Caching (Built-in)
Many proprietary LLM providers such as OpenAI, Google, etc. also provide built-in prompt caching.
Cached input tokens can be significantly cheaper than uncached input tokens.
For AI agents, caching can matter even more because system instructions, tool definitions and other context may be repeated across many model calls.
However, caching can be implemented externally by us at the application level as well. So, if you could do that, then the factor kind of becomes irrelevant.
Now what about the LLM provider?
We listed our preferences and came to the decision that we need a small open-source LLM.
But now the question is which provider should we go with – Meta or Google? How should we choose the provider?
Option 1 – Ask any good LLM
Provide all your constraints to a decent LLM and it will tell you which model will work best in your case.
Option 2 – Test on your task
Don’t just compare the models on generic benchmarks.
Once you know which type of LLM you need, compare all the options on your own task. Treat it like a sports tournament where each LLM will compete against each other.
You being the best judge, choose the best model on the parameters below:
Task success
Tool-call accuracy
Hallucination/error rate
Latency
Cost per successful task
Reliability
Caveat: Not all open-source models are truly open source
Open-source models are made available under various licenses.
Permissive licenses, such as Apache 2.0, give you broad freedom to use, modify, and distribute the model, including for commercial purposes.
Restricted licenses, such as CC BY-SA 3.0, impose additional conditions on how the model can be used or distributed.
They may allow commercial use but require you to comply with specific obligations, so check whether those terms fit your use case.
Non-commercial licenses, such as CC BY-NC-SA 4.0 or certain proprietary licenses, prohibit commercial use.
These licenses generally aren’t suitable if you plan to build and monetize applications using the model.
Conclusion
By now you must have realised that the best LLM isn’t necessarily the one with the highest benchmark score, the largest parameter count, or the most popular name.
It’s the one that works best for your application, your users and your constraints.
So, identify your constraints and then shortlist the models that satisfy those constraints.
We have created an LLM Recommender. Just add your preference/ constraint and it will suggest the right LLM.
If you have questions, feedback, or disagree with something in this article, I’d love to hear your perspective. Connect with me on LinkedIn: https://www.linkedin.com/in/nikhileshtayal/
Views:1
Share it with your senior tech friends and colleagues
