Adding an LLM Layer to a .NET Backend with Semantic Kernel
By Reliqio ·
How we add AI features to existing .NET backends: Semantic Kernel, RAG on Azure AI Search, tested prompts, logging and a few guardrails.
Almost every client with an existing .NET backend asks us the same thing sooner or later: "Can we add some AI to this?" Usually they mean summaries, better search or a support assistant. And usually someone on the team has already suggested a separate Python service for it.
You can do that. But in most of the projects we've worked on, it wasn't needed. Semantic Kernel fits into an ASP.NET Core app just fine, and you keep your existing pipeline, logging and tests. Here's roughly how we set it up.
Hide the model behind Semantic Kernel
Treat the LLM like any other external dependency, say a payment provider. Semantic Kernel already has an IChatCompletionService abstraction [1], so your code never touches a vendor SDK directly. Register it in DI, and moving from Azure OpenAI to another provider becomes a config change.
What that looks like for us: the kernel is registered once in Program.cs, every use case gets its own small service (TicketSummaryService, for example), and deployment name, temperature and token limits live in an options class. Controllers only know about the service. That also makes unit tests easy, because you can swap in a fake completion service.
Try RAG before you think about fine-tuning
Fine-tuning sounds impressive, but for business data RAG is almost always enough. We index documents in Azure AI Search with vector fields [2], pull the top few chunks for each request and pass them to the model. It's cheaper, you can see exactly what the model was given, and when a document changes the answers change the same day.
Some things that helped answer quality more than we expected:
- Split documents by headings and sections, not by a fixed number of characters.
- Keep tenant, language and date next to each chunk, and filter on them before ranking.
- Use hybrid search (keywords + vectors). On technical docs, pure vector search kept missing exact terms like error codes.
- Always return the sources with the answer. Users trust it more, and you'll spot bad retrieval faster.
Prompts are code, so test them
Prompt templates go into the repo next to the service that uses them. They get reviewed in pull requests like everything else. We also keep a small set of 20-50 questions with expected answers and run it in CI. If someone "improves" a prompt and the answers get worse, the build fails. This has saved us more than once.
Log every call
For every request we log prompt and completion tokens, latency, model version and the IDs of the documents that were retrieved. Semantic Kernel emits OpenTelemetry, so all of this shows up in Application Insights next to the rest of the traces. Without it you can't explain a wrong answer, and you have no idea what each feature actually costs.
Guardrails
- Validate structured output against a JSON schema before your code uses it.
- Run user input and model output through Azure AI Content Safety [3].
- Put per-tenant rate limits and token budgets in middleware, not in the code that calls the model.
Expect it to fail sometimes
LLM endpoints get throttled, time out and now and then return broken JSON. Wrap the calls with retries, timeouts and a circuit breaker (Polly or Microsoft.Extensions.Http.Resilience) and show the user a proper fallback instead of a 500. Streaming helps a lot too: people wait much more patiently once the first words appear.
Start small
What has worked for us:
- Pick one internal, low-risk feature first. Ticket summaries are a good candidate.
- Ship it behind a feature flag to a few users and ask for feedback on the answers.
- Look at traces and costs every week and adjust retrieval and prompts before adding anything new.
- Show it to customers only when the eval results and costs stop jumping around.
To sum up
Done this way, the AI part is just one more dependency in your architecture. You keep your .NET code, your deployment and your monitoring, and you can switch models when something better comes out.
This is the setup we use at Reliqio [4]. We work with founders as a technical partner on AI-powered B2C products in healthtech, fintech and e-commerce. If you've solved any of this differently, I'd like to hear how in the comments.
Sources
- ↑ Introduction to Semantic Kernel – Microsoft Learn (2026) · learn.microsoft.com
- ↑ Vector search in Azure AI Search – Microsoft Learn (2026) · learn.microsoft.com
- ↑ What is Azure AI Content Safety? – Microsoft Learn (2026) · learn.microsoft.com
- ↑ Reliqio - AI B2C product development partner – Reliqio (2026) · reliqio.com
FAQ
Do I need a separate Python service to add AI to a .NET app?
In most cases, no. Semantic Kernel and the Azure OpenAI SDK for .NET handle chat, embeddings and tool calling, so the AI part can live in your existing ASP.NET Core services.
Should I fine-tune a model or use RAG?
Start with RAG. It's cheaper, you can see what the model was given, and document changes show up right away. Fine-tuning makes sense mostly for tone or output format problems.
How do I keep LLM costs under control?
Log token usage per request and per feature, set token budgets per tenant in middleware, and cache answers to questions that come up again and again.