Recent LLMs for local Estim AI

Ainavirin

The speed at which new AI models evolve is insane.

The list of suggested LLM models on the guide (https://eu.daimonia.app/articles/estimAI-start) is getting quite old.
Do you have any recent model suggestions for 16GB VRAM ?

Also, do you plan to support Ollama? It seems to be used by more and more sites for its ease of use...

KairosJS

I'm not planning to support Ollama, as I think KoboldCpp is light and easy to use. But I will keep an eye on it to check if the integration can be done quickly. Last time I checked (a few years ago), their API was different from other software.

I haven't tested local models for quite some time, but the recently released base models to look for abliteration/finetunes are:

Gemma 4 (12B for smaller GPUs)
Qwen 3.5 & Qwen 3.6 (there is a Qwen 3.5 9B for smaller GPUs)

Otherwise, it's probably still going to be some Mistral finetunes.

yngmstr

Hi,
where can we find these finetunes?

KairosJS

yngmstr I've answered to a similar request here:
https://space.daimonia.app/d/21-ai-model-comparison-thread/12

estimer

how can i connect a local llm to eudaimonia?

KairosJS

estimer You can find a tutorial here:
https://eu.daimonia.app/articles/estimAI-start

It's outdated, but the koboldcpp installation is the same.

frederickzweiten

I've been trying to get a local llm to work again with koboldcpp. Followed the instructions, worked with Claude to try to isolate the problem. Here's what claude says. It might be right. The errors seem to be happening, and I'm not sure why in any case. Any suggested settings I need to change is appreciated.

Hi,

I've been trying to use the app with a local KoboldCpp instance. After extensive debugging with Chrome DevTools, I've identified what appears to be a response format mismatch that prevents the integration from working.

The problem:

When using the hosted (subscription remote) model, the app correctly parses responses from your internal API, which returns:

output.text (the generated content)

However, KoboldCpp's OpenAI-compatible endpoint returns the standard OpenAI chat completion format:

choices[0].message.content

The app's parser at SessionModule (0h2kpmx5rqptu.js:173) attempts to read .content from the response, but because the structure is different from what it expects, it crashes with:

Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'content')

Evidence:

KoboldCpp returns HTTP 200 with a valid, well-formed response — the generation itself is working.
The hosted model works end-to-end without errors.
Comparing the two response payloads shows the structural difference clearly — your hosted API wraps the output in an output.text field, while KoboldCpp uses the OpenAI choices[] array.
The crash occurs in the client-side JavaScript response handler, not in KoboldCpp itself.

My setup:

KoboldCpp latest release (well, fairly recent)
Model: Meta-Llama-3.1-8B-Instruct-Q4_K_M (also tried some Mistral versions that didn't do quite as well)
RTX 4060 8GB
Prompt template: Llama-3 (or mistral when using that model)
The full system prompt is being sent correctly (7168 prompt tokens confirmed)

Request:
(FZ: maybe this is the right request - or maybe there is some setting changes I need to try. Interested if anybody else ran into this and of course, if they managed to solve it)

Could you add a response format adapter for the OpenAI-compatible endpoint, so that when the app is pointed at a local KoboldCpp instance, it reads choices[0].message.content instead of output.text? Alternatively, documentation on the exact response format your parser expects would allow the community to write a proxy adapter.

Happy to share the full Network tab payloads from both the working hosted session and the failing KoboldCpp session if that helps.

frederickzweiten

More work with Claude -- I'm going to edit this.

I tried some of the models in the thread over here: https://space.daimonia.app/d/21-ai-model-comparison-thread/12

I think I'm getting good responses on the first request, but now seems to be failing on a secondary, FX request:

Here is Claude's write up. I'll add notes in [FZ XXX messages]

Hi again,

There are actually two separate bugs, not one.

Bug 1 (previously reported): Response format mismatch
The main generation call returns OpenAI format (choices[0].message.content) but the parser expects your internal format (output.text). This causes the initial JavaScript crash. [FZ this seems better if I carefully set the Prompt Template and Parameters presets in the setup]

Bug 2 (new finding): FX Director sends empty messages array
This is the more critical bug. After the main generation succeeds, the FX Director makes a second API call to KoboldCpp to process the stimulation commands. That second call is sent with a completely empty messages array:

"messages": []

With no context, the model has nothing to work with and free-generates random content. This means the FX Director agent never gets a valid JSON block back, which is exactly what the error says:

No JSON block found in FX director output

Key evidence:

First call: 7168 prompt tokens, correct full system prompt, good structured output
Second call: 5 tokens, empty messages array, model hallucinates random content
This is reproducible across multiple models — I tested both Meta-Llama-3.1-8B-Instruct and Gemma4-12B, same result every time
The first call output is actually good — correct beat formatting, STATE XML, DELAY tags, phase transitions, inline JSON stim commands all present
The bug is entirely in how the second FX Director call is constructed for local endpoints

My setup:

KoboldCpp latest release, Ubuntu, RTX 4060 8GB
Tested with: Meta-Llama-3.1-8B-Instruct-Q4_K_M and Gemma4-12B-QAT-Q4_K_M
Both models running on GPU (confirmed via nvidia-smi)
GPU inference speed: ₂₇ T/s on Llama, ₁₅ T/s on Gemma 4

What I think is happening:
The FX Director call on your hosted backend presumably goes to Koboldcp and the dialoguea separate internal endpoint or has its own context injection path. When pointed at a local KoboldCpp instance, that context injection step is being skipped entirely, so the messages array arrives empty.

Request: [FZ: again, this is what Claude thinks. Hoping it helps someone as a bread crumb or gives someone an idea of how to fix]
The fix likely just needs the FX Director call to populate its messages array correctly when using a local endpoint, the same way the main generation call does. If you can share what the FX Director prompt is supposed to contain, I could potentially write a proxy layer to handle it locally while you work on a proper fix.

Can share full Network tab captures of both calls if useful. The local model integration is very close to working — the main generation quality is genuinely good.

KairosJS

frederickzweiten Hey, thanks for your report, I'm looking into it, and yes it seems that there is an issue currently when using local LLM and local agents.
I will try to fix this as soon as possible.

frederickzweiten

Thanks for checking K. I figured you were busy with the August release.

I will say, It looks like the update for August fixed this. Or possibly the tooltip in the setup: I had my context set to 8192 which was too low. Increased to something like 16k (when the tooltip indicated sufficient size to carry more context). Things seem to be working, even on my (slightly underpowered for this purpose) GPU.