LLM support

LLM usage is optional and must be explicitly configured. You can maintain full control by running local models on your own systems through Ollama, LM Studio, or similar tools. No data is sent to external services unless you configure a cloud provider.

Supported providers

Sure supports two first-class LLM providers:

OpenAI (or any OpenAI-compatible endpoint, including Ollama, LM Studio, OpenRouter, Groq, Together AI, etc.)
Anthropic (native Claude support, including custom Anthropic-compatible endpoints such as Amazon Bedrock or Google Vertex)

Set the active provider with the LLM_PROVIDER environment variable (or the LLM Provider setting in the UI). It defaults to openai. The matching access token must be configured for the selected provider.

`LLM_PROVIDER`	Required credential
`openai` (default)	`OPENAI_ACCESS_TOKEN`
`anthropic`	`ANTHROPIC_ACCESS_TOKEN` (or `ANTHROPIC_API_KEY`)

Chat routing also honors the model name: requests with claude-* models are routed to the Anthropic provider, and other models to the OpenAI provider, when both are configured.

Configuration via settings

Configure AI settings through the web interface:

Go to Settings → Self-Hosting
Scroll to the AI Provider section

The AI Provider form contains the following fields: Provider & credentials

LLM Provider — openai or anthropic
Access Token — API key for the selected provider
URI Base / Base URL — Custom endpoint URL (optional; required for Ollama, LM Studio, Bedrock, Vertex, etc.)
Model — Model name (required for custom endpoints)

Behavior

JSON Mode — Output mode: auto, strict, json_object, or none
Chat Response Timeout — Whole-turn watchdog in seconds (see Chat response timeout)

Token Budget

Context Window — Total tokens the model accepts (default: 2048)
Max Response Tokens — Tokens reserved for the model’s reply (default: 512)
Max Items Per Batch — Upper bound for auto-categorize and merchant detection batches (default: 25)

Settings in the UI override environment variables.

Choosing a provider

Sure ships with two first-class LLM providers:

OpenAI (and OpenAI-compatible) — default. Works with OpenAI directly, Ollama, LM Studio, OpenRouter, Groq, and any other OpenAI-compatible endpoint.
Anthropic — native Claude support via the Messages API, including auto-categorization, merchant detection, and provider-merchant enhancement.

Select the active provider with the LLM_PROVIDER environment variable:

Variable	Description	Default
`LLM_PROVIDER`	Active provider. Set to `openai` or `anthropic`	`openai`

OpenAI compatible API

Sure supports any OpenAI-compatible API endpoint, giving you flexibility to use:

OpenAI - Direct access to GPT models
Ollama - Run models locally on your hardware
LM Studio - Local model hosting with a GUI
Google Gemini - Use Gemini models directly with a Gemini API key
OpenRouter - Access to multiple providers (Anthropic, Google, etc.)
Other providers - Groq, Together AI, Anyscale, Replicate, and more

OpenAI

OPENAI_ACCESS_TOKEN=sk-proj-...
# No other configuration needed

Recommended models:

gpt-4.1 - Default, best balance of speed and quality
gpt-5 - Latest model, highest quality
gpt-4o-mini - Cheaper, good quality

Ollama (local)

# Dummy token (Ollama doesn't need authentication)
OPENAI_ACCESS_TOKEN=ollama-local

# Ollama API endpoint
OPENAI_URI_BASE=http://localhost:11434/v1

# Model you pulled
OPENAI_MODEL=llama3.1:13b

Install and run Ollama:

# macOS
brew install ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

# Start Ollama
ollama serve

# Pull a model
ollama pull llama3.1:13b

LM Studio (local)

Download from lmstudio.ai
Download a model through the UI
Start the local server
Configure Sure:

OPENAI_ACCESS_TOKEN=lmstudio-local
OPENAI_URI_BASE=http://localhost:1234/v1
OPENAI_MODEL=your-model-name

Google Gemini

Gemini provides an OpenAI-compatible endpoint that works with a Gemini API key from Google AI Studio:

OPENAI_ACCESS_TOKEN=your-gemini-api-key
OPENAI_URI_BASE=https://generativelanguage.googleapis.com/v1beta/openai/
OPENAI_MODEL=gemini-3.6-flash

Recommended models:

gemini-3.6-flash - Fast and capable

OpenRouter

Access multiple providers through a single API:

OPENAI_ACCESS_TOKEN=your-openrouter-api-key
OPENAI_URI_BASE=https://openrouter.ai/api/v1
OPENAI_MODEL=google/gemini-2.0-flash-exp

Recommended models:

google/gemini-2.5-flash - Fast and capable
anthropic/claude-sonnet-4.5 - Excellent reasoning
anthropic/claude-haiku-4.5 - Fast and cost-effective

Anthropic

Sure can talk to Anthropic’s Messages API directly, with full chat (text + streaming + tool use) parity with the OpenAI provider.

LLM_PROVIDER=anthropic
ANTHROPIC_ACCESS_TOKEN=sk-ant-...
# Optional — defaults to claude-sonnet-4-6
ANTHROPIC_MODEL=claude-sonnet-4-6

Recommended models:

claude-sonnet-4-6 - Default, best balance of speed and quality
claude-haiku-4-5 - Fast and cost-effective

Custom Anthropic-compatible endpoints

Set ANTHROPIC_BASE_URL to route requests through Amazon Bedrock, Google Vertex, or any other Anthropic-compatible proxy. ANTHROPIC_MODEL is required when a custom base URL is set, because proxy providers use their own model-ID conventions (for example, Bedrock IDs like anthropic.claude-sonnet-4-5-20250929-v1:0).

LLM_PROVIDER=anthropic
ANTHROPIC_ACCESS_TOKEN=your-proxy-token
ANTHROPIC_BASE_URL=https://your-proxy.example.com
ANTHROPIC_MODEL=anthropic.claude-sonnet-4-5-20250929-v1:0

Anthropic environment variables

Variable	Description	Default
`ANTHROPIC_ACCESS_TOKEN` (or `ANTHROPIC_API_KEY`)	API key for Anthropic (or a compatible proxy)	—
`ANTHROPIC_MODEL`	Model name. Required when `ANTHROPIC_BASE_URL` is set	`claude-sonnet-4-6`
`ANTHROPIC_BASE_URL`	Custom endpoint URL (Bedrock, Vertex, or other proxy). Leave blank for `api.anthropic.com`	—
`ANTHROPIC_REQUEST_TIMEOUT`	HTTP timeout in seconds	`600`
`ANTHROPIC_MAX_TOKENS`	Maximum tokens returned by the model on each chat turn	`4096`

This first release of the Anthropic provider supports chat only (text, streaming, and tool use). Auto-categorization, merchant detection, PDF processing, and RAG features still require the OpenAI provider — calling them while LLM_PROVIDER=anthropic raises a clear “not yet implemented” error. Native Anthropic support for these features is being added in follow-up releases.

Batch operations

Auto-categorization, merchant detection, and provider-merchant enhancement are implemented on Anthropic via forced tool calls: a single output tool whose input_schema mirrors the desired result, invoked with tool_choice: { type: "tool", disable_parallel_tool_use: true }. This guarantees schema-valid output without JSON-mode fallbacks. Each batch operation caps the number of items at 25 per request, matching the OpenAI provider. Requests above this limit are rejected with a clear error before they are sent.

Token budget

Sure applies a token budget to every outbound LLM call — chat history, auto-categorization, merchant detection, and PDF processing. The defaults are conservative (2048-token context window) so small-context local models like Ollama work out of the box. If you use a cloud provider or a larger-context local model, raise these values.

Configure via settings UI

Go to Settings → Self-Hosting
Scroll to the AI Provider section
Under Token Budget, configure:
- Context Window — total tokens the model accepts (default: 2048)
- Max Response Tokens — tokens reserved for the model’s reply (default: 512)
- Max Items Per Batch — upper bound for auto-categorize and merchant detection batches (default: 25)

Configure via environment variables

Environment variables take precedence over the settings UI.

Variable	Description	Default
`LLM_CONTEXT_WINDOW`	Total tokens the model will accept	`2048`
`LLM_MAX_RESPONSE_TOKENS`	Tokens reserved for the model’s reply	`512`
`LLM_MAX_HISTORY_TOKENS`	Maximum tokens for conversation history. Derived automatically if unset (`context_window - max_response_tokens - system_prompt_reserve`)	Derived
`LLM_SYSTEM_PROMPT_RESERVE`	Tokens reserved for the system prompt	`256`
`LLM_MAX_ITEMS_PER_CALL`	Upper bound on auto-categorize / merchant detection batch size	`25`

Large batches of transactions are automatically sliced to fit the configured context window. You no longer need to worry about the previous 25-item hard limit — it is now a soft default that adapts to your model’s capacity.

Recommended values

Setup	Context window	Max response tokens	Max items per batch
Ollama / small local models	`2048` (default)	`512` (default)	`25` (default)
Cloud OpenAI (gpt-4.1, gpt-5)	`16384` or higher	`4096`	`50`
Large-context local models	`8192`	`2048`	`50`

Responses API routing

Sure automatically routes chat requests to the OpenAI Responses API when using the official OpenAI endpoint, and falls back to the Chat Completions API for custom providers. You can override this behavior with the OPENAI_SUPPORTS_RESPONSES_ENDPOINT environment variable.

Variable	Description	Default
`OPENAI_SUPPORTS_RESPONSES_ENDPOINT`	Set to `true` to force the Responses API on a custom provider, or `false` to force the Chat Completions API on all providers	Auto-detected

AI cache management

Sure caches AI-generated results (like auto-categorization and merchant detection) to avoid redundant API calls and costs.

What is the AI cache?

When AI rules process transactions, Sure stores:

Enrichment records - Which attributes were set by AI (category, merchant, etc.)
Attribute locks - Prevents rules from re-processing already-handled transactions

This caching means:

Transactions won’t be sent to the LLM repeatedly
Your API costs are minimized
Processing is faster on subsequent rule runs

When to reset the AI cache

You might want to reset the cache when:

Switching LLM models - Different models may produce better categorizations
Improving prompts - After system updates with better prompts
Fixing miscategorizations - When AI made systematic errors
Testing - During development or evaluation of AI features

Resetting the AI cache will cause all transactions to be re-processed by AI rules on the next run. This will incur API costs if using a cloud provider.

How to reset the AI cache

Via UI (recommended):

Go to Settings → Rules
Click the menu button (three dots)
Select Reset AI cache
Confirm the action

The cache is cleared asynchronously in the background. Automatic reset: The AI cache is automatically cleared for all users when the OpenAI model setting is changed. This ensures that the new model processes transactions fresh.

What happens when cache is reset

AI-locked attributes are unlocked - Transactions can be re-enriched
AI enrichment records are deleted - The history of AI changes is cleared
User edits are preserved - If you manually changed a category after AI set it, your change is kept

Evaluation system

Test and compare different LLMs for your specific use case. The eval system helps you benchmark models for transaction categorization, merchant detection, and chat assistant functionality. See the evaluation framework documentation for details on:

Running evaluations
Comparing models
Creating custom datasets
Langfuse integration for tracking experiments

Additional environment variables

These optional variables fine-tune provider selection and the behavior of the OpenAI-compatible provider. See Anthropic environment variables for the Anthropic-specific options.

Variable	Description	Default
`LLM_PROVIDER`	Active LLM provider: `openai` or `anthropic`	`openai`
`OPENAI_ACCESS_TOKEN`	API key for the provider	—
`OPENAI_URI_BASE`	Custom endpoint URL (leave blank for OpenAI)	—
`OPENAI_MODEL`	Model name (required for custom endpoints)	`gpt-4.1`
`OPENAI_REQUEST_TIMEOUT`	HTTP timeout in seconds for each individual HTTP call to the model	`60`
`AI_RESPONSE_TIMEOUT`	Whole-turn watchdog timeout in seconds. See Chat response timeout for sizing guidance	`90`
`ASSISTANT_MAX_TOOL_CALL_ITERATIONS`	Maximum number of tool-call rounds per chat turn	`8`
`OPENAI_SUPPORTS_PDF_PROCESSING`	Set to `false` for endpoints without vision support	`true`
`OPENAI_SUPPORTS_RESPONSES_ENDPOINT`	Override Responses API vs Chat Completions routing	Auto-detected
`LLM_JSON_MODE`	JSON output mode: `auto`, `strict`, `json_object`, or `none`	—

See Token budget for LLM_CONTEXT_WINDOW, LLM_MAX_RESPONSE_TOKENS, and related variables.

Chat response timeout

Sure uses two separate timeouts for AI chat turns:

OPENAI_REQUEST_TIMEOUT — bounds each individual HTTP call to the model. The default is 60 seconds.
AI_RESPONSE_TIMEOUT — a whole-turn watchdog that starts when the message is queued and covers everything: queue wait, every model call, and all tool executions in between. The default is 90 seconds, which is sized for typical cloud latency.

Why the default 90 seconds is often too short for local models

Custom OpenAI-compatible providers (Ollama, LM Studio, etc.) route through a synchronous code path, so nothing renders until the full reply is generated. On top of that, a tool-using turn runs multiple model calls in sequence. The total time the watchdog must cover is:

(1 + ASSISTANT_MAX_TOOL_CALL_ITERATIONS) × OPENAI_REQUEST_TIMEOUT
  + tool execution time + queue wait

With the defaults (ASSISTANT_MAX_TOOL_CALL_ITERATIONS=8, OPENAI_REQUEST_TIMEOUT=60), a worst-case turn takes up to 540 seconds of model time — far beyond the 90-second watchdog. The chat will report “assistant not available” even though the model is still generating a reply.

Recommended approach

The cheapest lever is lowering ASSISTANT_MAX_TOOL_CALL_ITERATIONS rather than setting a very large timeout. Fewer iterations means a shorter worst-case bound, and long tool chains fail with a clear limit error instead of a silent timeout. For a local Ollama setup with slow generation:

OPENAI_REQUEST_TIMEOUT=300
ASSISTANT_MAX_TOOL_CALL_ITERATIONS=2
# (1 + 2) × 300 = 900s model time + ~300s headroom = 1200
AI_RESPONSE_TIMEOUT=1200

When the tool-call round limit is reached, the assistant sends one final request with no tools available so the model can answer in prose with whatever it gathered. This means the conversation always produces a response rather than ending with an error banner.

For cloud providers (OpenAI, Anthropic) the defaults are usually fine. If you see timeouts, raise AI_RESPONSE_TIMEOUT using the formula above.

AI_RESPONSE_TIMEOUT must be sized as a sum, not just set above OPENAI_REQUEST_TIMEOUT. The watchdog covers the entire turn — queue wait, every model call, and all tool executions — so the bound is:

(1 + ASSISTANT_MAX_TOOL_CALL_ITERATIONS) × OPENAI_REQUEST_TIMEOUT
  + tool execution time + queue wait

Setting AI_RESPONSE_TIMEOUT equal to or below OPENAI_REQUEST_TIMEOUT means any tool-using turn will always time out. The 90-second default is sized for typical cloud latency; use the formula above once per-call latency approaches the timeout.

Configure via settings UI

AI_RESPONSE_TIMEOUT is also configurable from the web interface:

Go to Settings → Self-Hosting
Scroll to the AI Provider section
Set Chat Response Timeout to your desired value in seconds

The environment variable takes precedence over the UI setting. The minimum accepted value is 30 seconds.

Docker compose example

Basic Ollama setup

services:
  sure:
    environment:
      - OPENAI_ACCESS_TOKEN=ollama-local
      - OPENAI_URI_BASE=http://ollama:11434/v1
      - OPENAI_MODEL=llama3.1:13b
      # Optional: raise token budget for larger-context models
      # - LLM_CONTEXT_WINDOW=8192
      # - LLM_MAX_RESPONSE_TOKENS=2048
      # - LLM_MAX_ITEMS_PER_CALL=50
    depends_on:
      - ollama

  ollama:
    image: ollama/ollama:latest
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    # Uncomment if you have an NVIDIA GPU
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: 1
    #           capabilities: [gpu]

volumes:
  ollama_data:

Advanced AI setup with OpenClaw

For advanced AI features including code execution and tool use, you can use the local-ai profile with OpenClaw:

services:
  sure:
    profiles:
      - local-ai
    environment:
      - OPENAI_ACCESS_TOKEN=ollama-local
      - OPENAI_URI_BASE=http://openclaw:8080/v1
      - OPENAI_MODEL=llama3.1:13b
    depends_on:
      - openclaw
      - ollama

  openclaw:
    image: openclaw/openclaw:latest
    profiles:
      - local-ai
    ports:
      - "8080:8080"
    environment:
      - OLLAMA_HOST=http://ollama:11434
    depends_on:
      - ollama

  ollama:
    image: ollama/ollama:latest
    profiles:
      - local-ai
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama

volumes:
  ollama_data:

To use the local AI setup:

# Start with local AI profile
docker compose --profile local-ai up

# Or include it with other profiles
docker compose --profile local-ai --profile external-assistant up

The local-ai profile includes:

Ollama - Local LLM inference
OpenClaw - Gateway providing enhanced AI capabilities and tool use
Automatic routing between Sure and the AI stack

Getting started

Key concepts

App features

Development

Authentication

Third party providers

Knowledge Base

Supported providers

Configuration via settings

Choosing a provider

OpenAI compatible API

OpenAI

Ollama (local)

LM Studio (local)

Google Gemini

OpenRouter

Anthropic

Custom Anthropic-compatible endpoints

Anthropic environment variables

Batch operations

Token budget

Configure via settings UI

Configure via environment variables

Recommended values

Responses API routing

AI cache management

What is the AI cache?

When to reset the AI cache

How to reset the AI cache

What happens when cache is reset

Evaluation system

Additional environment variables

Chat response timeout

Why the default 90 seconds is often too short for local models

Recommended approach

Configure via settings UI

Docker compose example

Basic Ollama setup

Advanced AI setup with OpenClaw

​Supported providers

​Configuration via settings

​Choosing a provider

​OpenAI compatible API

​OpenAI

​Ollama (local)

​LM Studio (local)

​Google Gemini

​OpenRouter

​Anthropic

​Custom Anthropic-compatible endpoints

​Anthropic environment variables

​Batch operations

​Token budget

​Configure via settings UI

​Configure via environment variables

​Recommended values

​Responses API routing

​AI cache management

​What is the AI cache?

​When to reset the AI cache

​How to reset the AI cache

​What happens when cache is reset

​Evaluation system

​Additional environment variables

​Chat response timeout

​Why the default 90 seconds is often too short for local models

​Recommended approach

​Configure via settings UI

​Docker compose example

​Basic Ollama setup

​Advanced AI setup with OpenClaw

Supported providers

Configuration via settings

Choosing a provider

OpenAI compatible API

OpenAI

Ollama (local)

LM Studio (local)

Google Gemini

OpenRouter

Anthropic

Custom Anthropic-compatible endpoints

Anthropic environment variables

Batch operations

Token budget

Configure via settings UI

Configure via environment variables

Recommended values

Responses API routing

AI cache management

What is the AI cache?

When to reset the AI cache

How to reset the AI cache

What happens when cache is reset

Evaluation system

Additional environment variables

Chat response timeout

Why the default 90 seconds is often too short for local models

Recommended approach

Configure via settings UI

Docker compose example

Basic Ollama setup

Advanced AI setup with OpenClaw