A local model your tools can call

T11-L02 · Run your own AI · Level 2 Power user · 30-minute read

At this level, a model result can enter a team document or pipeline. A route change that looks private but sends data elsewhere, or a weaker result that passes unnoticed, can therefore affect colleagues as well as you.

2. The workflow must stay inside the building

You have a small workflow that sends text to a cloud API and returns structured output. In the Lab version, it extracts fields from literature notes before a colleague checks them. In the Company version, it classifies supplier messages before they enter a shared queue. The code works, but the next approved dataset must not leave the organisation's controlled machine or network.

You install a local model and receive a successful response from its chat screen. That does not yet help the workflow. The script needs an API address, and its existing cloud library expects a particular request and response shape. Even if the local endpoint accepts that shape, its output may differ enough to break the checks.

Your task is not merely to make the model answer. You will expose Ollama only on the workstation's loopback address, point the existing workflow at it through one endpoint profile, and rerun the same checks. Then you will decide whether local, an EU-hosted candidate, or an approved frontier endpoint is the right route for this task.

3. After this you can

• Expose an installed local model through a loopback API that a script can call.
• Switch an OpenAI-compatible workflow by changing its endpoint profile rather than rewriting its task logic.
• Verify structured output with the same checks used for the hosted route.
• Explain when a gateway keeps provider changes out of downstream tools.
• Choose a local, EU-hosted, or frontier route for one named task and data class.

4. Prerequisites

• T11-L01 · A model on your own laptop.
• An approved workstation with Ollama installed and one approved model already downloaded. Use the exact identifier shown by ollama list; this book deliberately does not recommend a model name.
• Python in an approved environment and the OpenAI Python package installed according to its current documentation.
• A real hosted test route approved for the chosen lane and synthetic fixture, plus its approved credential mechanism. You cannot complete the route-switch exercise without a passing hosted baseline.
• Permission to run a loopback service and change test-only environment variables.
• A text editor and terminal, plus 60 minutes for the independent exercise.

Use only the synthetic fixtures below. Do not send unpublished findings, participant or patient data, customer or supplier records, contracts, credentials, or confidential text merely because the endpoint is local. If you cannot identify the permitted data class, stop before making the request.

5. The idea in one page

An API endpoint is an address a tool calls. Ollama can accept an OpenAI-compatible chat request at http://127.0.0.1:11434/v1. For an existing compatible tool, the connection profile normally contains a base URL, API key reference, and model identifier. Change that profile, not the extraction or classification logic. Ollama's client key value is a required placeholder for this local compatibility path, not access control. Keep the service on loopback for this exercise; do not bind it to all interfaces or expose the port to a LAN.

Compatible means that selected request and response fields have a familiar shape. It does not mean identical model behavior, context limits, structured-output reliability, tool support, latency, or error handling. Change your behaviour: after any endpoint or model change, run the same task checks against fixed fixtures. A successful HTTP response is not a passing workflow.

A gateway such as LiteLLM can give clients one stable base URL and a route alias such as bounded-worker. The gateway maps that alias to Ollama or an approved hosted provider and keeps provider credentials on the gateway side. Downstream tools no longer need a separate provider edit. Change your behaviour: version the route mapping, restrict who can change it, record the previous mapping, and test every change. The gateway is another trust boundary: it can see requests, its logs may retain data, and a wrong route can affect every client.

Choose a route per task, not once for the whole organisation:

"EU-hosted" is not a complete boundary. It may describe inference while account data, diagnostics, support, or backups follow another path. IONOS, Scaleway, OVHcloud, and EUrouter are candidates to investigate, not pre-approved conclusions. Verify current primary documentation, the selected configuration, signed terms, and your organisation's decision. Likewise, "frontier" describes a capability choice, not a location or privacy guarantee.

For every route, draw workflow -> gateway if used -> endpoint -> logs/support, name the identity at each hop, and state what data may cross it. Keep keys in an approved secret mechanism, not source code or screenshots. Local inference narrows one data path; it does not settle device access, retrieval permissions, telemetry, backup, or retention.

6. The worked example: switch the route and keep the checks

The Lab and Company lanes use the same script, endpoint profile, and pass rule. Only the synthetic input, requested operation, and expected fields differ.

Run one unchanged checker

Save this as route_check.py in a temporary exercise folder. It asks for JSON, parses the response, and compares every field with expected fixture values. It does not trust the model's own claim that it passed.

import json
import os
import sys

from openai import OpenAI

LANES = {
    "lab": {
        "instruction": (
            "Extract the synthetic literature note. Return JSON only with keys "
            "record_id, material, temperature_c, and result. Use null when absent."
        ),
        "source": (
            "Synthetic literature fixture L-17. Record ID: L-17. Material: polymer film. "
            "The film was held at 18 C. Result: no visible change."
        ),
        "expected": {
            "record_id": "L-17",
            "material": "polymer film",
            "temperature_c": 18,
            "result": "no visible change",
        },
    },
    "company": {
        "instruction": (
            "Classify the synthetic supplier message. Return JSON only with keys "
            "record_id, label, and basis. Label must be ACTION or INFORMATION."
        ),
        "source": (
            "Synthetic supplier fixture C-17. Record ID: C-17. Delivery remains scheduled "
            "for Tuesday. No reply or approval is requested."
        ),
        "expected": {
            "record_id": "C-17",
            "label": "INFORMATION",
            "basis": "No reply or approval is requested.",
        },
    },
}

lane_name = os.environ.get("LANE", "lab")
if lane_name not in LANES:
    sys.exit("FAIL: LANE must be lab or company")

required = ["MODEL_BASE_URL", "MODEL_API_KEY", "MODEL_NAME"]
missing = [name for name in required if not os.environ.get(name)]
if missing:
    sys.exit("FAIL: missing " + ", ".join(missing))

lane = LANES[lane_name]
client = OpenAI(
    base_url=os.environ["MODEL_BASE_URL"],
    api_key=os.environ["MODEL_API_KEY"],
)
response = client.chat.completions.create(
    model=os.environ["MODEL_NAME"],
    temperature=0,
    response_format={"type": "json_object"},
    messages=[
        {"role": "system", "content": lane["instruction"]},
        {"role": "user", "content": lane["source"]},
    ],
)

try:
    actual = json.loads(response.choices[0].message.content)
except (json.JSONDecodeError, TypeError) as error:
    sys.exit(f"FAIL: response was not JSON: {error}")

if actual != lane["expected"]:
    print("FAIL: field check")
    print("Expected:", json.dumps(lane["expected"], sort_keys=True))
    print("Actual:  ", json.dumps(actual, sort_keys=True))
    sys.exit(1)

print(f"PASS: {lane_name} via {os.environ['MODEL_BASE_URL']}")

Record the required hosted baseline

Before starting or selecting the local route, load the real approved hosted test profile for your chosen lane. Set MODEL_BASE_URL, MODEL_API_KEY, and MODEL_NAME through your organisation's approved mechanism, set LANE to lab or company, and run python route_check.py. Do not put a credential value in the record.

Proceed only if the process exits with status 0. Record the approval reference, route label, non-secret base URL, model identifier, lane, fixture ID, checker identifier, command, timestamp, exit status, and complete PASS line. If no approved hosted profile exists, or the hosted check fails, stop and obtain approval or fix that route. There is no not available substitute: without this real passing baseline, you have not demonstrated a route switch.

Start and prove the local endpoint

Now confirm the downloaded identifier:

ollama list

If Ollama is not already serving requests, run ollama serve in a separate terminal. If it reports that the address is already in use, inspect the existing listener rather than starting a duplicate. Do not alter the listening address for this exercise.

A request succeeding at 127.0.0.1 does not prove that the same service is unavailable on other interfaces. Use the command for your operating system to inspect every TCP listener on port 11434. Each command fails if there is no listener or if any listener is bound to an address other than 127.0.0.1 or ::1.

Windows PowerShell:

$listeners = @(Get-NetTCPConnection -State Listen -LocalPort 11434 -ErrorAction Stop)
if ($listeners.Count -eq 0) { throw "FAIL: no listener on port 11434" }
$listeners | Format-Table LocalAddress, LocalPort, OwningProcess
$outsideLoopback = @($listeners | Where-Object { $_.LocalAddress -notin @("127.0.0.1", "::1") })
if ($outsideLoopback.Count -gt 0) { throw "FAIL: port 11434 is listening beyond loopback" }
"PASS: every port 11434 listener is loopback-only"

Linux:

ss -H -ltn 'sport = :11434' | awk '
BEGIN { seen=0; bad=0 }
{ seen=1; if ($4 != "127.0.0.1:11434" && $4 != "[::1]:11434") { print "FAIL: non-loopback listener " $4; bad=1 } }
END { if (!seen) { print "FAIL: no listener on port 11434"; exit 1 }; if (bad) exit 1; print "PASS: every port 11434 listener is loopback-only" }'

macOS:

listeners="$(lsof -nP -a -iTCP:11434 -sTCP:LISTEN -Fn 2>/dev/null | sed -n 's/^n//p')"
[ -n "$listeners" ] || { echo "FAIL: no listener on port 11434"; exit 1; }
for address in $listeners; do
  case "$address" in
    "127.0.0.1:11434"|"[::1]:11434") ;;
    *) echo "FAIL: non-loopback listener $address"; exit 1 ;;
  esac
done
echo "PASS: every port 11434 listener is loopback-only"

If the check reports 0.0.0.0, ::, a LAN address, or any other non-loopback address, stop Ollama and restore its approved loopback configuration before continuing. Do not use an external port scanner or weaken a firewall to perform this check. Retain the command and output as evidence.

Only after the hosted baseline and listener check both pass, replace the placeholder below with the exact identifier from ollama list and make the direct synthetic test:

ollama run <EXACT_DOWNLOADED_MODEL_ID> "Reply with exactly: MODEL READY"

Then replace the hosted connection profile with the local profile:

MODEL_BASE_URL=http://127.0.0.1:11434/v1
MODEL_API_KEY=ollama
MODEL_NAME=<EXACT_DOWNLOADED_MODEL_ID>
LANE=lab

Use your shell's normal environment-variable syntax and keep the same LANE used for the baseline. The three endpoint values form one connection profile. An existing GUI tool with fields for an OpenAI-compatible provider uses the same mapping: base URL, placeholder key, and exact model identifier.

Lab framing: extraction stays exact

Set LANE=lab and run python route_check.py. The literature-shaped fixture contains no real paper, sample, or unpublished result. The test passes only when the local model preserves all four fields, including the number and the negative result. If it changes 18 or turns "no visible change" into a stronger conclusion, record failure.

The recorded hosted run is the baseline. Replace only its endpoint profile with the local values; do not edit the fixture, prompt, expected object, lane, or checker between runs. This isolates the route change.

Company framing: classification keeps its basis

Set LANE=company and run the same command. The supplier-shaped fixture is fictional. The test passes only when the route returns INFORMATION and preserves the sentence explaining why. An invented request for approval must fail the comparison even if the label happens to be correct.

The two lanes have parallel stakes: a wrong Lab extraction can enter a literature table; a wrong Company classification can move a supplier message into the wrong shared queue. In either lane, teammates inherit the error if you accept a response merely because the endpoint returned status success.

Put a stable alias in front later

When several tools need the route, configure a reviewed LiteLLM proxy alias instead of editing each client. A minimal local mapping has this shape; check the current primary documentation before running it:

model_list:
  - model_name: bounded-worker
    litellm_params:
      model: ollama_chat/<EXACT_DOWNLOADED_MODEL_ID>
      api_base: http://127.0.0.1:11434

Clients then use the proxy base URL and MODEL_NAME=bounded-worker. To change provider later, an owner updates the alias mapping, uses a server-side secret reference for any hosted credential, reruns both checks, and retains the prior configuration as rollback. This loopback example is not a shared production gateway: shared access requires authentication, authorization, TLS, scoped client keys, request limits, protected logs, and an operator.

7. What goes wrong

The route changes but the checks do not run

Symptom: the local endpoint answers successfully, so its output enters the shared pipeline without comparison to the fixed fixture.

Fix: run the exact pre-switch checker after every endpoint, model, prompt, or gateway-mapping change. Keep failure visible and restore the prior profile if the check does not pass.

Compatible is mistaken for identical

Symptom: a request field is ignored, JSON differs, or an error is shaped differently even though both endpoints are called OpenAI-compatible.

Fix: test only the fields your workflow uses. Parse and validate the response, exercise failure handling, and remove unsupported assumptions rather than hiding them with retries.

The local port is exposed to other machines

Symptom: the service listens beyond loopback or a firewall rule permits unauthenticated network callers.

Fix: stop it and restore loopback-only access. Do not make this Level 2 exercise a team service; authenticated shared deployment belongs to later books and an approved owner.

Every tool has its own provider configuration

Symptom: changing routes means hunting through scripts, desktop tools, and automation settings, leaving some on the old provider.

Fix: after the single-tool test, place a controlled gateway alias in front. Inventory clients, version the mapping, separate client and provider credentials, and test rollback.

Local is too slow for the workload

Symptom: one fixture passes, but a realistic batch blocks other work or misses the required completion window.

Fix: time an approved synthetic batch of representative size. Reduce the task, use a fitting model, schedule the work, or assess an approved hosted route; do not infer capacity from one request.

A location label replaces a data-flow review

Symptom: a provider is accepted as "EU-hosted" without identifying inference region, logs, support, backups, retention, subprocessors, and contractual scope.

Fix: mark the route blocked until those boundaries are verified for the exact service and configuration. Product headquarters or a regional marketing page is not complete evidence.

Secrets and payloads enter logs

Symptom: keys appear in code or screenshots, or debug mode records full prompts and responses by default.

Fix: use approved secret storage, redact authorization headers, and retain only necessary metadata such as request ID, route, status, latency, and error category under a reviewed retention rule.

8. Do it yourself: a 60-minute route switch

Minutes 0-8: choose one bounded Lab extraction or Company classification step. Freeze one synthetic fixture, expected JSON, and the current checker. Record the permitted data class, hosted-route approval reference, and rollback profile without recording key values.

Minutes 8-18: load the real approved hosted test profile and run the checker. Save the route label, non-secret base URL, model identifier, timestamp, command, exit status, and complete result. Continue only after this baseline exits 0; absence of an approved route is a blocker, not a passing option.

Minutes 18-28: use ollama list to record the exact local model identifier, send MODEL READY, and identify where its model files and application logs are stored. Run the OS-appropriate listener inspection above and retain its output. Do not continue unless every port 11434 listener is 127.0.0.1 or ::1.

Minutes 28-38: change only the connection profile to the Ollama base URL, placeholder key, and exact model identifier. Run the unchanged checker with the unchanged lane, fixture, and expected object.

Minutes 38-46: investigate any failure once. Check the listener table, selected model, JSON parsing, field values, and elapsed time. Do not loosen an expected value merely to produce a pass. Either correct a connection mistake, choose a more suitable already-approved local model, or restore the hosted profile.

Minutes 46-52: draw the actual path from script to endpoint, including any gateway, logs, backup, and support access. Write the allowed data class and identity at each hop. Confirm no real data or key appears in the evidence.

Minutes 52-57: complete a three-row choice note for local, one EU-hosted candidate, and one approved frontier candidate. For each, record task fit, unresolved location or retention questions, test result if run, and the decision use, reject, or blocked pending evidence.

Minutes 57-60: assemble the single test record described below. Stop the local model when it is no longer needed. If either route test or the listener check failed, you do not have the required passing exit artifact; fix the route before claiming completion.

9. Exit check

Deliver exactly one artifact: one passing route-switch test record for either the Lab or Company lane.

The record must contain the lane and synthetic fixture ID; unchanged checker identifier; hosted-route approval reference, route label, non-secret base URL, model identifier, command, timestamp, exit status 0, and complete baseline PASS line; then the local base URL, exact local model identifier, command, timestamp, exit status 0, and complete local PASS line. Include the OS-appropriate listener command, its output showing only 127.0.0.1 and/or ::1, and its loopback-only PASS line. State that no real data or credential value was captured.

It passes when another person can verify that the approved hosted baseline ran first, rerun the same checker and fixture against the recorded local profile, receive the expected JSON, see the process exit successfully, and reproduce the loopback-only listener result. These are fields in the one route-switch test record, not additional artifacts. A screenshot, route diagram, provider comparison, reachability check, or successful HTTP response alone does not replace the passing test.

10. Rule to remember

Change the endpoint, then re-run the tests.

11. Further reading & tools

• Taught: Local & Private Models - maps the endpoint, gateway, identity, logs, and retrieval boundaries around a local service.
• Taught: Where an AI model runs - records provider, data class, identity, verification, and rollback per route.
• Taught: One traffic controller for AI models - explains stable client routes and the gateway's own security boundary.
• Taught: LiteLLM - separates client keys, provider secrets, route policy, logging, and rollback.
• Taught · Primary: Ollama OpenAI compatibility (opens in a new tab) - official request examples and the currently supported compatibility surface.
• Taught · Primary: Ollama API introduction (opens in a new tab) - official base-URL and API guidance.
• Taught · Primary: LiteLLM Proxy quick start (opens in a new tab) - official proxy and configuration guidance; verify the current schema before use.
• Catalogued · Primary: IONOS AI Model Hub (opens in a new tab), Scaleway Generative APIs (opens in a new tab), OVHcloud AI Endpoints (opens in a new tab), and EUrouter (opens in a new tab) - provider product pages to investigate, not evidence of approval or of the complete data path.
• Catalogued: T05-L02 · Automations with AI in the middle - applies checked model output inside a workflow.
• Catalogued: T11-L03 · Containers, honestly - continues from a personal endpoint to inspectable service packaging.
• Catalogued: Tools index - compare current tools only after defining task, data, location, identity, quality, and rollback boundaries.