T11-L02 · Run your own AI · Level 2 Power user · 30-minute read

At this level, a model result can enter a team document or pipeline. A route change that looks private but sends data elsewhere, or a weaker result that passes unnoticed, can therefore affect colleagues as well as you.

1. The workflow must stay inside the building

You have a small workflow that sends text to a cloud API and returns structured output. In the Lab version, it extracts fields from literature notes before a colleague checks them. In the Company version, it classifies supplier messages before they enter a shared queue. The code works, but the next approved dataset must not leave the organisation's controlled machine or network.

You install a local model and receive a successful response from its chat screen. That does not yet help the workflow. The script needs an API address, and its existing cloud library expects a particular request and response shape. Even if the local endpoint accepts that shape, its output may differ enough to break the checks.

Choose a tool workflow from your own project and the useful result it will prepare for a colleague. You will expose Ollama only on the workstation's loopback address, connect the chosen compatible tool through its endpoint profile, check the actual output, and record quality and speed limits. A hosted-to-local comparison is optional; a useful local-only workflow does not require a paid or hosted baseline. If you claim a route comparison, you must actually run both permitted routes with unchanged task checks.

2. After this you can

  • Expose an installed local model through a loopback API that a script can call.
  • Switch an OpenAI-compatible workflow by changing its endpoint profile rather than rewriting its task logic.
  • Verify structured output with the same checks used for the hosted route.
  • Explain when a gateway keeps provider changes out of downstream tools.
  • Choose a local, EU-hosted, or frontier route for one named task and data class.

3. Prerequisites

  • T11-L01 · A model on your own laptop.
  • An approved workstation with Ollama installed and one approved model already downloaded. Use the exact identifier shown by ollama list; this book deliberately does not recommend a model name.
  • A chosen approved tool that accepts the local endpoint. The optional Python checker below requires Python and the OpenAI Python package installed according to its current documentation.
  • Only for a hosted/local comparison: a hosted test route approved for the chosen input, plus its approved credential mechanism. No hosted route is needed for a local-only outcome.
  • Permission to run a loopback service and change test-only environment variables.
  • A text editor and terminal, plus 60 minutes for the independent exercise.

Use permitted input from your chosen project or the optional synthetic fixtures below. Do not send unpublished findings, participant or patient data, customer or supplier records, contracts, credentials, or confidential text merely because the endpoint is local. If you cannot identify the permitted data class and complete processing path, stop before making the request.

4. The idea in one page

An API endpoint is an address a tool calls. Ollama can accept an OpenAI-compatible chat request at http://127.0.0.1:11434/v1. For an existing compatible tool, the connection profile normally contains a base URL, API key reference, and model identifier. Change that profile, not the extraction or classification logic. Ollama's client key value is a required placeholder for this local compatibility path, not access control. Keep the service on loopback for this exercise; do not bind it to all interfaces or expose the port to a LAN.

Compatible means that selected request and response fields have a familiar shape. It does not mean identical model behavior, context limits, structured-output reliability, tool support, latency, or error handling. Change your behaviour: after any endpoint or model change, run the same task checks against fixed fixtures. A successful HTTP response is not a passing workflow.

A gateway such as LiteLLM can give clients one stable base URL and a route alias such as bounded-worker. The gateway maps that alias to Ollama or an approved hosted provider and keeps provider credentials on the gateway side. Downstream tools no longer need a separate provider edit. Change your behaviour: version the route mapping, restrict who can change it, record the previous mapping, and test every change. The gateway is another trust boundary: it can see requests, its logs may retain data, and a wrong route can affect every client.

Choose a route per task, not once for the whole organisation:

RoutePrefer it whenCheck before use
LocalInput must stay on the controlled host or network, offline operation matters, and a fitting model passes the task checks at the required speed.Host, listening address, callers, model source and licence, logs, backups, output quality, capacity, and shutdown path.
EU-hosted candidateYou need managed capacity or availability and an approved review requires a defined European processing route.The exact service and region for inference, storage, logs, backups, support access, subprocessors, retention, deletion, contract, and organisation operating the provider.
Approved frontier endpointThe task needs capability the tested local or regional route does not meet, and the data is permitted on that exact service.Account, model route, data terms, retention, location, access, cost controls, output tests, and fallback.

"EU-hosted" is not a complete boundary. It may describe inference while account data, diagnostics, support, or backups follow another path. IONOS, Scaleway, OVHcloud, and EUrouter are candidates to investigate, not pre-approved conclusions. Verify current primary documentation, the selected configuration, signed terms, and your organisation's decision. Likewise, "frontier" describes a capability choice, not a location or privacy guarantee.

For every route, draw workflow -> gateway if used -> endpoint -> logs/support, name the identity at each hop, and state what data may cross it. Keep keys in an approved secret mechanism, not source code or screenshots. Local inference narrows one data path; it does not settle device access, retrieval permissions, telemetry, backup, or retention.

5. The worked example: switch the route and keep the checks

Connect the tool you actually need

Choose one workflow that can use the OpenAI-compatible connection fields described above: for example, preparing a source-based draft or extracting fields for a colleague's review. Record the previous configuration before changing it. After the listener check, enter the local base URL, placeholder key, and exact installed model ID in that tool's documented provider settings. Disable any automatic hosted fallback for a local-only task, or do not send the material if you cannot establish the actual route. Local model inference does not make a hosted client, its uploads, or its logs local.

Run a small permitted item through the tool itself, not just the model's chat screen. Open the produced document or record, check the claims or values that matter, measure elapsed time, and have the colleague use the reviewed result. Save the non-secret profile and known limitations. The unchanged checker below is an optional concrete compatibility exercise; its Lab/Company fixture is not the required team deliverable.

The Lab and Company lanes use the same script, endpoint profile, and pass rule. Only the synthetic input, requested operation, and expected fields differ.

Run one unchanged checker

Save this as route_check.py in a temporary exercise folder. It asks for JSON, parses the response, and compares every field with expected fixture values. It does not trust the model's own claim that it passed.

import json
import os
import sys

from openai import OpenAI

LANES = {
    "lab": {
        "instruction": (
            "Extract the synthetic literature note. Return JSON only with keys "
            "record_id, material, temperature_c, and result. Use null when absent."
        ),
        "source": (
            "Synthetic literature fixture L-17. Record ID: L-17. Material: polymer film. "
            "The film was held at 18 C. Result: no visible change."
        ),
        "expected": {
            "record_id": "L-17",
            "material": "polymer film",
            "temperature_c": 18,
            "result": "no visible change",
        },
    },
    "company": {
        "instruction": (
            "Classify the synthetic supplier message. Return JSON only with keys "
            "record_id, label, and basis. Label must be ACTION or INFORMATION."
        ),
        "source": (
            "Synthetic supplier fixture C-17. Record ID: C-17. Delivery remains scheduled "
            "for Tuesday. No reply or approval is requested."
        ),
        "expected": {
            "record_id": "C-17",
            "label": "INFORMATION",
            "basis": "No reply or approval is requested.",
        },
    },
}

lane_name = os.environ.get("LANE", "lab")
if lane_name not in LANES:
    sys.exit("FAIL: LANE must be lab or company")

required = ["MODEL_BASE_URL", "MODEL_API_KEY", "MODEL_NAME"]
missing = [name for name in required if not os.environ.get(name)]
if missing:
    sys.exit("FAIL: missing " + ", ".join(missing))

lane = LANES[lane_name]
client = OpenAI(
    base_url=os.environ["MODEL_BASE_URL"],
    api_key=os.environ["MODEL_API_KEY"],
)
response = client.chat.completions.create(
    model=os.environ["MODEL_NAME"],
    temperature=0,
    response_format={"type": "json_object"},
    messages=[
        {"role": "system", "content": lane["instruction"]},
        {"role": "user", "content": lane["source"]},
    ],
)

try:
    actual = json.loads(response.choices[0].message.content)
except (json.JSONDecodeError, TypeError) as error:
    sys.exit(f"FAIL: response was not JSON: {error}")

if actual != lane["expected"]:
    print("FAIL: field check")
    print("Expected:", json.dumps(lane["expected"], sort_keys=True))
    print("Actual:  ", json.dumps(actual, sort_keys=True))
    sys.exit(1)

print(f"PASS: {lane_name} via {os.environ['MODEL_BASE_URL']}")

Optional comparison: record a hosted baseline

Before starting or selecting the local route, load the real approved hosted test profile for your chosen lane. Set MODEL_BASE_URL, MODEL_API_KEY, and MODEL_NAME through your organisation's approved mechanism, set LANE to lab or company, and run python route_check.py. Do not put a credential value in the record.

For a claim that a previously passing workflow still passes after switching, require a hosted exit status of 0. Record the approval reference, route label, non-secret base URL, model identifier, lane, fixture ID, checker identifier, command, timestamp, exit status, and complete result. If the route is unavailable or fails, report that honestly. You may continue with the local-only task, but cannot claim a successful hosted-to-local switch or an observed quality/speed comparison without the corresponding runs.

Start and prove the local endpoint

Now confirm the downloaded identifier:

ollama list

If Ollama is not already serving requests, run ollama serve in a separate terminal. If it reports that the address is already in use, inspect the existing listener rather than starting a duplicate. Do not alter the listening address for this exercise.

A request succeeding at 127.0.0.1 does not prove that the same service is unavailable on other interfaces. Use the command for your operating system to inspect every TCP listener on port 11434. Each command fails if there is no listener or if any listener is bound to an address other than 127.0.0.1 or ::1.

Windows PowerShell:

$listeners = @(Get-NetTCPConnection -State Listen -LocalPort 11434 -ErrorAction Stop)
if ($listeners.Count -eq 0) { throw "FAIL: no listener on port 11434" }
$listeners | Format-Table LocalAddress, LocalPort, OwningProcess
$outsideLoopback = @($listeners | Where-Object { $_.LocalAddress -notin @("127.0.0.1", "::1") })
if ($outsideLoopback.Count -gt 0) { throw "FAIL: port 11434 is listening beyond loopback" }
"PASS: every port 11434 listener is loopback-only"

Linux:

ss -H -ltn 'sport = :11434' | awk '
BEGIN { seen=0; bad=0 }
{ seen=1; if ($4 != "127.0.0.1:11434" && $4 != "[::1]:11434") { print "FAIL: non-loopback listener " $4; bad=1 } }
END { if (!seen) { print "FAIL: no listener on port 11434"; exit 1 }; if (bad) exit 1; print "PASS: every port 11434 listener is loopback-only" }'

macOS:

listeners="$(lsof -nP -a -iTCP:11434 -sTCP:LISTEN -Fn 2>/dev/null | sed -n 's/^n//p')"
[ -n "$listeners" ] || { echo "FAIL: no listener on port 11434"; exit 1; }
for address in $listeners; do
  case "$address" in
    "127.0.0.1:11434"|"[::1]:11434") ;;
    *) echo "FAIL: non-loopback listener $address"; exit 1 ;;
  esac
done
echo "PASS: every port 11434 listener is loopback-only"

If the check reports 0.0.0.0, ::, a LAN address, or any other non-loopback address, stop Ollama and restore its approved loopback configuration before continuing. Do not use an external port scanner or weaken a firewall to perform this check. Retain the command and output as evidence.

Only after the listener check passes, replace the placeholder below with the exact identifier from ollama list and make the direct synthetic test:

ollama run <EXACT_DOWNLOADED_MODEL_ID> "Reply with exactly: MODEL READY"

Then set the chosen tool's local connection profile, preserving any previous profile for recovery:

MODEL_BASE_URL=http://127.0.0.1:11434/v1
MODEL_API_KEY=ollama
MODEL_NAME=<EXACT_DOWNLOADED_MODEL_ID>
LANE=lab

For the script, use your shell's normal environment-variable syntax and choose lab or company; keep the same LANE across any comparison. The three endpoint values form one connection profile. An existing GUI tool with fields for an OpenAI-compatible provider uses the same mapping: base URL, placeholder key, and exact model identifier.

Lab framing: extraction stays exact

Set LANE=lab and run python route_check.py. The literature-shaped fixture contains no real paper, sample, or unpublished result. The test passes only when the local model preserves all four fields, including the number and the negative result. If it changes 18 or turns "no visible change" into a stronger conclusion, record failure.

If you perform the optional hosted comparison, replace only its endpoint profile with the local values; do not edit the fixture, prompt, expected object, lane, or checker between runs. This isolates the route change. Without that run, record the local result on its own.

Company framing: classification keeps its basis

Set LANE=company and run the same command. The supplier-shaped fixture is fictional. The test passes only when the route returns INFORMATION and preserves the sentence explaining why. An invented request for approval must fail the comparison even if the label happens to be correct.

The two lanes have parallel stakes: a wrong Lab extraction can enter a literature table; a wrong Company classification can move a supplier message into the wrong shared queue. In either lane, teammates inherit the error if you accept a response merely because the endpoint returned status success.

Put a stable alias in front later

When several tools need the route, configure a reviewed LiteLLM proxy alias instead of editing each client. A minimal local mapping has this shape; check the current primary documentation before running it:

model_list:
  - model_name: bounded-worker
    litellm_params:
      model: ollama_chat/<EXACT_DOWNLOADED_MODEL_ID>
      api_base: http://127.0.0.1:11434

Clients then use the proxy base URL and MODEL_NAME=bounded-worker. To change provider later, an owner updates the alias mapping, uses a server-side secret reference for any hosted credential, reruns both checks, and retains the prior configuration as rollback. This loopback example is not a shared production gateway: shared access requires authentication, authorization, TLS, scoped client keys, request limits, protected logs, and an operator.

6. What goes wrong

The route changes but the checks do not run

Symptom: the local endpoint answers successfully, so its output enters the shared pipeline without comparison to the fixed fixture.

Fix: run the exact pre-switch checker after every endpoint, model, prompt, or gateway-mapping change. Keep failure visible and restore the prior profile if the check does not pass.

Compatible is mistaken for identical

Symptom: a request field is ignored, JSON differs, or an error is shaped differently even though both endpoints are called OpenAI-compatible.

Fix: test only the fields your workflow uses. Parse and validate the response, exercise failure handling, and remove unsupported assumptions rather than hiding them with retries.

The local port is exposed to other machines

Symptom: the service listens beyond loopback or a firewall rule permits unauthenticated network callers.

Fix: stop it and restore loopback-only access. Do not make this Level 2 exercise a team service; authenticated shared deployment belongs to later books and an approved owner.

Every tool has its own provider configuration

Symptom: changing routes means hunting through scripts, desktop tools, and automation settings, leaving some on the old provider.

Fix: after the single-tool test, place a controlled gateway alias in front. Inventory clients, version the mapping, separate client and provider credentials, and test rollback.

Local is too slow for the workload

Symptom: one fixture passes, but a realistic batch blocks other work or misses the required completion window.

Fix: time an approved synthetic batch of representative size. Reduce the task, use a fitting model, schedule the work, or assess an approved hosted route; do not infer capacity from one request.

A location label replaces a data-flow review

Symptom: a provider is accepted as "EU-hosted" without identifying inference region, logs, support, backups, retention, subprocessors, and contractual scope.

Fix: mark the route blocked until those boundaries are verified for the exact service and configuration. Product headquarters or a regional marketing page is not complete evidence.

Secrets and payloads enter logs

Symptom: keys appear in code or screenshots, or debug mode records full prompts and responses by default.

Fix: use approved secret storage, redact authorization headers, and retain only necessary metadata such as request ID, route, status, latency, and error category under a reviewed retention rule.

7. Do it yourself: a useful local-model workflow

Minutes 0-8: choose one tool and a bounded colleague-facing output from your project. Declare the permitted input and checks for a useful result. Record the actual processing path and previous connection profile without key values. The Lab/Company checker is optional practice.

Minutes 8-18: inspect the chosen tool's documented endpoint configuration and available runtime/model. If comparing routes, run and record the approved hosted baseline now using the same permitted input and checks. Otherwise prepare the local-only task; mark hosted comparison not performed.

Minutes 18-28: use ollama list to record the exact local model identifier and identify its model/log locations. Run the OS-appropriate listener inspection above and retain its output. Do not continue unless every port 11434 listener is 127.0.0.1 or ::1; only then send the MODEL READY request.

Minutes 28-38: connect the chosen tool to the Ollama base URL, placeholder key, and exact model identifier. Run the permitted task through that tool and retain the actual output and elapsed time. For a route comparison, hold task input, prompt, and checks unchanged.

Minutes 38-46: inspect the result against the declared task checks. Investigate a failure once: listener, selected model, relevant response fields, factual quality, and elapsed time. Do not loosen expected values to produce a pass. Correct a connection mistake, choose a fitting already-approved model, or return to the previous permitted/manual workflow; never fall back to a hosted service with local-only data.

Minutes 46-52: draw the actual path from script to endpoint, including any gateway, logs, backup, and support access. Write the allowed data class and identity at each hop. Confirm no real data or key appears in the evidence.

Minutes 52-57: have the colleague use the reviewed result in the intended task. Record local quality, speed, resource limits, and the manual or approved-route fallback. Include comparisons only for routes you actually observed; a candidate's location claim is not a measured result.

Minutes 57-60: retain the useful output with its non-secret configuration and limits. Stop the local model when no longer needed. A failed local task or listener check remains unfinished; lack of a hosted comparison does not invalidate a successful local-only handoff.

8. Exit check

Deliver one local-workflow handoff: the chosen tool's useful colleague-facing output with a concise run/configuration note, observed quality/speed limits, and recovery instructions.

Record the tool, permitted input identity, local base URL, exact model identifier, command or GUI configuration, timestamp, actual result, task checks, elapsed time, and colleague use. Include the OS-appropriate listener command and its observed loopback-only output. Exclude credentials and unapproved source content. If claiming a hosted-to-local switch, also retain both actual route results, their non-secret profiles, approval references, and unchanged input/prompt/checker identity.

It passes when the chosen tool has actually used local inference to prepare an output the colleague can use after review, its task checks pass, and listener inspection shows only loopback access. The recorded steps must let another person repeat the attempt within the stated limits. A successful HTTP response alone is insufficient. Two observed routes are required only for claims comparing or successfully switching between them.

9. Rule to remember

Change the endpoint, then re-run the tests.

10. Further reading & tools