Self-hosted AI Coding Agent

Quick Take

• Replace cloud AI coding assistants with a self-hosted alternative that respects privacy
• Run Mistral Small 4 locally with Docker on ARM64 hardware
• Integrate SearXNG for web search without exposing your queries to third parties

You’ve outgrown cloud-hosted AI coding tools. The latency, the privacy concerns, the subscription costs, it all adds up. OpenHands gives you a local AI agent that writes code, fixes bugs, and searches the web without ever leaving your network. Here’s how to set it up right.

Deploy OpenHands with Docker on ARM64

sudo bash /scripts/recreate-openhands.sh

This script creates a Docker container named openhands on ARM64 hardware. The container runs a lightweight runtime environment optimized for coding tasks. ARM64 support matters because it means you can run this on a Raspberry Pi 5, an NVIDIA Jetson Orin (v1.1), or any other ARM-based server without x86 overhead. The container image is tagged as openhands/openhands:0.5.1-arm64 for version clarity.

The container exposes Mistral Small 4 (v1.0) through a local OpenAI-compatible API endpoint. You don’t need an internet connection to use the model, just point your agent to https://cline.bot/v1. The api_key field is set to "not-needed-local" because the endpoint doesn’t require authentication when running locally.

Gotcha: If you’re using an x86 machine, replace the container image with the AMD64 variant (openhands/openhands:0.5.1-amd64). The ARM64 build won’t work there. You’ll see an error like standard_init_linux.go:228: exec user process caused: exec format error if you attempt to run the wrong architecture.

Configure the Agent with config.toml

[llm]
model = "openai/Mistral-Small-4"
base_url = "https://cline.bot/v1"
api_key = "not-needed-local"
native_tool_calling = true
drop_params = true
modify_params = true

[agent]
enable_prompt_extensions = false
system_prompt_filename = "custom_system_prompt.md"

[sandbox]
additional_networks = ["config_default"]
volumes = "/data/secrets/git-credentials:/root/.gitcredentials:ro,/data/openhands-state/.gitconfig:/root/.gitconfig:ro"

The [llm] section tells OpenHands to use Mistral Small 4 via the local endpoint. native_tool_calling enables function calling so the agent can execute shell commands and modify files directly. drop_params and modify_params optimize the API calls for local inference by reducing unnecessary parameters in requests.

The [agent] section disables prompt extensions to avoid role confusion with Mistral Small 4. The system_prompt_filename points to a custom prompt file you mount into the container at /etc/openhands/custom_system_prompt.md. If you see errors like Error: system_prompt_addition is not supported in this version, it means you’re running OpenHands v0.59.0 or later. Switch to system_prompt_filename and mount the file into the container.

The [sandbox] section configures the agent’s execution environment. additional_networks attaches the container to the same Docker network as SearXNG, so the agent can make web searches without exposing your queries to external services. The volumes mount your Git credentials and config file read-only so the agent can clone private repos. For example, /data/secrets/git-credentials contains your HTTPS credentials for private repositories, while /data/openhands-state/.gitconfig sets your name and email for commits.

Gotcha: If system_prompt_addition errors appear, it means you’re running OpenHands v0.59.0 or later. Switch to system_prompt_filename and mount the file into the container.

Mount Secrets and Config Files

docker run -v /data/secrets/git-credentials:/root/.gitcredentials:ro \
           -v /data/openhands-state/.gitconfig:/root/.gitconfig:ro \
           -v /etc/openhands/custom_system_prompt.md:/etc/openhands/custom_system_prompt.md:ro \
           openhands/openhands:0.5.1-arm64

These mounts give the agent access to your Git credentials and config without letting it modify them. The .gitcredentials file contains HTTPS credentials for private repos, typically stored in /home/username/.git-credentials on your host machine. The .gitconfig file sets your name and email for commits, usually located at /home/username/.gitconfig.

Gotcha: If the agent can’t clone a private repo, double-check the permissions on /data/secrets/git-credentials. The container runs as root, so the file must be readable by root. You might see errors like fatal: could not read Username for 'https://github.com': No such device or address if permissions are incorrect.

Integrate SearXNG for Private Web Search

OpenHands runs in the same Docker network as SearXNG. The agent uses SearXNG’s /search endpoint to perform web searches without leaking queries to Google or Bing. SearXNG is a privacy-respecting metasearch engine that aggregates results from multiple sources while keeping your queries local.

docker network create config_default
docker run -d --network config_default --name searxng -p 8080:8080 searxng/searxng:latest

Attach the OpenHands container to this network:

docker run --network config_default -p 30000:30000 openhands/openhands:0.5.1-arm64

Now when the agent needs to search for documentation or examples, it queries SearXNG instead of an external API. The results are cached locally, so repeated searches are faster. You can verify the setup by visiting http://localhost:8080 in your browser to see SearXNG’s interface.

Gotcha: If SearXNG isn’t reachable, verify the container names and network attachment. Docker’s default bridge network isolates containers unless you explicitly attach them to a custom network. You might see errors like Failed to fetch search results: Connection refused if the network isn’t properly configured.

Avoid Memory Limits for Large Tasks

docker run --memory=2g --memory-swap=2g openhands/openhands:0.5.1-arm64

By default, Docker containers have memory limits that can throttle large coding tasks. The Mistral Small 4 model needs at least 2GB of RAM to run smoothly. If you omit the --memory flag, Docker may impose a lower limit that causes crashes. On ARM64 hardware with 4GB or less RAM, consider using a smaller model like TinyLlama (v1.1) or Phi-2 (v2.0) instead of Mistral Small 4.

Gotcha: If you see errors like OOM killer terminated this process in your container logs, increase the memory limit to 3g or 4g. You can check memory usage with docker stats openhands.

What I Actually Use

• Mistral Small 4 (v1.0): The only model that balances performance and accuracy for coding tasks
• SearXNG (v2024.6.15): Replaces cloud search engines with a self-hosted alternative
• Docker on ARM64: Runs efficiently on low-power hardware like a Jetson Orin (v1.1)