Does it work offline, and do conversations leave the device?

After deployment it runs offline; the first deploy needs internet access to pull the image. Inference runs entirely on the Jetson, and conversations travel only between your app and the device.

How fast are replies?

The first request after deployment warms up for 2-5 minutes, while the model list returns 503. After warm-up, speed depends on prompt and output length, so time a typical request of your own on the device.

How do I connect it to my own application?

The API is OpenAI-compatible on port 8080 of the device. Point an existing OpenAI SDK or a framework such as LangChain at it, or connect it to the OpenClaw gateway to serve several chat platforms.

Can I change the context length? What if startup runs out of memory?

Yes. Context length and GPU layers are deploy parameters, 1024 and 20 by default. If startup runs out of memory, set GPU layers to 16 and context to 512, then redeploy.

Can I use a different model?

The server is llama.cpp, which loads GGUF models. Swapping means replacing the model file in the container and adjusting the launch parameters; GPU memory use follows the model size.

Can several people use it at once?

This is one model behind one server, suited to individuals or small teams trying it out. For many concurrent users, assess GPU memory and queueing delay, and move to a larger-memory host or several hosts if needed.

What should we check on model licensing for a commercial product?

OpenAI publishes the GPT OSS 20B weights under its own licence terms, and llama.cpp keeps its open-source licence. Before shipping a closed product, confirm both suit your product.

GPT OSS 20B on Jetson

One-click SSH deployment of GPT OSS 20B on NVIDIA Jetson with a prebuilt Docker image.

SenseCraftBeginner10minOn-Device LM

Reusable Site Templates

Sites that need a large-language-model backend with no cloud connection, where conversations must not leave the local network.

Backend for a local chat app — the app calls the device in OpenAI format, with no cloud API key.
Offline voice assistant — speech recognition in front hands text to the local model for the reply.
Multi-platform chat gateway — with the OpenClaw gateway, WeChat, Telegram and other platforms share this one local model.

Not a fit: long-context or many-user concurrent workloads. The default context is 1024 tokens; keep single requests under 1000 tokens.

How it works

GPT OSS 20B device composition: a reComputer J40 Series host serves chat to an app or gateway over HTTP

One Jetson host is all it takes. The model and inference server sit in one Docker container and open an OpenAI-compatible API on the LAN for your app or gateway to call.

The 20B model needs about 12-15GB of GPU memory, which is why the host must carry the 16GB module.

Item	Detail
Host	reComputer J40 Series (J4012, Orin NX 16GB)
Prerequisites	Docker and the NVIDIA container runtime installed, SSH reachable
GPU memory	About 12-15GB; container memory capped at 12GB

The container runs the llama.cpp inference server with GPT OSS 20B quantized to 4 bits.

20 GPU layers by default: lower it, for example to 16, if memory runs out.
1024-token context by default: a larger context uses more memory; drop to 512 if startup runs out.
The first request warms up: the first call after deployment can wait 2-5 minutes.

How well it works

The table lists what this design provides on the reComputer J40 Series (J4012, Orin NX 16GB), and its limits.

What it does	Limit
A 20-billion-parameter model answers on the device, with no cloud	Default context is 1024 tokens; keep requests under 1000 tokens
OpenAI-compatible API for existing SDKs and gateways	The first request after deployment warms up for 2-5 minutes
The container starts with the device	Needs the Orin NX 16GB module; the 8GB module does not fit it

GPU layers and context length are deploy-time settings; raising them costs more GPU memory.

What you get out of it

Once deployed, this is the interface.

Interface	What comes out	Notes
HTTP 8080 chat completions	OpenAI-format requests and replies	An existing OpenAI SDK only needs a new base URL
HTTP 8080 model list	Whether the model is ready	Returns 503 while warming up
HTTP 8080 web page	A browser chat page	For quick trials

The API is served on the LAN with no built-in authentication; put a reverse proxy with access control in front before exposing it to other networks.

How to deploy

Check the setup and its product list, then follow the deployment guide.

Deployment Options

Jetson GPT OSS 20B Service

Deploy GPT OSS 20B inference service to Jetson with one click.

2 steps · ~10min

Open the deployment guide →

Edge Device

recomputer_j40

FAQ

A reComputer J40 Series model with the Orin NX 16GB module, such as J4012. The 20B model needs about 12-15GB of GPU memory, which the 8GB module cannot hold. Local deployment also needs 20GB or more of free disk.

Didn't find your answer?

We Are Glad to Be Your Hardware Partner !