Hugging Face Serverless Inference vs Inference Endpoints

Serverless Inference in one line

Hugging Face Serverless Inference is a free (or Pro-boosted) shared pool that lazy-loads any Hub model on demand. The first request to a cold model returns HTTP 503 with an `estimated_time` field telling your client when to retry; warm requests return in well under a second. Rate limits are strict, apply per-token and per-model, and change without notice. Serverless is the fastest way to prototype against any of the million-plus models on the Hub without provisioning infrastructure, and it is completely wrong for production workloads with SLOs.

Inference Endpoints in one line

An Inference Endpoint is a dedicated instance of a specific model on a specific GPU SKU in a specific region. It stays warm, has predictable latency, no shared rate limits, and bills per-second — even when idle unless you enable scale-to-zero. Endpoints are the production answer: private networking, org ACLs, custom containers, autoscaling policies, and a stable inference URL that survives model updates. In exchange you accept a monthly bill that scales with wall-clock time rather than usage, and you take on the capacity-planning problem.

Pick Serverless when

You're prototyping or building an internal demo
Traffic is spiky, low, and can tolerate an occasional cold-load 503
You'll test many different models before committing to one
You're a Pro user and the quota comfortably fits your usage
The model is small and popular enough that it stays warm in the shared pool

Pick Endpoints when

You have production traffic with an SLO
You hit Serverless rate limits regularly
You need a specific GPU SKU for a specific model size
You need private networking, VPC peering, or org-level access control
You need custom container images or non-standard runtimes (vLLM, TGI, custom kernels)
Predictable per-request latency matters more than absolute cost

The third option: the HF router

router.huggingface.co is an OpenAI-compatible endpoint that fans out to many hosted providers (HF, Together, Fireworks, Replicate, and more). It is useful when you want one HF token but multi-provider fallback, and when you would rather compare Serverless vs Inference Endpoints against external providers on the same billing account. The router adds a small HF margin over provider pricing but eliminates the vendor-integration tax. For teams that want provider diversity without integration overhead, it is often the best default.

Rate limits, quotas, and what they actually mean

Serverless Inference publishes a tokens-per-minute quota that varies by plan (Free, Pro, Enterprise). In practice the effective quota also depends on model popularity — a rarely-used model may be evicted from the shared pool between calls and re-loaded on demand, costing you cold starts even within quota. Endpoints have no shared quota; the limit is the physical throughput of the SKU you rent, which for TGI on an L4 with a 7B model is around 30–60 tokens/second aggregate. The router inherits its provider's limits per underlying route.

Migrating from Serverless to Endpoints without rewriting

Both accept the same model repo IDs; the client shape is very close. You swap the base URL from api-inference.huggingface.co to your endpoint URL, replace your HF token with the endpoint token, and drop any client-side retry logic that watched for the Serverless-specific 503 with `estimated_time`. Endpoints never return that shape — they either serve, queue, or 429. If you built on huggingface_hub's `InferenceClient` you can pass the endpoint URL to the same constructor and the switch is a one-line change.

Cost sanity check

Serverless — $0 or your Pro subscription; no per-request charge
Endpoints on an L4 always-on — ~$936/mo baseline; scale-to-zero can drop this to ~$250/mo
Router with a Together L4-class provider — typically $0.20–$0.60 per million tokens
Rule of thumb: Endpoints win once daily traffic exceeds a few thousand requests of a small/mid model

FAQ

Can I move from Serverless to Endpoints without changing code?

Mostly yes. Both accept the same model repo IDs and the request shape is nearly identical. You swap the base URL, replace the HF token with the endpoint token, and remove any retry logic that watched for the Serverless-specific 503 with an `estimated_time` field. Endpoints do not return that shape — they serve, queue, or return 429.

Which is cheaper for a low-traffic app?

Serverless is essentially free for low traffic — either $0 on the free tier or covered by a Pro subscription. Even at Pro's higher quota, you can serve tens of thousands of small requests per month without paying anything above the flat subscription. An Endpoint at the same volume would be dominated by idle instance cost.

Which is faster once warm?

For the same model on the same SKU, they are indistinguishable — both run TGI or vLLM on identical hardware. The difference is variance: an Endpoint has flat latency because you own the instance, while Serverless can spike if the shared pool evicts your model or queues behind a burst of unrelated traffic.

How do I know if I've hit a Serverless rate limit?

You will get HTTP 429 with a body describing the limit and a Retry-After header. If you see repeated 429s at expected traffic levels, upgrade to Pro (higher quota) or move that model to an Endpoint. There is no soft-degradation mode; the API either serves or 429s.

Do Endpoints support autoscaling?

Yes. You configure min-replicas, max-replicas, and a scale-to-zero idle timeout. Autoscaling reacts to request queue depth, which for TGI and vLLM is a reasonable proxy for load because those servers batch aggressively. Set max-replicas as a hard cost ceiling — a runaway retry storm can 10x your bill overnight.

Can I use custom Docker images on Endpoints?

Yes. Endpoints support both the managed TGI/text-embeddings-inference images and custom containers. Custom containers unlock non-standard runtimes (vLLM with specific plugins, a Rust inference server, custom kernels) but you take on responsibility for start-up time, health checks, and process management.

Is the HF router a good fit for production?

For workloads where provider diversity or multi-provider fallback matters, yes. For workloads that need deep observability into a specific provider's behavior, going directly to that provider (or renting an Endpoint) is often cleaner. The router adds an extra hop; that hop is negligible on latency but real for debugging.

What happens to a Serverless request while the model is loading?

The API returns HTTP 503 immediately with an `estimated_time` field indicating how many seconds until the model is expected to be warm. Your client is expected to sleep for that duration and retry. The huggingface_hub Python client does this automatically; if you rolled your own client you need to implement it explicitly.