Model providersGroq Inc.

Groq

Decision matrix

Groq provider decision matrix
Overview

GroqCloud provides hosted inference (Llama, GPT-OSS, Whisper and others) on Groq's own LPU hardware, from about $0.075 per million input tokens for the smallest models. Pick it when response speed on a supported open model matters; check the current model list, since the specific lineup shifts over time.

Other good fits
  • Trying fast inference on models like GPT-OSS 120B or Llama 3.3 70B without buying dedicated hardware.
Pros

GroqCloud runs a curated set of open and hosted models, currently including OpenAI's GPT-OSS 120B and 20B, Meta's Llama 3.1/3.3, Whisper speech-to-text and others, on Groq's own custom inference hardware (LPUs), built specifically for fast token generation. It's a hosted API: you don't manage any accelerator hardware yourself.

Pick it when…

  • Response speed matters for your application and one of Groq's supported models fits the task.
  • You want to try a specific open-weight model (GPT-OSS, Llama) without provisioning your own GPU.
  • You're building an interactive prototype where latency is part of the user experience.
Cons
Read cons

Groq serves a curated model catalogue, not every provider's flagship; if your task needs a specific closed model (GPT-5, Claude Opus), Groq isn't the way to reach it. Fast inference on hardware doesn't guarantee correct output; a quick wrong answer is still wrong, so review results the same as with any hosted model.

Pick something else when…

  • You need a specific closed-source flagship model. Call that provider's own API directly, or route through OpenRouter.
  • You want to compare Groq's speed against another fast-inference provider on the same model. Cerebras is the closest direct comparison.
  • Prompts must stay entirely on your own hardware. A local runtime like Ollama or LM Studio keeps inference local instead.

Avoid if

  • Your prompts must remain fully on your own hardware; Groq is a hosted API, not a local runtime.
Connection

Call Groq's OpenAI-compatible chat-completions endpoint with your API key and a supported model name. Client code written for OpenAI's API format typically needs only the base URL and model name changed to work with Groq.

Billing

Prices checked 23 September 2026 (model list (opens in a new tab)): rates vary by model, from around $0.075 per million input tokens for GPT-OSS 20B up to higher rates for larger models like GPT-OSS 120B (around $0.60 per million output tokens); check the current console for the exact rate on your chosen model. Some models are available on a free tier; paid usage bills per token through your GroqCloud account. Rate limits apply separately from spend, constraining request or token throughput even within your budget.

Data handling

Groq's documentation states inference content isn't retained by default, with exceptions for optional persistence features and reliability or abuse investigations; usage metadata is retained separately. Configurable zero-data-retention controls exist; review those settings for your account before sending anything sensitive.

First request

Documentation-based suggestion — not run here.

Send a short, capped drafting request, for example a one-sentence acknowledgement of a made-up service request, to a supported model like GPT-OSS 120B, and check both the response content and how it felt in terms of speed for your use case, not just whether it succeeded.

Done when you've inspected the response against your input for accuracy, and formed a real opinion on whether the speed made a practical difference for your application.

Links & evidence
Documentation-based · not hands-on · Sep 23, 2026

Rests on Groq's quick-start, supported-models and data-handling documentation, read on 2026-09-23. No API call was made for this review.

  • Groq supported models · Sep 23, 2026 — GPT-OSS 120B/20B, Llama 3.1/3.3, Whisper; pricing from ~$0.075/M input tokens; 131,072-token context.
  • Groq quick start · Sep 23, 2026 — OpenAI-compatible API call structure.
  • Groq data handling · Sep 23, 2026 — Default non-retention of inference content, exceptions, zero-data-retention controls.
Further review notes & alternatives
Catalogue facts

Run supported open-weight models through Groq's inference service. Use fast responses to iterate on prompts and batch processing. Bring your own API key for course and personal workflows.

Groq Inc. · free-tier · cloud

  • fast llm inference
  • high token throughput
  • hardware acceleration
  • speech-to-text
  • batch summarization
  • open-weight models
  • low latency api
  • real-time annotation
  • automate
  • talk-think
  • write-code
  • draft-docs

Browse all tools