Deepgram Brings Billable-Unit and Per-GPU Observability to Self-Hosted Speech AI on SageMaker

Speech AI gets a clearer operational layer on SageMaker

AWS published a technical update on August 27, 2026 describing two observability capabilities for Deepgram speech-to-text and text-to-speech models deployed as self-hosted Amazon SageMaker AI endpoints.

The important change is not a new speech model. It is better visibility into the cost and behavior of models already running inside a customer's AWS account.

Deepgram Enhanced Metrics exposes billing and usage data through Amazon CloudWatch, while Prometheus and OpenTelemetry support exposes engine, host and GPU-level telemetry through SageMaker AI detailed observability.

Billing data follows the existing logging path

Deepgram's container writes CloudWatch Embedded Metric Format records to standard output. SageMaker already forwards endpoint container logs to CloudWatch Logs, where those records become metrics.

That design matters because AWS Marketplace model packages run with network isolation. The model container does not need a new outbound telemetry connection, a sidecar collector or additional IAM permissions just to export Deepgram's billing and usage metrics.

AWS says the dimensions are deliberately low-cardinality and do not contain transcripts, TTS input or per-request identifiers.

ConsumedUnits can be reconciled against Marketplace billing

The Deepgram/SageMakerInference namespace includes a ConsumedUnits metric that uses the same billable-unit values that drive AWS Marketplace metering.

That gives platform and FinOps teams a direct way to compare operational traffic with billed consumption rather than inferring cost from generic endpoint request counts.

The same namespace can break usage down by category, model and transport. AWS also lists audio duration and synthesized character count alongside consumed units.

One limitation is important: this billing stream aggregates across Deepgram endpoints in an AWS account and Region. It is not designed for per-endpoint or per-instance slicing.

The second layer goes deeper into each GPU

For detailed operational diagnosis, Deepgram exposes Prometheus metrics from the inference engine while SageMaker detailed observability collects host and accelerator telemetry through an AWS-managed OpenTelemetry collector.

That creates a different view from the account-level billing stream. Teams can query engine behavior, host resources and per-GPU metrics with PromQL through CloudWatch, Grafana or another Prometheus-compatible tool.

The split is useful: one telemetry layer helps answer what did we consume and pay for?, while the detailed layer helps answer which endpoint, host or GPU is under pressure?

Why this is useful for production AI teams

Self-hosted models are attractive when organizations want inference data to remain within their own cloud boundary, but they also shift more operational responsibility to the customer.

Better native telemetry reduces a common blind spot. Capacity planning can be tied to actual feature usage, cost dashboards can use billable units, and engineers can investigate GPU-level bottlenecks without adding a separate vendor-controlled telemetry path.

It also supports stronger operational separation between model data and monitoring data: AWS says audio and transcripts stay in the customer's AWS account, while the published metrics avoid PII-bearing dimensions.

What teams should still evaluate

Observability does not remove the need for architecture and compliance review. Customers still need to assess their own logging retention, access controls, CloudWatch costs, alerting strategy and shared-responsibility obligations.

The two metric systems also answer different questions. Account-and-Region billing metrics should not be mistaken for endpoint-level performance metrics, and detailed observability will create its own telemetry volume.

AWS says these Deepgram capabilities are available now on SageMaker AI deployments. For teams running production speech AI, the broader signal is that vendor-specific metering and model-engine telemetry can coexist with a cloud-managed, network-isolated deployment instead of forcing a choice between privacy boundaries and useful operational visibility.