LLM optimization integration for Amazon SageMaker Python SDK
The Amazon SageMaker Python SDK v3 now exposes generative AI inference recommendations in Amazon SageMaker AI directly in your notebook. Benchmark an endpoint, generate data-driven deployment recommendations, and deploy the recommended configuration without leaving your notebook workflow.
Optimizing generative AI inference deployments requires benchmarking endpoints, evaluating instance configurations, and iterating on deployment settings. The Amazon SageMaker Python SDK v3 now exposes generative AI inference recommendations in Amazon SageMaker AI directly in your notebook workflow. These recommendations are also accessible through the Amazon SageMaker AI UI and Boto3 APIs. With this release, you can benchmark an endpoint, generate data-driven deployment recommendations, and deploy the recommended configuration directly from a notebook using the Amazon SageMaker Python SDK v3.
In this post, we demonstrate how to use the new SDK interface for the end-to-end workflow to optimize generative AI inference deployments.
Benefits of generative AI inference recommendations in Amazon SageMaker AI
Generative AI inference recommendations in Amazon SageMaker AI automate inference optimization by:
Benchmarking a live Amazon SageMaker endpoint against a synthetic or real-traffic workload, measuring throughput, time-to-first-token (TTFT), end-to-end latency, and more.
Generating deployment recommendations ranked by cost-performance tradeoff using your actual usage patterns.
Deploying the top-ranked configuration directly to an Amazon SageMaker real-time endpoint.
Previously, these capabilities required using Amazon SageMaker Studio or constructing AWS SDK for Python (Boto3) API calls. With this launch, they become Python SDK operations, fitting naturally into existing notebook and pipeline workflows.
New SDK interfaces
The new functionality is available under the sagemaker.serve.ai_inference_recommender package starting with version 3.17.0 and exposes the following primary operations:
Entry point
What it does
ModelBuilder.from_jumpstart_config(…)
Builds a ModelBuilder from a JumpStart model ID and compute config
start_benchmark(endpoint, …)
Runs a load test against a deployed endpoint with a configurable synthetic workload
mb.generate_deployment_recommendations(…)
Explores instance/framework configs against your workload and returns ranked recommendations
mb.deploy(…)
Deploys the top recommendation to a real-time endpoint
ModelBuilder.from_recommendation_job(job_name)
Hydrates a ModelBuilder from a completed recommendation job — deploy in a different process or session
Prerequisites
Verify you have the latest version of the Amazon SageMaker Python SDK installed:
A deployed Amazon SageMaker real-time endpoint (or a JumpStart model to deploy; see the following section).
Solution overview
Consider a common scenario: you have a generative AI model ready for production and need to determine the optimal instance type, framework configuration, and serving parameters. Traditionally, this involves manual trial and error across multiple instance types, container versions, and concurrency settings. With the Amazon SageMaker Python SDK integration, you can automate this entire workflow in a single notebook. The following walkthrough guides you through the end-to-end journey using this notebook:
Generate deployment recommendations: Let the service explore instance and framework configurations against your workload profile and return ranked options.
Interpret and select: Review the ranked results, understand the tradeoffs, and pick the best fit.
Deploy: Push the winning configuration to a live Amazon SageMaker endpoint.
Benchmark: Validate the deployed endpoint under realistic load conditions.
Compare frameworks: Optionally run LMI and vLLM head-to-head to find the best serving stack.
Generate recommendations from real traffic data
Your first step is to find the best deployment configuration for your model and workload. Rather than manually deploying across multiple instance types, call mb.generate_deployment_recommendations(…) to let the service explore instance types and framework configurations against your workload profile. The service deploys your model on each candidate, runs a load test matching your traffic pattern, and returns a ranked list of configurations optimized for your chosen performance target.
import time, uuid
from sagemaker.core.jumpstart.configs import JumpStartConfig
from sagemaker.serve import ModelBuilder
from sagemaker.train.configs import Compute
from sagemaker.serve import InferenceFramework, PerformanceTarget
uid = f"{int(time.time())}-{uuid.uuid4().hex[:8]}"
src_model_name = f"demo-rec-source-{uid}"
rec_ep_name = f"demo-rec-ep-{uid}"
mb = ModelBuilder.from_jumpstart_config(
jumpstart_config=JumpStartConfig(model_id=MODEL_ID),
compute=Compute(instance_type=INSTANCE_TYPE),
role_arn=ROLE,
)
source_model = mb.build(model_name=src_model_name)
rec_job = mb.generate_deployment_recommendations(
tokenizer="google/gemma-4-e2b-it",
concurrency=1,
request_count=10,
prompt_input_tokens_mean=32,
output_tokens_mean=32,
streaming=True,
performance_target=PerformanceTarget.TTFT_MS
instance_types=[INSTANCE_TYPE],
advanced_optimization=False,
framework=InferenceFramework.LMI,
role_arn=ROLE,
wait=True,
)
#Comparative table across all returned recommendations
print(mb.recommendations)
# .best is the top-ranked row
top = mb.recommendations.best
print(f"Throughput avg: {top.expected_performance.request_throughput.avg}")
print(f"TTFT p99: {top.expected_performance.time_to_first_token.p99}")
# auto_approve=True bypasses the ModelPackage approval-status check
rec_endpoint = mb.deploy(
endpoint_name=rec_ep_name,
role=ROLE,
wait=True,
auto_approve=True,
)
print(f"Deployed: {rec_endpoint.endpoint_name} ({rec_endpoint.endpoint_status})")
Recommendation results can also be represented as a Python data frame.
import pandas as pd
pd.set_option("display.width", 200)
pd.set_option("display.max_columns", None)
pd.set_option("display.max_colwidth", 60)
rows = []
for i, rec in enumerate(mb.recommendations):
raw = rec._raw
spec = raw.model_details.inference_specification_name
for m in raw.expected_performance:
rows.append({
"rank": i,
"spec_name": spec,
"instance": raw.deployment_configuration.instance_type,
"metric": m.metric, # ← attribute, not subscript
"stat": m.stat,
"value": float(m.value),
"unit": m.unit,
})
df_long = pd.DataFrame(rows)
print(df_long)
The recommendations table shows two candidate configurations (rank 0 and rank 1), both on ml.g6.2xlarge but with different LMI container versions. Here’s how to read the key metrics and choose between them:
Key metrics to compare:
RequestThroughput (avg): Requests the endpoint can serve per second. Higher is better.
OutputTokenThroughput (avg): Total tokens generated per second across all concurrent requests. Higher is better.
RequestLatency (p50/p90/p99): End-to-end time from request to full response. Lower is better.
TimeToFirstToken (p50/p90): How quickly the user sees the first streamed token. Lower is better.
InterTokenLatency (p50/p90): Delay between successive tokens during streaming. Lower is better.
Choosing between the two configurations in this example:
Rank 0 (lmi-26-0-0) delivers 112.8 req/s throughput and 3,609 tokens/s, with p90 TTFT of 983 ms and p90 latency of 1,000 ms. Rank 1 (lmi-27-0-0) delivers 96.9 req/s throughput and 3,099 tokens/s, with p90 TTFT of 1,088 ms and p90 latency of 1,122 ms. Rank 0 wins on every dimension: approximately 16% higher throughput and approximately 10 percent lower latency. The service ranks it first because the job was configured with performance_target=PerformanceTarget.TTFT_MS, meaning the optimizer prioritized configurations that minimize time-to-first-token.
General decision framework
Latency-sensitive applications (chatbots, interactive UIs): Prioritize low TTFT (p90/p99) so users perceive fast responses.
Throughput-sensitive workloads (batch summarization, offline processing): Prioritize high RequestThroughput and OutputTokenThroughput to maximize tokens per dollar.
If two configurations are close on your primary metric, use the secondary metrics as tiebreakers, then factor in cost (a smaller instance at similar performance saves money).
In this example, the top-ranked configuration (lmi-26-0-0) is the clear choice because it dominates across all metrics at the same concurrency level (64).
Deploy from previously run recommendation job
In production workflows, you often generate recommendations in one session and deploy in another. For example, a data scientist might run the recommendation job during experimentation, while an MLOps pipeline deploys the result during a release cycle. Use ModelBuilder.from_recommendation_job(job_name) to hydrate a ModelBuilder from a completed job:
from sagemaker.serve import ModelBuilder
# Hydrate a fresh ModelBuilder from a completed recommendation job
mb = ModelBuilder.from_recommendation_job("my-rec-job-name")
print(f"Loaded {len(mb.recommendations)} recommendations")
print(mb.recommendations)
# Deploy the top-ranked recommendation
endpoint = mb.deploy(
role=ROLE,
wait=True,
auto_approve=True,
)
Deploy a JumpStart model and benchmark it
After you have deployed your recommended configuration, the next step is to validate its performance under controlled conditions. Benchmarking confirms that the endpoint meets your latency and throughput requirements before serving production traffic. The SDK makes this straightforward: deploy a JumpStart model and run a synthetic load test in only a few lines of code.
import time, uuid
from sagemaker.core.jumpstart.configs import JumpStartConfig
from sagemaker.serve import ModelBuilder, start_benchmark
from sagemaker.train.configs import Compute
from sagemaker.serve import InferenceFramework, PerformanceTarget
uid = f"{int(time.time())}-{uuid.uuid4().hex[:8]}"
ep_name = f"demo-bench-ep-{uid}"
model_name = f"demo-bench-model-{uid}"
# Build and deploy a JumpStart endpoint
mb = ModelBuilder.from_jumpstart_config(
jumpstart_config=JumpStartConfig(model_id=MODEL_ID),
compute=Compute(instance_type=INSTANCE_TYPE),
role_arn=ROLE,
)
core_model = mb.build(model_name=model_name)
core_endpoint = mb.deploy(endpoint_name=ep_name)
# Benchmark with a synthetic workload
job = start_benchmark(
endpoint=core_endpoint,
tokenizer="google/gemma-4-e2b-it",
concurrency=1,
request_count=10,
prompt_input_tokens_mean=32,
output_tokens_mean=32,
streaming=True,
role=ROLE,
wait=True,
)
result = job.show_result()
Reading benchmark results
After the benchmark completes, you need to understand whether your endpoint meets your service-level objectives. The benchmark returns a typed result object with a metrics accessor that gives you programmatic access to throughput, latency percentiles, and token-level timing. IDE autocomplete works on all fields.
# Well-known shortcuts — fully typed, IDE autocomplete works
print(f"Throughput avg: {result.metrics.request_throughput.avg} req/sec")
print(f"TTFT p99: {result.metrics.time_to_first_token.p99} ms")
print(f"E2E latency p90: {result.metrics.request_latency.p90} ms")
# Any metric AIPerf produced, by raw key
ott = result.metrics.get("output_token_throughput")
if ott:
print(f"Output token throughput p90: {ott.p90} {ott.unit}")
Benchmark results can similarly be represented as a Python data frame.
import pandas as pd
pd.set_option("display.width", 200)
pd.set_option("display.max_columns", None)
pd.set_option("display.max_colwidth", 60)
result = job.show_result()
rows = []
for name, m in result.metrics.all_metrics.items():
rows.append({
"metric": name,
"unit": m.unit,
"avg": m.avg,
"p50": m.p50,
"p90": m.p90,
"p99": m.p99,
})
df = pd.DataFrame(rows).set_index("metric")
print(df)
unit avg p50 p90 p99
metric
request_throughput requests/sec 3.841439 NaN NaN NaN
request_latency ms 256.288407 206.586304 257.917252 656.735558
request_count requests 10.000000 NaN NaN NaN
time_to_first_token ms 91.770545 23.977319 93.899204 636.935233
time_to_second_token ms 3.035445 3.400013 3.611624 3.670301
inter_token_latency ms 4.987234 5.518454 5.689482 5.695788
output_token_throughput tokens/sec 130.608919 NaN NaN NaN
output_token_throughput_per_user tokens/sec/user 1036.753582 181.210181 1041.238702 7969.310858
output_sequence_length tokens 34.000000 34.000000 35.000000 35.000000
input_sequence_length tokens 32.000000 32.000000 32.000000 32.000000
output_token_count tokens 34.000000 34.000000 35.000000 35.000000
inter_chunk_latency ms 5.307028 5.921907 6.133938 6.599335
total_output_tokens tokens 340.000000 NaN NaN NaN
benchmark_duration sec 2.603191 NaN NaN NaN
total_isl tokens 320.000000 NaN NaN NaN
total_osl tokens 340.000000 NaN NaN NaN
http_req_sending ms 0.504841 0.230543 0.527351 2.798082
http_req_waiting ms 91.130199 23.709262 93.193301 633.795432
http_req_data_received KB 7.954883 7.956055 7.970801 7.971592
http_req_connecting ms 0.257011 0.000000 0.257011 2.338797
http_req_connection_reused ratio 0.900000 1.000000 1.000000 1.000000
http_req_blocked ms 0.000000 0.000000 0.000000 0.000000
http_req_dns_lookup ms 0.042871 0.000000 0.042871 0.390126
http_req_chunks_received count 32.600000 33.000000 34.000000 34.000000
http_req_chunks_sent count 1.000000 1.000000 1.000000 1.000000
time_to_first_output_token ms 91.770545 23.977319 93.899204 636.935233
http_req_duration ms 258.657125 207.090978 260.330959 674.649103
http_req_data_sent KB 0.270020 0.269043 0.279687 0.282852
http_req_receiving ms 167.022085 182.795457 183.395102 183.933767
e2e_output_token_throughput tokens/sec/user 152.770842 164.113321 168.852245 169.710317
osl_mismatch_diff_pct % 6.250000 6.250000 9.375000 9.375000
prefill_throughput_per_user tokens/sec/user 1183.218498 1334.595197 1370.633072 1376.052063
http_req_connection_overhead ms 0.299882 0.000000 0.299882 2.728923
http_req_total ms 258.957006 207.090978 260.630841 677.378026
osl_mismatch_count requests 8.000000 NaN NaN NaN
total_token_throughput tokens/sec 253.534960 NaN NaN NaN
Benchmark metrics reference
This table breaks down the metrics reported by the benchmarking service.
Metric
Description
Available statistics
request_throughput
Completed requests per second
avg, p50, p90, p99
time_to_first_token
Time from request to first streamed token
avg, p50, p90, p99
request_latency
End-to-end request latency
avg, p50, p90, p99
output_token_throughput
Output tokens per second (system-wide aggregate)
avg only
Compare LMI and vLLM, then deploy the preferred option
Not sure which inference framework suits your model? Run two recommendation jobs in parallel, one for LMI and one for vLLM, compare the top results, and deploy from whichever wins.
import time, uuid
from concurrent.futures import ThreadPoolExecutor, as_completed
from sagemaker.core.jumpstart.configs import JumpStartConfig
from sagemaker.serve import ModelBuilder
from sagemaker.train.configs import Compute
def build_for(framework):
bid = f"{int(time.time())}-{uuid.uuid4().hex[:6]}"
mb = ModelBuilder.from_jumpstart_config(
jumpstart_config=JumpStartConfig(model_id=MODEL_ID),
compute=Compute(instance_type=INSTANCE_TYPE),
role_arn=ROLE,
)
mb.build(model_name=f"demo-fw-{framework.lower()}-{bid}")
return mb
mb_lmi = build_for("LMI")
mb_vllm = build_for("VLLM")
def run_rec(mb, framework):
return mb.generate_deployment_recommendations(
tokenizer="google/gemma-4-e2b-it",
concurrency=1, request_count=10,
prompt_input_tokens_mean=32, output_tokens_mean=32,
streaming=True,
performance_target=PerformanceTarget.TTFT_MS,
instance_types=[INSTANCE_TYPE],
advanced_optimization=False,
framework=framework,
role_arn=ROLE, wait=True,
)
# Run both recommendation jobs in parallel
with ThreadPoolExecutor(max_workers=2) as ex:
futures = {
ex.submit(run_rec, mb_lmi, "LMI"): InferenceFramework.LMI,
ex.submit(run_rec, mb_vllm, "VLLM"): InferenceFramework.VLLM,
}
for fut in as_completed(futures):
print(f"[{futures[fut]}] rec job complete.")
print("\n=== LMI ==="); print(mb_lmi.recommendations)
print("\n=== vLLM ==="); print(mb_vllm.recommendations)
# Pick the framework whose top recommendation has the higher throughput
winner_fw, winner_mb = max(
[("LMI", mb_lmi), ("VLLM", mb_vllm)],
key=lambda x: x[1].recommendations.best.expected_performance.request_throughput.avg or 0,
)
print(f"Winner: {winner_fw} recipe={winner_mb.recommendations.best.recommendation_spec_name}")
fw_endpoint = winner_mb.deploy(
endpoint_name=f"demo-fw-winner-ep-{uuid.uuid4().hex[:8]}",
role=ROLE, wait=True, auto_approve=True,
)
print(f"Deployed: {fw_endpoint.endpoint_name} ({fw_endpoint.endpoint_status})")
In this post, you walked through the complete inference optimization journey. You generated deployment recommendations that explore instance and framework configurations, interpreted the ranked results, and deployed the optimal configuration. You also validated performance with benchmarks under realistic load and compared serving frameworks head-to-head. With the Amazon SageMaker Python SDK integration for generative AI inference recommendations, this entire workflow lives in a single notebook, which removes the context-switching between console UIs and API calls.
Jat AI
Stay informed with the latest in artificial intelligence. Jat AI News Portal is your go-to source for AI trends, breakthroughs, and industry analysis. Connect with the community of technologists and business professionals shaping the future.
![KAIZEN Locations & Map Guide – Boss, Chest, Weapon, Style [KASHIMO]](https://www.destructoid.com/wp-content/uploads/2026/02/kaizen-locations-and-map-guide.jpg)
