Guidance for an India-Controlled Generative AI Gateway on AWS

#Overview

Bank applications can already integrate with generative AI independently. The resulting problem is not simply model access: every application can make different decisions about credentials, models, regions, sensitive data, budgets, retries and logging. The bank then has no consistent answer to who used a model, why that model was selected, what controls ran, what the call cost or why it was refused.

This guidance introduces one governed admission point. Browser users authenticate through Amazon Cognito, application traffic enters through a protected API, and private Fargate tasks run the banking policy middleware and LiteLLM. Only approved direct Amazon Bedrock model identifiers available in ap-south-1 can be invoked. An optional bank-hosted SageMaker route and a cold ap-south-2 recovery stack are conditional extensions, not hidden or automatic fallbacks.

The problem in one sentence

Teams can use AI today, but the bank cannot consistently prove who used which model, what controls ran, where inference occurred, what it cost or why a request was refused.

Design principles

Target use cases

• Customer-information assistant: Answers general product and service questions using synthetic demonstration data.
• Employee assistant: Uses approved model aliases with team budgets and stricter access controls.
• Developer integration: Provides an OpenAI-compatible API with normalized errors and request IDs.
• AI governance: Gives operations teams one place to inspect model use, refusals, spend and control effectiveness.

#Features and benefits

The published AWS guidance positions LiteLLM as a broad multi-provider gateway. This design keeps the capabilities that are useful for the bank, constrains features that could violate the regional boundary, and adds controls that remain independent of LiteLLM configuration or licensing.

LiteLLM responsibility versus bank responsibility

#Architecture overview

The static portal and inference API use separate paths. CloudFront and a CloudFront-scope WAF web ACL deliver only the portal from a private S3 origin using Origin Access Control. The browser then calls a Regional REST API at api.aigateway.ajitsury.people.aws.dev, protected by a Regional WAF web ACL and a Cognito user-pool authorizer. API Gateway connects through VPC Link V2 over HTTPS to private DNS and an internal load balancer. TLS terminates at the ALB; the final security-group-restricted hop to the task uses HTTP inside the isolated VPC. Fargate tasks have no public IP, NAT Gateway or internet route.

Reference architectureOne governed path from browser to approved regional model

Primary Region · ap-south-1

Expand diagram

Architecture steps

1. 1

   A user opens aigateway.ajitsury.people.aws.dev. Route 53 resolves it to CloudFront, whose WAF-protected distribution reads the dashboard from a private S3 bucket through signed OAC requests.

2. 2

   The single-page application redirects the user to Amazon Cognito and completes an authorization-code flow with PKCE. Static HTML can load publicly; business data and inference remain inaccessible without a valid token.

3. 3

   The dashboard sends POST /v1/responses to the separate Regional API hostname using the short-lived Cognito token.

4. 4

   The Regional WAF inspects the API request. API Gateway REST validates the request and Cognito token, then forwards it through VPC Link V2 to an internal Application Load Balancer.

5. 5

   The ALB sends traffic to the private Fargate service. The demo profile runs one task to reduce baseline cost; the resilience profile runs two tasks across separate Availability Zones.

6. 6

   The banking middleware resolves identity and model alias, checks regional policy, applies deterministic screening, reserves budget and evaluates rate limits.

7. 7

   The policy gateway translates the stateless Responses request to LiteLLM’s Chat Completions interface. LiteLLM invokes either an approved direct Bedrock model through the Bedrock Runtime VPC endpoint or the explicit bank-self-hosted SageMaker route when that optional endpoint is enabled.

8. 8

   For Bedrock routes, the gateway uses a pinned Guardrail for pre-screening and model input/output evaluation. IAM prevents an approved model invocation from silently omitting the required Guardrail.

9. 9

   PostgreSQL is authoritative for teams, budget reservations, usage evidence and the archive outbox. Alias and policy configuration is version-controlled with the deployment. Redis holds short-lived rate and concurrency counters plus eligible cache entries.

10. 10

    Metadata-only evidence is archived to an encrypted, versioned, Object Lock-enabled S3 bucket while CloudWatch receives operational metrics and content-free logs.

#Browser experience

The proof of concept includes a bank-branded dashboard rather than exposing LiteLLM’s administrator interface as the primary experience. The LiteLLM UI remains an optional, separately protected operator tool.

Opening the dashboard performs authenticated health, model-catalogue and summary reads only. A model is invoked only after the user submits a banking question or explicitly runs a developer scenario.

Production separation

#Request flow

Traffic flowWhat happens to one end-user request

Fail closed before inference

Expand diagram

OpenAI-compatible Responses request

{
  "model": "bank-standard",
  "input": [
    {
      "role": "user",
      "content": [
        {
          "type": "input_text",
          "text": "What documents are required for a savings account?"
        }
      ]
    }
  ],
  "max_output_tokens": 500,
  "store": false,
  "stream": false
}

Admission sequence

#Model routing and failover

Applications request bank aliases rather than provider model identifiers. The catalogue is intentionally unresolved until account discovery confirms directly invocable models, quotas and terms in Mumbai. SageMaker appears only as an optional bank-hosted route; it is not an automatic substitute for a Bedrock model.

Routing policyBank aliases resolve only to approved direct regional models

No inference profiles

Expand diagram

Routing decision order

1. Validate the identity and team.
2. Confirm the alias is allowed for that identity.
3. Confirm that every candidate is a direct ap-south-1 Bedrock model resource or the specifically approved SageMaker endpoint.
4. Apply token, budget, rate and concurrency limits.
5. Select a healthy primary candidate.
6. Use an equivalent same-region fallback only for retryable failures before response delivery.
7. Fail closed if no approved equivalent exists.

#AWS services in this guidance

Runtime VPC endpoints

The final endpoint list is derived from real runtime calls rather than copied from the original guidance. The implemented set is Bedrock Runtime, ECR API, ECR Docker, CloudWatch Logs and Secrets Manager interface endpoints, plus an S3 gateway endpoint. Metrics use Embedded Metric Format through CloudWatch Logs. SageMaker Runtime is added only if the optional hosted-model route is enabled.

#Plan your deployment

Selected deployment profile

Differences from the published AWS guidance

Prerequisites

• ✓
  Architecture boundary agreed

  Mumbai direct inference for the MVP; no global or geographic inference profiles. Hyderabad remains an explicit cold-DR phase.

• ✓
  Local Docker not required

  AWS CodeBuild will build and push the runtime image.

• !
  Target account and deployment role

  Confirm the AWS account, profile and least-privilege deployment permissions.

• !
  Direct regional model catalogue

  List directly invocable models, terms and quotas in ap-south-1.

• !
  Domain and certificate

  Verify ownership and certificates for aigateway.ajitsury.people.aws.dev and api.aigateway.ajitsury.people.aws.dev.

• !
  Bank policy decisions

  Confirm PII actions, budget dimensions, evidence retention and administrator roles.

#Security

Security is implemented as independent layers. No single Guardrail, network route, LiteLLM setting or dashboard permission is treated as sufficient by itself.

Guardrail strategy

#Banking governance

The gateway provides technical controls and evidence for the bank’s governance process. It is not presented as a compliance certificate and cannot replace model ownership, legal review, independent validation, risk acceptance or customer-remediation processes.

#Cost controls

Model spending and infrastructure spending are controlled separately. The bank middleware and PostgreSQL enforce authoritative application budgets; LiteLLM supplies usage data and Redis enforces short-lived rate and concurrency limits. CloudWatch monitors runtime behavior; an AWS Budget can be added after the bank supplies its notification recipients and approved threshold. No monthly total is stated until traffic, token, retention, endpoint and availability assumptions are agreed.

Cost-control sequence

1. Estimate input tokens and apply the explicit maximum output.
2. Calculate worst-case cost from the versioned approved-model price map.
3. Atomically reserve that amount in a PostgreSQL transaction.
4. Invoke only when the reservation succeeds.
5. Settle actual usage and release the unused reservation.
6. Persist zero-cost refusals and cache hits as evidence.

#Deploy the guidance

AWS CDK defines the infrastructure and runs strict synthesis with AWS Solutions validation. CodeBuild tests, builds and pushes the policy-gateway image so that the local machine does not require Docker. ECR scan-on-push is enabled; the bank can enable Amazon Inspector enhanced scanning as an account-level control.

Acceptance criteria

• ○
  The custom HTTPS URL loads and redirects to Cognito before dashboard data or inference is accessible.
• ○
  The S3 portal origin is private, uses OAC with SigningBehavior=always, and has no wildcard resource-policy principal.
• ○
  No Lambda function URL or AuthType: NONE resource is created.
• ○
  The ALB and Fargate tasks are private and have no general internet route.
• ○
  Unauthenticated inference returns 401 and disallowed aliases return 403.
• ○
  Global, geographic and arbitrary model IDs are rejected before provider invocation.
• ○
  Synthetic PII is blocked or masked without appearing in operational evidence.
• ○
  Rate, budget, cache, same-region failover and optional SageMaker-route scenarios create matching evidence.
• ○
  Every successful model call records the resolved direct model and ap-south-1.
• ○
  CloudWatch receives gateway, token, cost, policy, cache and failover metrics; alarms cover API 5xx responses and unhealthy private targets.

#Demonstration script

The demo starts with the business experience and then reveals the architecture and controls. All prompts, identities and identifiers are synthetic.

#Known limitations

#Resilience and recovery

AvailabilityOptional Multi-AZ resilience profile inside Mumbai

No cross-region inference

Expand diagram

The VPC and ALB span two Availability Zones. The default demo profile deliberately uses one Fargate task, one PostgreSQL instance and one Valkey node to reduce baseline cost. Setting productionResilience=true enables two tasks, Multi-AZ PostgreSQL and a Valkey replica with Multi-AZ failover. A full regional outage remains an accepted limitation until the bank approves and tests a separate India-region recovery design.

#Verification sources

The design uses primary AWS and LiteLLM documentation. Model availability, quotas, account entitlements, exact service endpoint requirements and commercial LiteLLM features must be verified again during account discovery.

• AWS Solutions Library: Guidance for Multi-Provider Generative AI Gateway on AWS
• AWS Solutions Library sample implementation repository
• Amazon Bedrock: Supported foundation models
• Amazon Bedrock: Supported inference APIs and Responses API limitations
• Amazon Bedrock: Inference profile support and destinations
• Amazon Bedrock: Enforce Guardrails with IAM
• Amazon API Gateway: Private integrations and VPC links
• Amazon API Gateway: Cognito user-pool authorizer for REST APIs
• Amazon CloudFront: Restrict S3 origin access with OAC
• Amazon SageMaker AI: Private API and Runtime access through PrivateLink
• Amazon RDS: Cross-Region read replicas
• Amazon S3: Object Lock retention
• AWS CodeBuild: Build and push Docker images
• Amazon Inspector: Enhanced ECR image scanning
• LiteLLM Proxy Server documentation
• LiteLLM retries and fallback documentation
• LiteLLM caching documentation

#Uninstall the guidance

Teardown is performed only after explicit approval and follows dependency order:

1. Disable new user access and inference traffic.
2. Export required evidence and confirm retention obligations.
3. Remove DNS, CloudFront/OAC, API custom-domain and Cognito integration.
4. Delete the API, VPC Link, ALB and Fargate runtime.
5. Remove RDS, Redis and endpoints after approved backups are complete.
6. Empty retained build and evidence buckets only after separate confirmation; Object Lock retention can prevent deletion until the approved date.
7. Delete KMS keys according to the approved waiting period and verify that no chargeable resource remains.

#Notices

• This page describes a synthetic-data proof of concept, not a production banking system.
• Product availability, quotas, model terms, licensing and pricing require account-level verification.
• Open-source dependencies require vulnerability, licence and version review before deployment.
• Third-party foundation models remain subject to provider terms and bank approval.
• No real customer, employee, credential or confidential bank data may be entered into the PoC.

Architecture prototype prepared for solution review. The linked AWS guidance retains its own copyright and licence terms.

Last reviewed: September 21, 2026