How to Configure Cloud Authentication for RAG Evaluation: Step-by-Step Guide for AWS IAM, Azure SPN & GCP Service Accounts

Evaluating production Retrieval-Augmented Generation (RAG) pipelines requires reliable, programmatic authentication. Unlike simple prototype scripts that rely on short-lived developer browser sessions or personal API keys, enterprise RAG evaluation demands robust, least-privilege cloud credentials to query vector search indices, retrieve grounded chunks, and invoke foundation generative models.

The RAG Pipeline Evaluation Workbench supports three premier enterprise cloud authentication patterns with Zero-Token Pre-flight Connectivity Verification:

• Amazon Web Services (AWS): Programmatic IAM User credentials (Access Key ID, Secret Access Key, and optional Session Token) validated via AWS STS (Security Token Service).
• Microsoft Azure: App Registration Service Principal (Tenant ID, Client ID, Client Secret) validated via Microsoft Entra ID OAuth2 token exchange.
• Google Cloud Platform (GCP): Service Account (Project ID, Location, and Service Account JSON Key) validated via Google OAuth2 cryptographic JWT token refresh.

This comprehensive guide walks you step-by-step through creating and provisioning these credentials with the exact least-privilege permissions required for Bedrock Knowledge Bases, Azure AI Search & Foundry, and Google Vertex AI Search & Gemini.

1. AWS Bedrock Knowledge Bases: Creating a Programmatic IAM User

Amazon Bedrock Knowledge Bases automate ingestion, chunking, embedding generation, and vector retrieval across Amazon OpenSearch Serverless, Pinecone, or Aurora PostgreSQL. To evaluate Bedrock RAG pipelines, you need an AWS IAM user with permissions to query knowledge bases and invoke foundation models.

Step 1: Create the IAM User in AWS Console

1. Sign in to the AWS Management Console and navigate to IAM (Identity and Access Management).
2. In the left navigation sidebar, click Users and then click Create user.
3. Enter a descriptive user name (for example, sylvaera-bedrock-rag-evaluator).
4. Under Select AWS access type, leave management console access unchecked (this user only needs programmatic API access). Click Next.

Step 2: Attach the Required Permissions Policy

Select Attach policies directly and click Create policy. Select the JSON tab and paste the following least-privilege policy document:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "BedrockKnowledgeBaseRetrieval",
      "Effect": "Allow",
      "Action": [
        "bedrock:Retrieve",
        "bedrock:RetrieveAndGenerate",
        "bedrock:GetKnowledgeBase"
      ],
      "Resource": "arn:aws:bedrock:*:*:knowledge-base/*"
    },
    {
      "Sid": "BedrockModelInvocation",
      "Effect": "Allow",
      "Action": [
        "bedrock:InvokeModel",
        "bedrock:InvokeModelWithResponseStream"
      ],
      "Resource": [
        "arn:aws:bedrock:*::foundation-model/*",
        "arn:aws:bedrock:*:*:inference-profile/*"
      ]
    },
    {
      "Sid": "ZeroTokenSTSVerification",
      "Effect": "Allow",
      "Action": [
        "sts:GetCallerIdentity"
      ],
      "Resource": "*"
    }
  ]
}

Name the policy SylvaeraBedrockRagEvaluationPolicy, click Create policy, return to the user creation tab, search for the newly created policy, check the checkbox, and click Next and Create user.

Step 3: Generate Programmatic Access Keys

1. Click on the newly created user (sylvaera-bedrock-rag-evaluator).
2. Navigate to the Security credentials tab.
3. Scroll down to the Access keys section and click Create access key.
4. Select the use case Application running outside AWS (or Command Line Interface (CLI)), acknowledge the recommendation, and click Next.
5. Add a tag description (e.g. Sylvaera RAG Evaluation Workbench) and click Create access key.
6. Copy both the Access key ID (starts with AKIA...) and the Secret access key. Download the .csv file for secure offline backup.

CLI Automation (Optional)

If you prefer automated scripting via AWS CLI:

# 1. Create IAM User
aws iam create-user --user-name sylvaera-bedrock-rag-evaluator

# 2. Attach Bedrock RAG Policy
aws iam put-user-policy \
  --user-name sylvaera-bedrock-rag-evaluator \
  --policy-name BedrockRagEvaluationPolicy \
  --policy-document file://bedrock-policy.json

# 3. Generate Access Key
aws iam create-access-key --user-name sylvaera-bedrock-rag-evaluator

2. Azure AI Search & AI Foundry: Creating an App Registration (SPN)

Microsoft Azure RAG architectures typically couple Azure AI Search (for BM25 lexical + dense vector + Bing Semantic Ranker hybrid retrieval) with Azure AI Foundry or Azure OpenAI Service (for model answer generation). Azure uses Microsoft Entra ID (formerly Azure Active Directory) Service Principals (SPN) for secure server-to-server authorization.

Step 1: Register an Application (SPN) in Microsoft Entra ID

1. Sign in to the Azure Portal.
2. Search for and select Microsoft Entra ID.
3. In the left navigation menu under Manage, click App registrations, then click New registration.
4. Enter a name (for example, sylvaera-rag-evaluator-spn).
5. Under Supported account types, select Accounts in this organizational directory only (Single tenant).
6. Leave Redirect URI blank and click Register.
7. In the Overview screen of your registered app, copy and save:
   • Application (client) ID (GUID format: 00000000-0000-0000-0000-000000000000)
   • Directory (tenant) ID (GUID format: 00000000-0000-0000-0000-000000000000)

Step 2: Create a Client Secret

1. In the left sidebar of the app registration, select Certificates & secrets.
2. Click the Client secrets tab, then click New client secret.
3. Enter a description (e.g. Sylvaera RAG Benchmark Secret) and set an expiration period (e.g. 180 days or 365 days).
4. Click Add.
5. CRITICAL: Immediately copy the secret Value (not the Secret ID). The value will be masked permanently once you leave the page.

Step 3: Assign Azure RBAC Roles

Your Service Principal needs two role assignments to retrieve search chunks and invoke the generative model:

A. Permission on Azure AI Search Service

1. Navigate to your Azure AI Search resource in the Azure Portal.
2. In the left sidebar, click Access control (IAM), then click Add > Add role assignment.
3. Select the role Search Index Data Reader (grants read access to search index documents and vector chunks).
4. Under Assign access to, select User, group, or service principal, click Select members, search for sylvaera-rag-evaluator-spn, select it, and click Review + assign.
5. Note: If your test setup also queries service-level metrics or index schema definitions, also assign the role Search Service Contributor.

B. Permission on Azure AI Foundry / Azure OpenAI Service

1. Navigate to your Azure AI Foundry hub or Azure OpenAI Service resource.
2. In the left sidebar, click Access control (IAM) > Add role assignment.
3. Select the role Cognitive Services OpenAI User (grants permissions to generate completions with deployed models such as GPT-4o or Claude 3.5 Sonnet on Foundry).
4. Select your Service Principal sylvaera-rag-evaluator-spn and click Review + assign.

CLI Automation (Optional)

# 1. Create Service Principal
SPN_JSON=$(az ad sp create-for-rbac --name "sylvaera-rag-evaluator-spn" --skip-assignment)

# 2. Extract credentials
CLIENT_ID=$(echo $SPN_JSON | jq -r .appId)
CLIENT_SECRET=$(echo $SPN_JSON | jq -r .password)
TENANT_ID=$(echo $SPN_JSON | jq -r .tenant)

# 3. Assign Search Index Data Reader Role
az role assignment create \
  --assignee $CLIENT_ID \
  --role "Search Index Data Reader" \
  --scope "/subscriptions/{SUB_ID}/resourceGroups/{RG}/providers/Microsoft.Search/searchServices/{SEARCH_NAME}"

# 4. Assign Cognitive Services OpenAI User Role
az role assignment create \
  --assignee $CLIENT_ID \
  --role "Cognitive Services OpenAI User" \
  --scope "/subscriptions/{SUB_ID}/resourceGroups/{RG}/providers/Microsoft.CognitiveServices/accounts/{AI_NAME}"

3. Google Cloud Vertex AI Search: Creating a Service Account (SA)

Google Cloud Vertex AI Search (formerly Gen App Builder / Enterprise Search) connects data stores (PDFs, BigQuery, Google Drive, cloud buckets) with Gemini generative models. Google uses Service Accounts (SA) with private key cryptographic signing for machine-to-machine authorization.

Step 1: Create the Service Account in Google Cloud Console

1. Sign in to the Google Cloud Console.
2. Select your target GCP project (e.g. sylvaera-rag-production).
3. Navigate to IAM & Admin > Service Accounts.
4. Click Create Service Account at the top of the page.
5. Enter a Service account name (e.g. sylvaera-vertex-rag-sa) and description (e.g. RAG Evaluation Workbench retrieval and model invocation account).
6. Click Create and continue.

Step 2: Grant Required IAM Roles

In the Grant this service account access to project step, add the following two roles:

• Role 1: Discovery Engine Viewer (roles/discoveryengine.viewer) — Allows querying Vertex AI Search Data Stores and serving configs to retrieve grounded context snippets. (If your evaluation pipeline manages datastores dynamically, choose Discovery Engine Editor).
• Role 2: Vertex AI User (roles/aiplatform.user) — Allows invoking Vertex AI publisher models (Gemini 2.0 Flash, Gemini 1.5 Pro) for synthesis and scoring.

Click Continue and then click Done.

Step 3: Generate and Download the JSON Private Key

1. In the Service Accounts list, click on your newly created service account (sylvaera-vertex-rag-sa@...).
2. Select the Keys tab.
3. Click Add key > Create new key.
4. Ensure JSON key type is selected, and click Create.
5. A .json key file will download automatically to your computer.
6. Open the downloaded JSON file in a text editor (e.g. VS Code, Notepad). The file contains fields like project_id, private_key, client_email, etc.
7. Copy the entire JSON string and paste it directly into the Service Account JSON Key (SA) field in the Sylvaera project modal.

CLI Automation (Optional)

# 1. Create Service Account
gcloud iam service-accounts create sylvaera-vertex-rag-sa \
  --display-name="Sylvaera Vertex RAG Evaluator"

# 2. Assign Discovery Engine Viewer Role
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member="serviceAccount:sylvaera-vertex-rag-sa@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/discoveryengine.viewer"

# 3. Assign Vertex AI User Role
gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \
  --member="serviceAccount:sylvaera-vertex-rag-sa@YOUR_PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/aiplatform.user"

# 4. Generate JSON Key File
gcloud iam service-accounts keys create sa-key.json \
  --iam-account=sylvaera-vertex-rag-sa@YOUR_PROJECT_ID.iam.gserviceaccount.com

4. Why Zero-Token Pre-flight Verification Matters

In standard evaluation tools, verifying credentials requires sending test prompts to expensive LLMs, burning tokens before any real benchmarking even begins. The Sylvaera Evaluation Engine implements Zero-Token Cryptographic Pre-flight Verification:

5. Security & Credential Protection Best Practices

• Client-Side Masking: Sensitive secret keys are displayed only once during input and are masked with •••••••• in the browser UI to prevent shoulder-surfing.
• Encrypted Backend Persistence: Credentials stored in the Sylvaera DynamoDB state store are encrypted at rest using AWS KMS.
• Least-Privilege Scoping: Never use administrative or root account keys. Always restrict the IAM user, SPN, or Service Account strictly to the resource scopes of your Knowledge Bases and target foundation models.
• Credential Rotation: When rotating secrets in AWS IAM or Azure Entra ID, you can update credentials seamlessly via the Edit Project button on any existing evaluation project.

Ready to benchmark your knowledge retrieval stack? Head over to the RAG Pipeline Evaluation Workbench, select your cloud provider, enter your IAM / SPN / SA credentials, click Test Connection, and start evaluating!