> ## Documentation Index
> Fetch the complete documentation index at: https://alliancebioversityciat-alliance-risk-analysis-tool.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitoring

> Monitor Lambda functions, CloudWatch logs, and track system health

Monitor the CGIAR Risk Intelligence Tool's Lambda functions, CloudWatch logs, and system metrics to ensure reliable operation.

## CloudWatch Log Groups

All Lambda functions automatically create CloudWatch log groups with structured logging:

### API Lambda

**Log Group:** `/aws/lambda/alliance-risk-api`

* HTTP request/response logs
* Authentication events (Cognito token verification)
* Database connection status
* Error stack traces

### Worker Lambda

**Log Group:** `/aws/lambda/alliance-risk-worker`

* Job processing lifecycle (PENDING → PROCESSING → COMPLETED/FAILED)
* Bedrock model invocations
* SQL execution logs (migrations via `run-sql` action)
* Job retry attempts and failures

## Accessing CloudWatch Logs

### AWS Console

1. Navigate to **CloudWatch** → **Log Groups**
2. Select `/aws/lambda/alliance-risk-api` or `/aws/lambda/alliance-risk-worker`
3. Click **Search all log streams**
4. Use filter patterns (see below)

### AWS CLI

```bash theme={null}
# Tail API Lambda logs (last 10 minutes)
aws logs tail /aws/lambda/alliance-risk-api --follow --since 10m

# Tail Worker Lambda logs with filter
aws logs tail /aws/lambda/alliance-risk-worker --follow \
  --filter-pattern "ERROR"

# Get logs for specific time range
aws logs filter-log-events \
  --log-group-name /aws/lambda/alliance-risk-api \
  --start-time $(date -d '1 hour ago' +%s)000 \
  --filter-pattern "statusCode>=500"
```

## Log Filter Patterns

### Error Detection

```text theme={null}
# All errors (NestJS Logger)
[level=error]

# HTTP 5xx errors
"statusCode":5??

# Specific error types
"PrismaClientKnownRequestError"
"ThrottlingException"
"CircuitOpenError"
```

### Authentication Issues

```text theme={null}
# Cognito errors
"NotAuthorizedException" OR "UserNotFoundException"

# Token verification failures
"JwtAuthGuard" "denied"

# Admin permission failures
"AdminGuard" "403"
```

### Performance Monitoring

```text theme={null}
# Slow Bedrock calls (>10s)
"Bedrock" "processingTime" { $.processingTime > 10000 }

# Database connection issues
"Database connected" OR "Database disconnected"

# Lambda cold starts
"INIT_START"
```

## Key Metrics to Monitor

### Lambda Metrics (CloudWatch)

| Metric | Threshold | Action |
| - | - | - |
| **Invocations** | Baseline | Track request volume trends |
| **Errors** | >1% of invocations | Investigate error logs |
| **Duration** | >25s (API), >14min (Worker) | Optimize or increase timeout |
| **Throttles** | >0 | Increase concurrency limit |
| **ConcurrentExecutions** | Near account limit | Request limit increase |
| **IteratorAge** | N/A (not using streams) | - |

### API Lambda Alerts

```text theme={null}
Metric: Errors
Threshold: Sum > 10 in 5 minutes
Action: SNS notification to operations team
```

```text theme={null}
Metric: Duration
Threshold: Average > 25000ms (approaching 29s timeout)
Action: Investigate slow queries/Bedrock calls
```

### Worker Lambda Alerts

```text theme={null}
Metric: Errors
Threshold: Sum > 5 in 15 minutes
Action: Check job queue, review Bedrock throttling
```

```text theme={null}
Metric: Throttles
Threshold: Sum > 0
Action: Increase reserved concurrency
```

## Log Analysis Examples

### Find Failed Jobs

```bash theme={null}
aws logs filter-log-events \
  --log-group-name /aws/lambda/alliance-risk-worker \
  --filter-pattern '"Job" "failed"' \
  --query 'events[*].[timestamp,message]' \
  --output text
```

### Track Bedrock Usage

```bash theme={null}
# Count Bedrock invocations by model
aws logs filter-log-events \
  --log-group-name /aws/lambda/alliance-risk-worker \
  --filter-pattern '"Invoking Bedrock model"' \
  --start-time $(date -d '1 day ago' +%s)000 | \
  jq -r '.events[].message' | \
  grep -oP 'modelId: \K[^"]+' | \
  sort | uniq -c
```

### Identify Prisma Connection Issues

```bash theme={null}
aws logs filter-log-events \
  --log-group-name /aws/lambda/alliance-risk-api \
  --filter-pattern '"Prisma" ("code" OR "meta")' \
  --max-items 50
```

## Structured Logging

The application uses NestJS Logger with structured output:

### Log Levels

| Level | Usage | CloudWatch Filter |
| - | - | - |
| `error` | Unhandled exceptions, critical failures | `[level=error]` |
| `warn` | HTTP 4xx, retryable failures | `[level=warn]` |
| `log` | HTTP requests, job lifecycle | `[level=log]` |
| `debug` | Detailed diagnostics (dev only) | `[level=debug]` |

### Sample Log Entries

**Successful Request:**

```json theme={null}
{
  "level": "log",
  "timestamp": "2026-03-04T14:23:45.123Z",
  "context": "RouterExplorer",
  "message": "GET /api/prompts/section/parser 200"
}
```

**Prisma Error:**

```json theme={null}
{
  "level": "error",
  "timestamp": "2026-03-04T14:23:45.123Z",
  "context": "PromptsService",
  "message": "Prisma error code=P2002 meta={\"target\":[\"slug\"]}"
}
```

**Bedrock Invocation:**

```json theme={null}
{
  "level": "log",
  "timestamp": "2026-03-04T14:23:45.123Z",
  "context": "BedrockService",
  "message": "Invoking Bedrock model: anthropic.claude-3-5-sonnet-20241022-v2:0"
}
```

## Setting Up CloudWatch Alarms

### Create Error Rate Alarm (AWS CLI)

```bash theme={null}
aws cloudwatch put-metric-alarm \
  --alarm-name alliance-risk-api-errors \
  --alarm-description "Alert when API Lambda error rate exceeds 1%" \
  --metric-name Errors \
  --namespace AWS/Lambda \
  --statistic Sum \
  --period 300 \
  --threshold 10 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 1 \
  --dimensions Name=FunctionName,Value=alliance-risk-api \
  --treat-missing-data notBreaching
```

### Create Duration Alarm

```bash theme={null}
aws cloudwatch put-metric-alarm \
  --alarm-name alliance-risk-api-slow \
  --alarm-description "Alert when API Lambda approaches timeout" \
  --metric-name Duration \
  --namespace AWS/Lambda \
  --statistic Average \
  --period 300 \
  --threshold 25000 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 2 \
  --dimensions Name=FunctionName,Value=alliance-risk-api
```

## Lambda Cold Starts

### Identifying Cold Starts

Look for `INIT_START` messages in logs:

```bash theme={null}
aws logs filter-log-events \
  --log-group-name /aws/lambda/alliance-risk-api \
  --filter-pattern "INIT_START" \
  --start-time $(date -d '1 hour ago' +%s)000
```

### Cold Start Duration

Cold starts add 2-5 seconds to first invocation:

* NestJS module initialization (\~1-2s)
* Prisma client generation (\~500ms-1s)
* Database connection pool setup (\~500ms)

### Mitigation Strategies

1. **Provisioned Concurrency** (costs \$\$\$):
   ```bash theme={null}
   aws lambda put-provisioned-concurrency-config \
     --function-name alliance-risk-api \
     --provisioned-concurrent-executions 2
   ```

2. **Keep-Warm Schedule** (EventBridge):
   * Invoke Lambda every 5 minutes with warmup event
   * Filter in code: `if (event.source === 'warmup') return`

3. **Accept Cold Starts** (recommended for MVP):
   * Cold starts are infrequent with moderate traffic
   * First request after inactivity will be slower

## Database Monitoring

### RDS CloudWatch Metrics

| Metric | Threshold | Action |
| - | - | - |
| **CPUUtilization** | >80% sustained | Upgrade instance type |
| **FreeableMemory** | Less than 100MB | Upgrade instance type |
| **DatabaseConnections** | >80% of max | Investigate connection leaks |
| **ReadLatency** | Greater than 100ms | Add indexes, optimize queries |
| **WriteLatency** | Greater than 100ms | Check disk I/O, upgrade storage |

### Connection Pool Monitoring

Prisma uses a connection pool per Lambda instance. Check logs for:

```text theme={null}
"Database connected"  → Successful pool creation
"Database disconnected" → Clean shutdown
```

**Warning Signs:**

* Multiple "Database connected" per invocation (connection leak)
* Lambda timeouts with no error (event loop not draining)

## Bedrock Monitoring

### Throttling Detection

Bedrock has model-specific rate limits. Check for:

```bash theme={null}
aws logs filter-log-events \
  --log-group-name /aws/lambda/alliance-risk-worker \
  --filter-pattern "ThrottlingException"
```

### Circuit Breaker Status

The app uses a circuit breaker for Bedrock calls:

* **CLOSED:** Normal operation
* **OPEN:** 3+ consecutive failures, all requests fail fast
* **HALF\_OPEN:** Testing recovery, 2 successes → CLOSED

Look for: `"Circuit breaker is open"` in logs

### Retry Behavior

Bedrock calls retry 3 times with exponential backoff:

* Base delay: 200ms
* Max delay: 5s
* Retries: `ThrottlingException`, `ServiceUnavailableException`

## Next Steps

* [Troubleshooting Guide](https://alliancebioversityciat-alliance-risk-analysis-tool.mintlify.site/deployment/troubleshooting.md) — Common errors and solutions
* [Infrastructure Setup](https://alliancebioversityciat-alliance-risk-analysis-tool.mintlify.site/deployment/infrastructure) — CloudFormation stack details


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.