SQS to Database Service

This service provides API endpoints to poll messages from AWS SQS and store them in a database.

API Endpoints

1. Poll SQS Messages (Batch Processing)

Request Body

{
  "batchSize": 1000,  // Target number of messages to process (default: 1000)
  "maxMessages": 10   // Legacy parameter (ignored in batch mode)
}

Features

• Batch Processing: Collects messages in batches up to batchSize
• Retry Logic: Retries up to 5 times when getting partial responses
• Robust Collection: Doesn't give up on first partial response
• Atomic Operations: Inserts all messages to DB before deleting from SQS

Response

{
  "processed": 847,           // Number of successfully processed messages
  "failed": 0,               // Number of failed messages
  "totalMessagesReceived": 847, // Total messages received from SQS
  "batchSize": 847,          // Actual batch size processed
  "message": "Successfully processed 847 messages in batch"
}

2. Get SQS Queue Statistics

Response

{
  "queueUrl": "https://sqs...",
  "approximateNumberOfMessages": 447,
  "approximateNumberOfMessagesNotVisible": 0,
  "approximateNumberOfMessagesDelayed": 0,
  "totalMessages": 447,
  "createdTimestamp": "2024-01-01T00:00:00.000Z",
  "lastModifiedTimestamp": "2024-01-01T12:00:00.000Z",
  "visibilityTimeout": 30,
  "maximumMessageSize": 262144,
  "messageRetentionPeriod": 345600,
  "delaySeconds": 0,
  "receiveMessageWaitTimeSeconds": 0,
  "timestamp": "2024-01-01T12:00:00.000Z"
}

3. Health Check

Response

{
  "status": "healthy",
  "timestamp": "2024-01-01T12:00:00.000Z",
  "uptime": 3600,
  "environment": "production",
  "checks": {
    "database": "healthy",
    "sqs": "configured"
  },
  "responseTime": 45
}

Configuration

The service uses the following constants for SQS operations:

• MAX_MESSAGES_PER_CALL: 10 (SQS limit per ReceiveMessage)
• MAX_PARTIAL_RESPONSES: 5 (retry attempts for partial responses)
• MAX_EMPTY_RESPONSES: 3 (stop after consecutive empty responses)
• WAIT_TIME_SECONDS: 5 (long polling wait time)

Test Scripts

• ./test-retry-behavior.sh - Test the new retry logic
• ./test-batch-api.sh [batchSize] - Test batch processing
• ./test-parallel-api.sh - Test parallel API calls