How to monitor an SQS → Lambda → DynamoDB pipeline without writing code

An order lands in an SQS queue, a Lambda function processes it and the result is stored in DynamoDB. It is one of the most common patterns on AWS, and one of the quietest to fail: the queue accepts messages even when nobody reads them, the function retries until something works, and the table never complains about what it never received. In this article we walk through how this chain fails, which signal gives each failure away and how to watch it with AAA Sentinel step by step, using example names you can swap for your own.

The example pipeline

We use a fictional shop running this pipeline in account 123456789012, region eu-west-1:

How a pipeline like this fails

Almost every incident in this chain falls into one of five cases:

• Messages pile up: the function is stopped, the event source mapping is disabled or reserved concurrency was left at zero. The queue keeps accepting orders and nobody processes them.
• Poison messages: a message the function cannot handle fails on every attempt. When the visibility timeout runs out it returns to the queue, and once it reaches the redrive policy's maximum receive count it ends up in the DLQ. If the function processes batches without reporting per-message failures (ReportBatchItemFailures), a single bad message makes the whole batch retry.
• Function errors and throttles: exceptions, timeouts or throttling for lack of concurrency. An error spike tends to get noticed; a steady 3% does not.
• Retries that hide the failure: the message fails twice and succeeds on the third try. It never reaches the DLQ and the error rate does not stand out, but the order is stored minutes late.
• Items that are never written: the function finishes fine but does not write. An environment variable pointing at another table, a conditional write that gets dropped, a caught exception that only leaves a log line. No metric moves.

Which signal gives each failure away

CloudWatch already publishes almost everything you need. This is the signal that best catches each case:

Step 1: watch the queue, the DLQ, the function and the table

This part needs no code. In Monitoring → New → Monitor, pick the Cloud resource type, the account you already connected, the region and the resource from the list (or paste its ARN). If the account does not grant Monitoring read access yet, the form says so and offers "Grant the permission". AAA Sentinel reads the metrics with that read-only role, with no agents. Three monitors cover all four resources:

• orders-queue (SQS queue): considered down if there is any message in its dead-letter queue or if the oldest message has gone unconsumed for longer than the maximum age (60 minutes if you leave it empty). The DLQ needs no monitor of its own: the queue monitor finds it in the redrive policy and reads its message count straight from SQS, without waiting for CloudWatch.
• process-order (Lambda function): considered down if 10% or more of its executions fail or are throttled (with at least 10), or if all of them fail when there are between 3 and 9. You tune the percentage in "Failure rate (%)".
• orders (DynamoDB table): considered down if it rejects more than 10 requests for capacity in the period, reads and writes combined. You tune it in "Tolerated capacity rejections"; on a low-traffic on-demand table you can lower it to 0.

Step 2: what a flow does

A flow checks the whole chain with a test transaction. Every 15 minutes at most, AAA Sentinel sends a message to orders-queue with its own traceId (32 hexadecimal characters) and checks that it reaches each step within its deadline. The message carries the traceId in the x-aaa-trace message attribute and in the body as aaaTraceId, alongside a W3C traceparent; on a FIFO queue the test goes in its own message group. The possible steps are a line in CloudWatch Logs, a DynamoDB item, an S3 object or a callback your code makes to a URL of ours, up to five. For this pipeline we use two: the line in the process-order log and the item in orders.

Step 3: prepare the function (a few lines here)

To be honest about the title: watching the resources takes no code, but the flow needs the function to recognise the test. The test runs through your real chain, so your code has to handle it with no real effects (no charges, emails or orders) and leave two marks we can find:

// npm install @aaa-sentinel/health
import { propagateTrace } from '@aaa-sentinel/health'
import { DynamoDBClient } from '@aws-sdk/client-dynamodb'
import { DynamoDBDocumentClient, PutCommand } from '@aws-sdk/lib-dynamodb'

const db = DynamoDBDocumentClient.from(new DynamoDBClient({}))

export const handler = async (event) => {
  for (const record of event.Records) {
    const trace = propagateTrace(record)
    if (trace) {
      trace.log('order received')
      // AAA Sentinel test: no charges or emails, and the traceId in the key so the DynamoDB step can find it.
      await db.send(new PutCommand({
        TableName: process.env.ORDERS_TABLE,
        Item: {
          orderId: `aaa-test-${trace.traceId}`,
          ...trace.fields,
          // The table's TTL attribute: DynamoDB deletes test items on its own.
          expiresAt: Math.floor(Date.now() / 1000) + 86400,
        },
      }))
      continue
    }
    // Your real order logic goes here.
  }
}

import json, os, time
import boto3

table = boto3.resource("dynamodb").Table(os.environ["ORDERS_TABLE"])

def handler(event, context):
    for record in event["Records"]:
        attribute = record.get("messageAttributes", {}).get("x-aaa-trace", {})
        trace_id = attribute.get("stringValue")
        if trace_id:
            print(json.dumps({"msg": "order received", "traceId": trace_id}))
            # AAA Sentinel test: no real effects, and the traceId in the item key.
            table.put_item(Item={"orderId": f"aaa-test-{trace_id}", "aaaTraceId": trace_id, "expiresAt": int(time.time()) + 86400})
            continue
        # Your real order logic goes here.

• A line in its log with the traceId. If the function already logs the incoming event or the message body, the traceId shows up there and this step needs no change.
• An item whose primary key contains the traceId. The DynamoDB step does not search by attribute: it runs a strongly consistent GetItem on the key you define, with {traceId} replaced by each test's id, and only asks for the key attributes, not the item's content. So the key has to be a string that includes the traceId (here, orderId = aaa-test-{traceId}); storing it only in an attribute does not work, and secondary indexes are not searched either.

Step 4: create the flow in the wizard

In Monitoring → New → Flow, the wizard has three steps:

• Where the chain starts: pick the account, the region and, under "What starts the flow", SQS queue; paste the ARN of orders-queue and click "Discover the chain". Only the configuration is read; no test is sent yet. AAA Sentinel follows the event source mapping to process-order and proposes a log step for each function, with the text {traceId} and a 60-second deadline.
• What is checked: review the log step and add the last one under "Add the last step" → DynamoDB item. The table list puts first the tables named in the function's configuration ("appears in your function"): if ORDERS_TABLE points at orders, you will see it at the top. Under "Item key" type orderId and aaa-test-{traceId}. Under "Test content" you can add whatever your code needs so it does not discard the message.
• Name and alerts: the name, the severity and how often to test, every 15 minutes at the least, because each test writes into your chain and makes about twenty reads in your account.

Step 5: the flow role

Sending the message and reading the table is not done with the read role but with a separate one, FinOpsSentinelMonitoringFlow, created from a CloudFormation template in the account's access settings. That role only touches the resources you put in its parameters, and the wizard tells you which ones this flow needs under "Permissions your flow role needs":

• To extend it: open the stack in CloudFormation, click "Update", choose "Use current template", add the ARNs to whatever each parameter already holds, comma-separated, and continue to "Submit". Do not replace the existing ones or other flows will stop working.
• If you encrypt the queue with a customer-managed KMS key, add its ARN to KmsKeyArns too.
• On every run the session is also scoped down to what this flow does: sqs:SendMessage on orders-queue and dynamodb:GetItem on orders, even if the role allows more.
• Logs are read with Monitoring read access, which is where the account grants logs:FilterLogEvents.

How to read the result

The flow page shows the path of the latest test, with its traceId, and a row of dots for each recent run. Every step shows as "Arrived on time", "Arrived late", "Did not arrive" or "Could not be checked", with the time it took since the previous one. If a step fails it is marked "Breaks here", and the following ones read "Not checked: the chain broke earlier". Three typical results for this pipeline:

• Step 1 on time (+3 s) and step 2 on time (+1 s): the chain works end to end.
• Step 1 on time and step 2 "Did not arrive": the function received the order and did not write it. This is the never-written item, the one no metric sees. Check the table environment variable and any caught exceptions.
• Step 1 "Did not arrive": the message did not reach the function, or the function did not write the line. Check the orders-queue monitor (backlog, DLQ) and the process-order monitor (errors, throttling).
• "Could not be checked" is not the same as "Did not arrive": it means we could not look, for example because the role has no permission on the table. In that case the page shows which ARN is missing from each parameter.

When it alerts: learning and confirmation

A new monitor spends its first 24 hours learning: it observes, shows results and does not alert. After that, the state goes through suspect, confirming and down, and only "down" opens an incident and alerts by email, Slack, Microsoft Teams or webhook, the same channels as cost alerts. The incident closes by itself when the resource or the flow recovers.

• Resource monitors: two bad reads in a row declare the outage and one good read closes it. They read every 10 minutes by default, every 5 at the most.
• Flows: three failed tests in a row declare the outage and two good ones close it. With tests every 15 minutes, the alert arrives 30 to 45 minutes after the chain breaks; a test that fails once wakes nobody up.
• Confirmation from a second region (probes run from eu-west-1 and confirm from eu-central-1) belongs to HTTP monitors, where it rules out a network problem. For a resource or a flow, repeating from another region would read the same metrics from the same account, so consecutive results do the confirming there. If your pipeline has an API in front that writes to orders-queue, an HTTP monitor on it does get second-region confirmation.

What it costs

Resource monitors and flows come with the Monitoring pack: €49/month on the Pro and Business plans (€39 from the fourth pack), with 25 monitors, 3 flows, one status page and 90 days of history. This pipeline uses three monitors and one flow from a single pack. Probes run in our account; this is what shows up on your AWS bill:

• Metric reads with GetMetricData: up to about 0.20 USD per resource per month reading every 10 minutes, so at most about 0.60 USD for the three. Reading the DLQ means SQS calls that fit easily in its free tier of one million requests per month.
• Flow tests: every 15 minutes that is about 2,900 a month (if the flow uses up its monthly cost cap, through a broken chain or a very long and slow one, it drops to one every 6 hours until the 1st, and at that pace an incident takes longer to close), and each one is an SQS message, a process-order invocation, a write to orders and about twenty reads (GetItem and FilterLogEvents). It is a very small fraction of what real traffic moves, but factor it in if your margins are tight.

What it does not do

So you know where the limits are:

• Flows are AWS only. Resource monitors also cover Azure and Google Cloud.
• It is not an APM: it does not collect your logs, do distributed tracing or custom metrics. It looks for the test's traceId in your logs and your table, and reads the metrics AWS already publishes.
• The DynamoDB step needs the traceId in the primary key, as a string, in one or two key attributes. If you cannot shape the test item that way, a callback step or an S3 object are the alternatives.
• Discovery follows queues, functions and their destinations; if the chain goes through SNS, an event bus or Step Functions, what lies behind it is added by hand.

Checklist to get started

With these pieces in place, the next lost order is reported by AAA Sentinel, not by a customer.

• The AWS account connected with Monitoring read access.
• A resource monitor on orders-queue (it covers orders-dlq), another on process-order and another on orders.
• The function recognises the test by its traceId, writes the line and stores the item with orderId = aaa-test-{traceId}, with no real effects.
• TTL enabled on the table for test items.
• The flow role with orders-queue in TriggerQueueArns and orders in ObservedTableArns.
• The flow created with two steps and deadlines of 60 and 120 seconds.
• At least one alert channel tested before the 24-hour learning period ends.