I started the way most RAG systems start. I had a folder of documents, a Python script, and an S3 Vectors index. I pointed the script at one file, waited, and queried what came back. That was enough to prove that a Bedrock embedding model plus S3 Vectors could answer a question. It was not enough once the corpus moved to S3 and new files kept arriving.
This is the engineering of that change. A single Standard state machine orchestrates the work. One Lambda does the work. A table records what happened to each object. The chunker is a function the worker calls. This post is about the workflow around that function: how a run stays cheap while it is idle, how a failure stays visible, how a second run does not duplicate vectors, and how a new object is work rather than a new script.
What the manual version actually did
The first version ran on my laptop, one file at a time.
Read each page of a PDF.
Collapse whitespace.
Slice the page into fixed character windows with an overlap.
Call the embedding model once per window.
Write the vectors, with a freshly generated ID in every key.
A scan was a separate job. I sent the file to Textract (for image PDF files) , wrote a new PDF from the text, and then ran the same ingest script over the converted files. Querying was a third script: embed the question, take a few neighbours, ask a generation model to answer.
That is a real pipeline, with a person as the scheduler. Five things get worse as the data grows.
It is not re-runnable. Every ingest minted a new ID. Running the same file twice stored a second copy. Nothing recorded which files had already succeeded.
It is not traceable. Progress was the terminal. If the laptop slept, I found out when I came back. A throttle and a corrupt file looked the same: a stack trace, then nothing.
It does not scale with the corpus. The source was not a folder of a few dozen PDFs. It was an object store holding tens of thousands of documents, mostly structured text, with PDFs arriving later. A shell loop is the wrong shape for that, and the wrong shape for the day a new file appears while I am not at the keyboard.
The chunks were the wrong unit. A character window does not know where a document’s own units begin and end. Overlap then embeds the same sentence twice and still splits the unit a reader would cite. The fix belongs in the chunker. The workflow only needs to call it once per object and store what it returns.
Scans were a manual fork. I had to notice that a PDF was an image, convert it, and remember to ingest the conversion rather than the original. A new file type meant a new command.
What I wanted instead
Step Functions is the piece that makes the other four possible. Lambda is a good worker and a bad scheduler. It cannot fan out tens of thousands of documents, remember which child is still running, retry a throttle, and keep going when one file is malformed. That is a workflow. A queue of every key would also work, but then I would be building the history, the retry policy, and the “what is still in flight” view myself. Step Functions already has an execution history per document.
The rest of the system is small on purpose.
Nothing in that list runs while the corpus is quiet. The table and the buckets charge for what is stored. The workflow and the function charge for the run. The model charges for the tokens embedded. A retry of two documents costs two documents, not the corpus.
The state machine has four states
The definition is short. Tokens such as the function ARN are filled in by Terraform so the JSON can keep Step Functions’ $$ and $. syntax intact.
ListDocuments → EmbedDocuments (distributed map) → Finish
│
└─ any map-level failure → MarkFailedList invokes the worker with an action that means “build the work list.” The function pages through the source bucket, keeps the extensions it knows how to read, and writes the keys as a JSON array. It resets the summary item: total, done, failed, skipped, chunks, all zero, status running. The summary is keyed to this run, so its counters describe this execution, not the lifetime of the index.
Embed is a distributed map on a Standard workflow. The item reader loads that JSON array from the work bucket. The item selector hands each child the object key and the run ID. Concurrency is capped, so only a small number of documents are being embedded at once. The tolerated failure percentage is 100. One bad file must not abort the rest. The map writes its own results back to the work bucket.
Finish and MarkFailed both call the same function and set the summary status. The map’s catch is for the map itself falling over. A single document’s failure is recorded inside the worker and does not take this path, because the map is allowed to tolerate it.
Each child task has a timeout just past the function’s own limit. Its retry list is the part I had to learn in production. The names have to be the ones the runtime actually throws:
Lambda.TooManyRequestsException
Lambda.ServiceException
Lambda.SdkClientException
Lambda.Unknown
States.Timeout
ClientError
ThrottlingException
TooManyRequestsException
ServiceUnavailableException
ProvisionedThroughputExceededException
ModelTimeoutException
LimitExceededExceptionThe AWS SDK retries a throttle inside the function. If those attempts are exhausted, the exception the state machine sees isThrottlingException, not ClientError. The first retry list did not name ThrottlingException, so Step Functions treated those children as finished failures and did not try them again. The worker also does not mark a throttle as failed in the table. A retry of the same call can succeed, and a failed row would have hidden that.
A bad file is different. The worker recordsfailed, sends the key and the error to the queue, and raises an error type that is not in the retry list. Retrying identical bytes will fail the same way. Retrying a throttle might not.
One function, three jobs
The handler is a switch on the event.
A list action builds the manifest.
A finish or fail action updates the summary.
Anything else is one document: the object key and the run ID.
Processing one document is the contract with the workflow:
If this run ID has already marked the key done, return. A map retry must not embed it twice in the same run.
If the key should be ignored, mark it skipped and return. An empty result is skipped too. An empty file should not sit in the failure queue pretending to be a bug.
Read the object.
Turn it into text. The reader looks at the bytes, not at a list of filenames: structured text, a text PDF, or a scan that needs OCR.
Hand the text to the chunker.
Embed the chunks with a small thread pool and write them in batches.
Mark the row as done with the chunk count.
A selective re-run changes the list, not the worker. The list still walks the bucket, then drops every key whose status is already done or skipped. A new object has no row, so it stays on the manifest. A document that failed, or whose worker was killed before it could write a row, stays too. The successes stay put.
I do not pass a list of filenames. I start the same state machine with an input that means “only what is not already finished”:
aws stepfunctions start-execution \
--state-machine-arn "$STATE_MACHINE_ARN" \
--name "retry-$(date -u +%Y%m%dT%H%MZ)" \
--input '{"only_failed": true}'An optional prefix limits the listing when I know which folder changed. An empty input lists every supported object and embeds them again, because a document counts as already done for this run only when its stored run ID matches the run that just started. I use a full pass when the chunker itself has changed, and the vectors should be replaced. I do not use it to pick up one new file.
This is not an upload trigger. Putting a file in the bucket does not start the state machine. What is automatic is everything after the start: reading, chunking, embedding, and skipping work that is already done. Connecting object-created events to a start call, with the selective input, is a small next step. I kept the start explicit during the backfill, because a full-bucket execution is expensive and easy to launch by mistake.
Keys that can be written twice
The laptop script put a random ID in the vector key, so a second run could not find the first run’s vectors to replace them.
The workflow uses a key derived from the object key and the chunk’s position. The same document, processed again, writes the same keys. The vector write overwrites. “Run it again” replaces the document instead of duplicating the index.
One index holds the corpus. Differences between documents live in metadata, and the set of filterable names is fixed when the index is created. A new value on an existing field does not need a new index. A new field does. I decided that list before the backfill, not after.
What a large run taught me
The first execution listed every supported object and ran for many hours at a capped concurrency. The state machine finished green. The progress table did not say the corpus was finished.
A few hundred documents had failed, almost all with the same exception from the chunker. I fixed the chunker, deployed the same function, and started a selective run. The manifest was those failures, not the whole bucket. Nearly all of them completed.
Two were still missing, and they were missing from the failure count as well. Their child executions showed a throttle after the SDK had given up. Nothing was written, because a throttle is re-raised. They were picked up by a later selective run, after the state machine’s retry list included the exception name the runtime actually throws.
That is the traceability rule worth copying: the execution status and the document status are different facts. Allowing every item to fail lets the workflow succeed while documents fail. The summary item and the per-key rows are the corpus. The console is how I watch the map. The queue message is how I read a permanent error.
A later lesson was the opposite failure. One very large file never got a row. The map finished, and the summary was one document short. A function that hits its timeout is killed. It does not get to write failed. The selective list treats “no row” the same as “not done,” which is what I wanted. The next run included that file and nothing that was already done.
A file that arrived after the backfill behaved the same way. The next selective run listed the new object and the one that still had no row. Both completed. The rest of the index was not touched.
How the cost stays tied to the work
Affordable does not mean free. Embedding tens of thousands of documents is a model bill, and it should be. The design keeps the other bills tied to that work.
There is no container service and no instance to turn off.
The function exists only for the document it is processing. Concurrency is a cap. Raising it finishes sooner and throttles more. Throttles burn retries, which embed the same text again.
A small thread pool inside one document keeps a long file from spending its whole timeout waiting on the model one call at a time.
A Standard workflow is the right type because the map runs for hours and each child needs its own history. The transitions are cheap next to the embeddings, and they are zero when nothing is running.
The table is on-demand: one write per document, plus a summary. It is scanned only when building a selective manifest.
Re-runs are the main saving. The failure retry processed hundreds of documents. The following run processed two. A naive scheduler would have processed the whole corpus both times.
One index means I am not storing and querying several copies of the same vectors so that a filter can be a separate collection.
The chunker affects the bill too, without the workflow knowing how it cuts. Fixed windows embed the same sentence in two vectors and still break the unit a reader would cite. Fewer, more coherent chunks make a smaller index. The workflow’s job is to store whatever the chunker returns, under stable keys.
What I would copy, and what I would not
Copy the shape.
Put the documents in object storage and treat the key as the identity.
Use a workflow that can map over a manifest, cap concurrency, and retry named errors.
Use a ledger that is not the workflow status. One row per object.
Separate “retry, this might work” from “record this, the file is wrong.”
Make the vector key a function of the source key, so a second run overwrites.
Decide how to read a file from the bytes. Leave the cutting rules inside the chunker.
Start the same workflow for a backfill and for one new file. Change the input, not the script.
Do not copy the accidents.
Do not treat a green execution as a finished corpus when every item is allowed to fail.
Do not retry errors that mean the file itself is wrong. Do retry throttles, and name the exception the runtime actually throws.
Do not mark a throttle as a permanent failure. The row would hide a document the next retry could have finished.
Do not add filterable metadata fields casually. The index has to be recreated.
Do not start a full pass to pick up one new object.
Retrieval stays a separate system. This pipeline’s job ends when the vector is written. The application that answers questions should not have to know whether the source was structured text, a text PDF, or a scan.
A new file, end to end
Upload the file to the source bucket.
Start the state machine with the selective input.
Read the summary row. The total should be the new files plus anything that has no done or skipped row. If it is the whole corpus, the start input was wrong.
When the execution stops, the same row shows done, failed, and skipped for that run. A permanent failure is also in the queue, with the error. A throttle may have no row yet. The next selective run includes it if the workflow’s own retries were not enough.
The laptop is still where I build the function package and apply the infrastructure, and where I can try the chunker on one file before trusting it with a long invocation. It is not where the corpus runs.
That is the difference. The first script needed me to choose the file, the window size, and, for a scan, a conversion step. The state machine needs a start. After that, it reads a new object, writes vectors that a later run can replace, and leaves a row that says what happened.



