Skip to main content

Quickstart: Crash Recovery

Run a durable Spring AI booking agent, kill it mid-call, and watch the same run recover when you restart the app. The instance id you own is what lets a later call attach to that run to read its answer, instead of starting a second booking.

You will learn how to:

  • Schedule a durable ChatClient.call() under a caller-owned instance id (DurableAdvisor.INSTANCE_ID_KEY)
  • Survive a hard crash of the app (which also hosts the in-process workflow worker)
  • See the run recover on restart, with no request needed to nudge it
  • Attach to the recovered run with the same id to collect its result, with no duplicate side effect
  • Make a real, side-effecting tool safe to retry
  • Run the whole walkthrough offline, with no model account

Prerequisites​

1. Log in to Catalyst​

diagrid login
diagrid whoami

2. Clone and Navigate​

git clone https://github.com/diagridio/catalyst-quickstarts.git
cd catalyst-quickstarts/agents/spring-ai/crash-recovery

3. Explore the Code​

The booking agent is a named ChatClient bean so its run gets a per-agent workflow name. Each call sets a caller-owned instance id via DurableAdvisor.INSTANCE_ID_KEY — that id is the attach handle a retry re-uses:

@PostMapping("/crash/run")
public ResponseEntity<CrashRunResponse> run(@RequestBody CrashRunRequest request) {
  String id = request.id();
  if (id == null || id.isBlank()) {
    return ResponseEntity.badRequest().body(new CrashRunResponse(id, null, "id is required"));
  }
  String reference = request.reference() == null ? "ABC123" : request.reference();
  try {
    String answer = agent.prompt()
        .user("Confirm the booking with reference " + reference + ".")
        .advisors(a -> a.param(DurableAdvisor.INSTANCE_ID_KEY, id))   // caller-owned id → attach on retry
        .call()
        .content();
    return ResponseEntity.ok(new CrashRunResponse(id, answer, null));
  } catch (DurableCallTimeoutException e) {
    // Wait budget elapsed (not a failure): the run is still going. Re-issue the same id to attach.
    return ResponseEntity.accepted().body(new CrashRunResponse(e.instanceId(), null,
        "still running as " + e.instanceId() + ", re-issue POST /crash/run with the same id to attach"));
  }
}

public record CrashRunRequest(
    String id,
    String reference,
    @JsonProperty("kill_after_seconds") Integer killAfterSeconds) {}

public record CrashRunResponse(String id, String result, String message) {}

The booking tool is a global @Tool bean (SlowBookingTools.commitReservation) that sleeps ~30s — long enough to crash mid-call. It must be a bean (not a per-call tool) so it's re-registered on the restarted worker and the resumed activity can run it. Its confirmation code is derived from the reference, so a re-attached call returns the same code — visible proof the booking wasn't redone.

The model is a durable activity too, which is why this app ships one rather than skipping it: the agent's tool choice is the only path to commitReservation, so there is no crash window without a model. CannedChatModel supplies that offline, and it reads the turn from the conversation rather than from a counter, so the activity is safe to re-enter after the restart.

4. Use a real model (optional)​

This quickstart needs no API key. It is about durable execution rather than model quality, so it ships an offline model (CannedChatModel) that always books the reference you send and reports the confirmation code the tool returns. That is what makes the crash and the recovery the only moving parts, and it is why every run gives the same answer. The offline model announces itself in the startup log, so no such line means the app is talking to a real provider.

To run against OpenAI instead, set both variables:

export DIAGRID_QUICKSTART_MODEL="openai"
export OPENAI_API_KEY="your-key-here"

On that path a missing or wrong key no longer stops the app from starting. The provider rejects the first request instead, and because the model call is a durable activity that failure is retried a few times before it surfaces, as a 500 whose message carries the provider's error.

5. Install Dependencies​

mvn package -DskipTests

6. Run with Catalyst Cloud​

Create the Catalyst project with managed workflow enabled (and set it as the default for this session), register the agent, then run:

diagrid project create spring-ai-crash-recovery --enable-managed-workflow --deploy-managed-kv --wait --use
diagrid agent create spring-ai-crash-recovery --wait
diagrid dev run -f dev-spring-ai-crash-recovery.yaml --approve

7. Crash and recover​

7.1 Book under an id you own (blocks ~30s)​

From Terminal A, this schedules the booking under trip-42 and blocks while the slow tool "commits":

curl -X POST "http://localhost:8080/crash/run" \
  -H "Content-Type: application/json" \
  -d '{"id":"trip-42","reference":"ABC123"}'

Watch the app log for the >>> commitReservation(ABC123) line, which announces the ~30s window and tells you to kill the app now.

Two terminals instead of three. The request takes an optional kill_after_seconds. Send it and the app halts itself that many seconds into the booking, at a known point inside the window, so you never have to aim a kill at a moving target:

curl -X POST "http://localhost:8080/crash/run" \
  -H "Content-Type: application/json" \
  -d '{"id":"trip-42","reference":"ABC123","kill_after_seconds":8}'

Send this instead of the request above and skip 7.2: the app crashes on its own. Leave the field out and nothing changes, and you crash the app yourself from Terminal B. Either way the rest of the walkthrough is identical.

Keep the value below crash-recovery.delay-seconds (30 by default) so the crash lands inside the booking rather than after it has finished. The clock starts when commitReservation starts, not when the request arrives, so the budget is measured against that tool's own sleep and does not have to cover the LLM turn ahead of it. That is also why the field is safe to send on the re-issue in 7.4: the timer only starts when the tool actually runs, and a call that attaches to an existing run replays the recorded result instead of re-invoking it.

7.2 Crash the app mid-call​

Skip this step if you sent kill_after_seconds in 7.1. The app crashes itself, and there is nothing to do here.

POST /crash/kill is demo scaffolding. Do not copy it into a real service.

It is an unauthenticated endpoint that lets any caller that can reach the port terminate the process, and it exists here only to make a crash reproducible on demand.

From Terminal B, during that window:

curl -X POST "http://localhost:8080/crash/kill"

The app process dies before it can answer, so this request itself reports a connection reset rather than a status code — on Windows that surfaces as a PowerShell error, which is expected. Terminal A's blocked call sees a reset too. The workflow trip-42 keeps living in Catalyst.

7.3 Restart the app​

The project and agent already exist, so just run:

diagrid dev run -f dev-spring-ai-crash-recovery.yaml --approve

That is the whole recovery. You do not have to send anything. The run is not waiting on you: Catalyst has been retrying the interrupted tool call the entire time the app was down, and it hands the pending work back the moment the restarted app's worker reconnects. The durable runtime resumes instance trip-42 on its own, and the pre-crash LLM turn is not re-executed. Watch the app log: it is usually scrolling before Spring Boot has finished starting Tomcat.

7.4 Collect the answer​

The run recovered on its own, but the crash took the connection that was waiting for its result: Terminal A's call died with the process, and its answer had nowhere to go. Send the same call with the same id from Terminal A once more to open a new connection to the run that already finished:

curl -X POST "http://localhost:8080/crash/run" \
  -H "Content-Type: application/json" \
  -d '{"id":"trip-42","reference":"ABC123"}'

It attaches to the recovered run and returns the same confirmation code, with no second booking. It resumes nothing, because nothing was waiting. The response is the one JSON shape every crash demo in this repo returns:

{
  "id": "trip-42",
  "result": "Booking ABC123 confirmed. Confirmation code: BK-...",
  "message": null
}

The code after BK- is derived from the reference, so it is the same code the killed call would have returned. With the offline model the wording is the tool's own, so result is exactly the line above; a real provider chooses its own wording around the same code.

If the call's wait budget elapses first, the same shape comes back as a 202 with result null and the attach instruction in message. That is not a failure: send the same request with the same id again to attach again. A request with a missing or blank id is a 400 whose message is id is required. A reference outside A-Z a-z 0-9 _ - or longer than 64 characters is also a 400, with its own message naming that constraint.

The instance id is a bearer handle you own

Guard it like a primary key. A durable activity is at-least-once, so a side-effecting tool should key off a business value (here, the booking reference) to stay idempotent under recovery. The tool's sleep is the Spring property crash-recovery.delay-seconds in src/main/resources/application.properties, which defaults to 30; keep it comfortably below diagrid.spring-ai.completion-timeout (2m in the same file) so the first /crash/run is still blocked when you kill the app.

8. View in the Catalyst Web Console​

Open the Catalyst Cloud web console, go to Workflows, and inspect instance trip-42 — the completed activity was not re-executed on recovery.

9. Clean Up​

Stop the app with Ctrl+C, then delete the project:

diagrid project delete spring-ai-crash-recovery

Summary​

In this quickstart, you:

  • Scheduled a durable ChatClient.call() under a caller-owned instance id
  • Killed the app mid-booking and saw the workflow survive in Catalyst
  • Restarted the app and saw the run recover on its own, with no request needed
  • Attached to the recovered run with the same id — same confirmation code, no double booking
  • Made a side-effecting tool safe to retry

Next Steps​

Spring is a trademark of Broadcom Inc. and/or its subsidiaries.

Last updated on