# Quickstart: Crash Recovery

Run a durable Spring AI booking agent, kill it mid-call, and watch the *same* run recover when you
restart the app. The instance id you own is what lets a later call **attach** to that run to read its
answer, instead of starting a second booking.

You will learn how to:

- Schedule a durable `ChatClient.call()` under a **caller-owned instance id** (`DurableAdvisor.INSTANCE_ID_KEY`)
- Survive a hard crash of the app (which also hosts the in-process workflow worker)
- See the run recover on restart, with no request needed to nudge it
- Attach to the recovered run with the same id to collect its result, with no duplicate side effect
- Make a real, side-effecting tool safe to retry
- Run the whole walkthrough offline, with no model account

```mermaid
sequenceDiagram
    participant A as Terminal A (caller)
    participant App as Agent app (in-process worker)
    participant C as Catalyst Workflow
    participant B as Terminal B

    A->>App: POST /crash/run {"id":"trip-42"}
    App->>C: schedule workflow "trip-42" (caller-owned id)
    C->>App: run commitReservation (~30s)
    B-->>App: POST /crash/kill (halt the JVM)
    Note over App,C: App dies mid-tool — workflow "trip-42" stays live in Catalyst
    Note over A,App: restart the app — the run resumes on its own
    A->>App: POST /crash/run (same id, to read the answer)
    App->>C: attach to existing "trip-42" (not a new run)
    C-->>App: same confirmation code — no second booking
    App-->>A: {"id":"trip-42","result":"Booking ABC123 confirmed. Confirmation code: BK-..."}
```

## Prerequisites

- [Diagrid Catalyst account](https://catalyst.diagrid.io/)
- [Diagrid CLI](https://docs.diagrid.io/getting-started/install-cli)
- [JDK 21](https://adoptium.net/) or later, and [Maven 3.9+](https://maven.apache.org/download.cgi)
- *(Optional)* [An OpenAI API key](https://platform.openai.com/api-keys), only if you want to run against a real model provider instead of the offline one

## 1. Log in to Catalyst

```bash
diagrid login
diagrid whoami
```

## 2. Clone and Navigate

```bash
git clone https://github.com/diagridio/catalyst-quickstarts.git
cd catalyst-quickstarts/agents/spring-ai/crash-recovery
```

## 3. Explore the Code

The booking agent is a **named `ChatClient` bean** so its run gets a per-agent workflow name. Each call
sets a caller-owned instance id via `DurableAdvisor.INSTANCE_ID_KEY` — that id is the attach handle a
retry re-uses:

```java
@PostMapping("/crash/run")
public ResponseEntity<CrashRunResponse> run(@RequestBody CrashRunRequest request) {
  String id = request.id();
  if (id == null || id.isBlank()) {
    return ResponseEntity.badRequest().body(new CrashRunResponse(id, null, "id is required"));
  }
  String reference = request.reference() == null ? "ABC123" : request.reference();
  try {
    String answer = agent.prompt()
        .user("Confirm the booking with reference " + reference + ".")
        .advisors(a -> a.param(DurableAdvisor.INSTANCE_ID_KEY, id))   // caller-owned id → attach on retry
        .call()
        .content();
    return ResponseEntity.ok(new CrashRunResponse(id, answer, null));
  } catch (DurableCallTimeoutException e) {
    // Wait budget elapsed (not a failure): the run is still going. Re-issue the same id to attach.
    return ResponseEntity.accepted().body(new CrashRunResponse(e.instanceId(), null,
        "still running as " + e.instanceId() + ", re-issue POST /crash/run with the same id to attach"));
  }
}

public record CrashRunRequest(
    String id,
    String reference,
    @JsonProperty("kill_after_seconds") Integer killAfterSeconds) {}

public record CrashRunResponse(String id, String result, String message) {}
```

The booking tool is a **global `@Tool` bean** (`SlowBookingTools.commitReservation`) that sleeps ~30s —
long enough to crash mid-call. It must be a bean (not a per-call tool) so it's re-registered on the
restarted worker and the resumed activity can run it. Its confirmation code is derived from the
reference, so a re-attached call returns the *same* code — visible proof the booking wasn't redone.

The model is a durable activity too, which is why this app ships one rather than skipping it: the
agent's tool choice is the only path to `commitReservation`, so there is no crash window without a
model. `CannedChatModel` supplies that offline, and it reads the turn from the conversation rather
than from a counter, so the activity is safe to re-enter after the restart.

## 4. Use a real model (optional)

**This quickstart needs no API key.** It is about durable execution rather than model quality, so it
ships an offline model (`CannedChatModel`) that always books the reference you send and reports the
confirmation code the tool returns. That is what makes the crash and the recovery the only moving
parts, and it is why every run gives the same answer. The offline model announces itself in the
startup log, so no such line means the app is talking to a real provider.

To run against OpenAI instead, set both variables:

**macOS/Linux**

```bash
export DIAGRID_QUICKSTART_MODEL="openai"
export OPENAI_API_KEY="your-key-here"
```

**Windows**

```powershell
$env:DIAGRID_QUICKSTART_MODEL = "openai"
$env:OPENAI_API_KEY = "your-key-here"
```

On that path a missing or wrong key no longer stops the app from starting. The provider rejects the
first request instead, and because the model call is a durable activity that failure is retried a few
times before it surfaces, as a `500` whose `message` carries the provider's error.

## 5. Install Dependencies

```bash
mvn package -DskipTests
```

## 6. Run with Catalyst Cloud

Create the Catalyst project with managed workflow enabled (and set it as the default for this session), register the agent, then run:

```bash
diagrid project create spring-ai-crash-recovery --enable-managed-workflow --deploy-managed-kv --wait --use
diagrid agent create spring-ai-crash-recovery --wait
diagrid dev run -f dev-spring-ai-crash-recovery.yaml --approve
```

## 7. Crash and recover

### 7.1 Book under an id you own (blocks ~30s)

From **Terminal A**, this schedules the booking under `trip-42` and blocks while the slow tool "commits":

**macOS/Linux**

```bash
curl -X POST "http://localhost:8080/crash/run" \
  -H "Content-Type: application/json" \
  -d '{"id":"trip-42","reference":"ABC123"}'
```

**Windows**

```powershell
Invoke-RestMethod -Method Post -Uri 'http://localhost:8080/crash/run' -ContentType 'application/json' -Body '{"id":"trip-42","reference":"ABC123"}'
```

Watch the app log for the `>>> commitReservation(ABC123)` line, which announces the ~30s window and tells you to kill the app now.

**Two terminals instead of three.** The request takes an optional `kill_after_seconds`. Send it and the app halts *itself* that many seconds into the booking, at a known point inside the window, so you never have to aim a kill at a moving target:

**macOS/Linux**

```bash
curl -X POST "http://localhost:8080/crash/run" \
  -H "Content-Type: application/json" \
  -d '{"id":"trip-42","reference":"ABC123","kill_after_seconds":8}'
```

**Windows**

```powershell
Invoke-RestMethod -Method Post -Uri 'http://localhost:8080/crash/run' -ContentType 'application/json' -Body '{"id":"trip-42","reference":"ABC123","kill_after_seconds":8}'
```

Send this instead of the request above and skip 7.2: the app crashes on its own. Leave the field out and nothing changes, and you crash the app yourself from Terminal B. Either way the rest of the walkthrough is identical.

Keep the value below `crash-recovery.delay-seconds` (30 by default) so the crash lands inside the booking rather than after it has finished. The clock starts when `commitReservation` starts, not when the request arrives, so the budget is measured against that tool's own sleep and does not have to cover the LLM turn ahead of it. That is also why the field is safe to send on the re-issue in 7.4: the timer only starts when the tool actually runs, and a call that attaches to an existing run replays the recorded result instead of re-invoking it.

### 7.2 Crash the app mid-call

**Skip this step if you sent `kill_after_seconds` in 7.1.** The app crashes itself, and there is nothing to do here.

:::warning\[`POST /crash/kill` is demo scaffolding. Do not copy it into a real service.]
It is an unauthenticated endpoint that lets any caller that can reach the port terminate the process, and it exists here only to make a crash reproducible on demand.
:::

From **Terminal B**, during that window:

**macOS/Linux**

```bash
curl -X POST "http://localhost:8080/crash/kill"
```

**Windows**

```powershell
Invoke-RestMethod -Method Post -Uri 'http://localhost:8080/crash/kill'
```

The app process dies before it can answer, so this request itself reports a connection reset rather than a status code — on Windows that surfaces as a PowerShell error, which is expected. Terminal A's blocked call sees a reset too. The workflow `trip-42` keeps living in Catalyst.

### 7.3 Restart the app

The project and agent already exist, so just run:

```bash
diagrid dev run -f dev-spring-ai-crash-recovery.yaml --approve
```

**That is the whole recovery. You do not have to send anything.** The run is not waiting on you:
Catalyst has been retrying the interrupted tool call the entire time the app was down, and it hands
the pending work back the moment the restarted app's worker reconnects. The durable runtime resumes
instance `trip-42` on its own, and the pre-crash LLM turn is not re-executed. Watch the app log: it is
usually scrolling before Spring Boot has finished starting Tomcat.

### 7.4 Collect the answer

The run recovered on its own, but the crash took the connection that was waiting for its result:
Terminal A's call died with the process, and its answer had nowhere to go. Send the **same** call with
the **same** id from **Terminal A** once more to open a new connection to the run that already finished:

**macOS/Linux**

```bash
curl -X POST "http://localhost:8080/crash/run" \
  -H "Content-Type: application/json" \
  -d '{"id":"trip-42","reference":"ABC123"}'
```

**Windows**

```powershell
Invoke-RestMethod -Method Post -Uri 'http://localhost:8080/crash/run' -ContentType 'application/json' -Body '{"id":"trip-42","reference":"ABC123"}'
```

It **attaches** to the recovered run and returns the **same confirmation code**, with no second booking. It resumes nothing, because nothing was waiting. The response is the one JSON shape every crash demo in this repo returns:

```json
{
  "id": "trip-42",
  "result": "Booking ABC123 confirmed. Confirmation code: BK-...",
  "message": null
}
```

The code after `BK-` is derived from the reference, so it is the same code the killed call would have returned. With the offline model the wording is the tool's own, so `result` is exactly the line above; a real provider chooses its own wording around the same code.

If the call's wait budget elapses first, the same shape comes back as a `202` with `result` null and the attach instruction in `message`. That is not a failure: send the same request with the same id again to attach again. A request with a missing or blank `id` is a `400` whose `message` is `id is required`. A `reference` outside `A-Z a-z 0-9 _ -` or longer than 64 characters is also a `400`, with its own `message` naming that constraint.

:::tip\[The instance id is a bearer handle you own]
Guard it like a primary key. A durable activity is at-least-once, so a side-effecting tool should key off a business value (here, the booking reference) to stay idempotent under recovery. The tool's sleep is the Spring property `crash-recovery.delay-seconds` in `src/main/resources/application.properties`, which defaults to 30; keep it comfortably below `diagrid.spring-ai.completion-timeout` (2m in the same file) so the first `/crash/run` is still blocked when you kill the app.
:::

## 8. View in the Catalyst Web Console

Open the [Catalyst Cloud web console](https://catalyst.diagrid.io/), go to **Workflows**, and inspect
instance `trip-42` — the completed activity was not re-executed on recovery.

## 9. Clean Up

Stop the app with `Ctrl+C`, then delete the project:

```bash
diagrid project delete spring-ai-crash-recovery
```

## Summary

In this quickstart, you:

- Scheduled a durable `ChatClient.call()` under a caller-owned instance id
- Killed the app mid-booking and saw the workflow survive in Catalyst
- Restarted the app and saw the run recover on its own, with no request needed
- Attached to the recovered run with the same id — same confirmation code, no double booking
- Made a side-effecting tool safe to retry

## Next Steps

- See the [durable memory quickstart](https://docs.diagrid.io/develop/agents/spring-ai/spring-ai-durable-memory) for how the synchronous advisor chain interacts with durability
- Learn more about [Spring AI + Dapr](https://docs.diagrid.io/develop/agents/spring-ai/)
- Learn how to [deploy AI agents to Kubernetes](https://docs.diagrid.io/develop/agents/kubernetes-deploy)

---

*Spring is a trademark of Broadcom Inc. and/or its subsidiaries.*