Architecting Persistent Systems: State Management for Ephemeral AI

In complex systems, AI models are inherently ephemeral—they execute, consume significant memory, and eventually exit. However, the business logic and the results they produce must persist, regardless of system reboots or process crashes.

To achieve this, we distinguish between Volatile Execution (the AI model) and Persistent Infrastructure (the Kernel). This creates a durable foundation where state is treated as a first-class citizen.

The Durable Foundation Pattern

By separating state management from the inference logic, we ensure that every request is traceable and recoverable. If a process dies mid-inference, the scheduler recognizes the PENDING state on restart and resumes the workflow.

1. Simple Implementation: State Persistence

The PersistentScheduler acts as the source of truth for the system’s current work, ensuring no task is lost.

import json
import uuid

class PersistentScheduler:
    def __init__(self, state_file="system_state.json"):
        self.state_file = state_file
        self.tasks = self._load()

    def _load(self):
        try:
            with open(self.state_file, 'r') as f: return json.load(f)
        except FileNotFoundError: return {}

    def schedule(self, model_id, status="PENDING"):
        task_id = str(uuid.uuid4())
        self.tasks[task_id] = {"model": model_id, "status": status}
        with open(self.state_file, 'w') as f: json.dump(self.tasks, f)
        return task_id

2. Complex Implementation: The AI Host Wrapper

This encapsulation ensures that the AI model does not manage its own persistence, following the Single Responsibility Principle.

class AIHost:
    def __init__(self, scheduler, storage):
        self.scheduler = scheduler
        self.storage = storage

    def run_inference(self, task_name, data):
        # 1. Register task before execution
        task_id = self.scheduler.schedule(task_name)
        
        try:
            # 2. Ephemeral execution
            result = self.perform_computation(data)
            # 3. Durable persistence
            self.storage.persist(f"{task_id}.bin", result)
            self.scheduler.update_status(task_id, "COMPLETED")
        except Exception:
            self.scheduler.update_status(task_id, "FAILED")

Architectural Principles for Reliability

• Idempotency: A task should be re-runnable without side effects. Always check for existing COMPLETED results in DurableStorage before invoking the compute-heavy AIHost.
• Atomic State Updates: Use temporary file writing (e.g., writing to a .tmp file and renaming it) when updating your system_state.json to prevent corruption if a crash occurs mid-write.
• Separation of Concerns: The AI model should be an “input-to-output” black box. All metadata (timing, status, retries) must reside in the PersistentScheduler, keeping the AI logic clean and swappable.

Enterprise Feature Roadmap

1. Recovery Worker: Implement a background service that scans the PersistentScheduler on system boot to identify and retry PENDING or FAILED tasks.
2. Blob Storage Integration: As scale increases, transition DurableStorage from a local disk path to a cloud object store (like S3), using the same interface.
3. Task Versioning: Include model version hashes in the state registry to ensure auditability—knowing exactly which model generated which result is vital for debugging model drift.

• Author: Amin Boulouma, Software Engineer
• Github source code: https://github.com/aminblm/ai_systems_design_from_scratch