• Author: Amin Boulouma, Software Engineer
• Github source code: https://github.com/aminblm/ai_systems_design_from_scratch
• Engineering Blog: https://aminblm.github.io/ai_systems_design_from_scratch/blog/

Intelligent File Watchers: Tracking State via Snapshots

In an agentic system, you often need to monitor a directory for changes. The naive approach—re-reading every file in every loop—is disastrous for performance, especially when dealing with large datasets. The efficient way is to take a snapshot of the system state using modification times (mtime) and compare it against the last known state.

Glossary for the Young Engineer

• Snapshot: A “picture” taken of your files at a specific moment in time so you can look at it later to see if anything changed.
• mtime (Modification Time): The “last updated” stamp on a file. It tells the computer exactly when the file was last changed.
• State Tracking: Keeping a list in your computer’s memory of what things look like right now, so you can spot the difference later.
• Redundant Processing: Doing the same work twice when you don’t need to. Like re-reading a book you already finished just to check if the last page changed.

The Problem: The “Re-scan” Bottleneck

If your agent needs to process data from 10,000 files, and you re-process every one of them every few seconds, your CPU will stay at 100%, and your logs will be flooded with noise.

Why we choose mtime snapshots over hashing: While hashing (calculating a fingerprint of the file contents) is more accurate, it is prohibitively expensive for large files because you must read the entire file to compute the hash. mtime is provided by the filesystem metadata, which is nearly instantaneous to retrieve, even for massive files.

Implementation

Simple Example: The Basic Tracker

This snippet takes a snapshot and checks if a file has been modified.

import os

snapshot = {}
full_path = "data.txt"

# Taking the snapshot
snapshot[full_path] = os.path.getmtime(full_path)

# Checking if it changed
if os.path.getmtime(full_path) != snapshot.get(full_path):
    print("File was modified!")

Complex Example: Production-Grade Directory Observer

A production-ready observer handles recursive scanning, exception safety (e.g., file deleted during scan), and state diffing.

import os

class DirectoryObserver:
    def __init__(self):
        self.snapshot = {}

    def get_changes(self, root_dir):
        changes = {"added": [], "modified": [], "removed": []}
        current_files = set()

        for dirpath, _, filenames in os.walk(root_dir):
            for f in filenames:
                full_path = os.path.join(dirpath, f)
                current_files.add(full_path)
                
                try:
                    mtime = os.path.getmtime(full_path)
                    if full_path not in self.snapshot:
                        changes["added"].append(full_path)
                    elif self.snapshot[full_path] != mtime:
                        changes["modified"].append(full_path)
                    self.snapshot[full_path] = mtime
                except OSError:
                    continue # File might have been deleted mid-scan

        # Identify removals
        changes["removed"] = list(set(self.snapshot.keys()) - current_files)
        for f in changes["removed"]:
            del self.snapshot[f]
            
        return changes

Quick Reference: When to use which?

Strategy	Use Case	Why?
mtime Snapshot	Standard file tracking	Fast, zero-dependency, filesystem-native.
File Hashing	Critical data integrity	Detects silent corruption, but slow.
OS Events (Inotify)	Real-time triggers	Low CPU, but platform-specific/complex.

Developer Checklist

• Atomicity: Are you taking the snapshot in a way that doesn’t miss changes during the scan?
• Error Handling: Does your code handle OSError if a file is moved/deleted during the scan?
• Memory Footprint: Is the snapshot dictionary growing too large? Consider clearing old entries.
• Granularity: Are you scanning the root directory or can you limit it to specific subfolders?

Final Takeaways

1. Metadata is your best friend. Always use filesystem metadata before resorting to content-based processing.
2. Fail safe. Filesystems are unpredictable; always wrap your getmtime calls in try/except blocks to handle ephemeral files.
3. Be lazy. Only perform heavy work when the mtime actually deviates from the snapshot.