Connect with Amin Boulouma Official

🏠 Documentation Hub 📝 Engineering Blog 💻 GitHub Repository

Robust File Handling and Error Logging in Python

Amin Boulouma, Software Engineer

When building automated systems, such as a static site generator, handling file system operations and unexpected errors gracefully is critical. In this tutorial, we explore how to leverage standard library modules to create a reliable processing pipeline.

Essential Libraries for File Systems

• pathlib: The modern, object-oriented approach to handling file system paths. It simplifies joining, traversing, and manipulating file extensions compared to the legacy os.path.
• os: While pathlib is preferred for paths, os remains essential for environment interaction and low-level system checks.
• logging: Essential for production-ready code. Unlike print(), it allows you to categorize output (INFO, ERROR, DEBUG) and direct it to files or external monitoring systems.
• traceback: Provides the ability to capture and log the full call stack when an exception occurs, which is invaluable for debugging production issues.

Implementation: The SiteGenerator Class

The following example demonstrates a SiteGenerator that handles file discovery, template rendering, and error management.

import os
import logging
import traceback
from pathlib import Path
from typing import Dict, Union, Tuple 

# Simulated imports for the project structure
from ai_systems_design.safe_yaml_parser import ConfigurationBuilder
from ai_systems_design.md_html import MarkdownConverterFacade
from ai_systems_design.utils import FileOperationsUtility

# Configure logging at the module level
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

class SiteGenerator:
    """Coordinator class to orchestrate the Markdown to Template-bound HTML building process."""

    def __init__(self, layout_path: Union[str, Path], config_file_path: Union[str, Path]) -> None:
        self.layout_path = Path(layout_path)
        self.config_file_path = Path(config_file_path)

        self.layout_template = self._load_layout()
        self.config_mappings = self._load_config()

    def _load_layout(self) -> str:
        return FileOperationsUtility.read_decoded(str(self.layout_path))
    
    def _load_config(self) -> Dict[str, str]:
        return ConfigurationBuilder().from_file(str(self.config_file_path)).build().to_dict()
    
    def _render_html(self, md_file_path: Path) -> str:
        """Injects compiled markdown content and config mappings into the layout."""
        md_html_content = MarkdownConverterFacade().convert_file(str(md_file_path))
        html = self.layout_template.replace('', md_html_content)

        for key, value in self.config_mappings.items():
            html = html.replace(f'{{{{ site.{key} }}}}', str(value))
        return html
    
    def _resolve_paths(self, md_file: Path, input_dir: Path, output_dir: Path) -> Tuple[Path, Path]:
        """Calculates input and output targets safely using modern path objects."""
        html_filename = f"{md_file.stem}.html"
        return md_file, output_dir / html_filename
    
    def generate_site(self, input_directory: Union[str, Path]) -> None:
        """Processes all Markdown files within the targeted input directory."""
        input_dir = Path(input_directory)

        if not input_dir.is_dir():
            raise ValueError(f'Target input path is not a valid directory: {input_dir}')

        output_dir = input_dir.parent / 'sg_output'
        output_dir.mkdir(parents=True, exist_ok=True)

        for file_path in input_dir.iterdir():
            # Skip non-markdown files
            if not file_path.is_file() or file_path.suffix.lower() != '.md':
                continue

            try:
                src_path, dest_path = self._resolve_paths(file_path, input_dir, output_dir)
                rendered_content = self._render_html(src_path)

                FileOperationsUtility.write_encoded(str(dest_path), rendered_content)
                logger.info(f"Successfully generated page: {dest_path.name}")

            except Exception as err:
                # Capture and log the error message
                logger.error(f"Failed processing {file_path.name}: {err}")
                # Log the full traceback for deeper debugging
                logger.debug(traceback.format_exc())

Best Practices Highlighted

1. Fail-Safe Loops: By using a try-except block inside the file loop, one corrupted file won’t crash the entire site generation process.
2. Contextual Logging: Note how we log the specific filename that failed. This turns a cryptic traceback into an actionable debugging item.
3. Path Safety: Utilizing pathlib.Path objects prevents OS-specific separator issues (e.g., forward slash vs backslash), making the code cross-platform compatible.

Connect with Amin Boulouma Official