Skip to content

SlapOS Code Walkthrough

SlapOS is a sophisticated, Python-based cloud orchestration and automation platform. Its primary purpose is to provide a reproducible and automated way to build, deploy, configure, and manage software services. It is not an application itself, but a framework for turning other software—from simple libraries to complex applications like GitLab or ERP5—into manageable cloud services.

The central technology is zc.buildout, a powerful Python-based build system, which SlapOS extends with its own orchestration logic.

Key Architectural Concepts & Technologies

  1. Buildout as the Core Engine:
    The entire system is orchestrated by zc.buildout. Every .cfg file is a buildout profile that defines “parts” to be built. SlapOS uses this to create completely self-contained software installations.

    • Reproducibility: By defining all dependencies, sources, and compilation steps in configuration files, SlapOS ensures that a piece of software can be built identically on any machine, eliminating “it works on my machine” issues.
    • File: setup.py shows this clearly by defining a slapos.cookbook package with numerous zc.buildout entry points, which are the recipes themselves.

  2. Recipes as Automation Primitives (slapos/recipe/):
    The “recipes” are the heart of the system’s automation logic. They are Python modules executed by Buildout that perform specific tasks. The code you provided contains a rich library of them:

    • Service Configuration: Recipes like slapos.recipe.postgres or slapos.recipe.redis take parameters (like IP, port, password) and generate the necessary configuration files (postgresql.conf, redis.conf.in).
    • Wrapper Generation: A very common pattern is creating executable wrapper scripts (e.g., slapos.recipe.wrapper.py). These scripts set up the correct environment (like PERL5LIB in perl-CPAN-package-create-wrapper.py) and then execute the actual binary. This ensures services run in a controlled and isolated environment.
    • Lifecycle Management: Recipes handle the entire lifecycle, from creating directories (mkdirectory.py) and users to configuring log rotation (logrotate.py) and setting up network tunnels (6tunnel).

  3. Build Hooks for Deep Customization (component/.../hooks.py):
    For low-level software components (like C libraries), SlapOS uses build hooks. These are Python scripts that run at specific points during the build process (pre_make_hook, post_make_hook).

    • Example (component/bzip2/bzip2-hooks.py): This script runs after the standard make command. It manually compiles a dynamic shared library (libbz2.so), copies it to the correct lib/ directory, and creates the necessary symbolic links. This is a powerful way to handle software that doesn’t natively support certain build requirements.
    • Example (component/ca-certificates/ca-certificates-pre-make-hook.py): This hook cleverly modifies a Makefile before compilation to ensure it uses a specific Python executable, avoiding a circular dependency during the system’s initial bootstrap.

  4. A Rich and Mature Testing Framework:
    The sheer volume and sophistication of the test files (/test/ directories) indicate that SlapOS is a mature and robust project.

    • End-to-End Testing: The tests are not simple unit tests. They are full-fledged integration tests that instantiate complete services (like MariaDB in component/mariadb/test/test.py or a full ERP5 instance in software/erp5/test/test/test_erp5.py), make network requests to them, and verify their state.
    • Custom Test Case: The framework uses a custom SlapOSInstanceTestCase which automates the process of requesting and deploying a software instance for testing purposes.

Code Structure and Organization

The code is logically structured into several key directories:

  • /component: This directory contains recipes for building fundamental software dependencies that other services will rely on. Examples include bzip2, gdal, mariadb, perl, tesseract. The focus here is on compiling and packaging these tools in a way that they are self-contained and can be reliably linked against.

  • /slapos/recipe: This is the “standard library” of SlapOS automation. It contains the reusable Python recipes for common tasks. This is where the core logic for managing services resides. The variety is impressive, covering everything from resilience (addresiliency), monitoring (check_page_content), random data generation (random.py), and service requests (request.py).

  • /software: This directory contains definitions for high-level, user-facing applications. Each subdirectory is a “Software Release” that bundles various components and recipes to deliver a complete service. The code shows examples for gitlab, peertube, erp5, hugo, etc. This demonstrates the ultimate goal of SlapOS: to package any third-party software into a deployable service.

  • /setup.py & slapos/__init__.py: These files define the slapos.cookbook Python package, which makes all the recipes available to the Buildout system as plugins.

Illustrative Code Examples

  • component/perl/perl-CPAN-package-create-wrapper.py: A perfect example of the wrapper pattern. After a Perl package is built, this post_make_hook creates a new directory of executable scripts. Each script is a simple shell wrapper that sets the PERL5LIB environment variable before calling the real script with the correct Perl interpreter. This ensures the script finds its libraries without polluting the global environment.

  • slapos/recipe/request.py: This recipe is the bridge between a SlapOS node and the SlapOS Master. It shows how an instance can programmatically “request” another service (like a database), providing parameters and receiving connection information back. This is fundamental to building distributed systems.

  • software/erp5/test/test/test_balancer.py: This test file is a masterclass in end-to-end testing. It programmatically starts multiple backend HTTP servers, instantiates a full SlapOS balancer instance to proxy requests to them, and then uses the requests library to make HTTP calls and verify complex behaviors like load balancing, cookie stickiness, and log rotation.

  • component/jupyter-erp5/ERP5kernel.py: This file is fascinating. It defines a custom Jupyter kernel that communicates with an ERP5 backend. It intercepts Jupyter “magics” (like %erp5_user) to configure its connection, and then sends code to be executed within the ERP5 environment. This showcases how SlapOS can be used to deploy highly specialized and deeply integrated applications.

Conclusion

The provided code paints a picture of a powerful, mature, and exceptionally well-tested orchestration framework. SlapOS is designed to solve the difficult problem of creating reproducible, self-contained, and manageable software deployments for a vast range of applications.

Its core philosophy revolves around: * Automation: Automating every step of the software lifecycle. * Reproducibility: Using declarative configurations (buildout) and build hooks to ensure builds are consistent. * Modularity: Building complex systems from a rich library of reusable components and recipes. * Isolation: Using wrappers and controlled environments to prevent conflicts between services.

It is a tool for expert DevOps engineers and cloud operators who require fine-grained control over their entire software stack, from the C libraries up to the final application.

Page last modified: 2025-08-26 18:26:16