Python ↔︎ Zig Interop

#zig #compilation #performance

While Python↔Rust integration is effectively a solved problem thanks to the mature PyO3/Maturin ecosystem, Python↔Zig integration remains problematic. The primary framework (Ziggy Pydust) is stalled at Zig 0.11 with no active development. Alternative approaches exist but are fragmented.

This note documents the current state of the art, reports findings from our experiments with four different FFI approaches, and proposes a path forward using HPy as a foundation.

Current recommendation: Use HPy + C shim. A thin C layer (~50-100 lines) handles Python binding via HPy, while Zig exports standard extern functions. This works with any Zig version and supports PyPy/GraalPy.

Alternatives:

• Ziggy Pydust: Best ergonomics but stuck on Zig 0.14 (current is 0.15.1)
• PyOZ: Targets Zig 0.15+ but only 3 weeks old with no PyPI release - watch but don’t adopt yet
• cffi: Simple, works everywhere, but limited to C-compatible types

This note documents the current state of the art based on experiments with four FFI approaches.

1. Python↔Rust: A Solved Problem

The PyO3 + Maturin combination has achieved production-grade maturity for Python-Rust interoperability.

Current State (December 2025)

• PyO3: Version 0.27.x, supporting Python 3.8+ with stable ABI
• Maturin: 5.2k+ GitHub stars, actively maintained
• Ecosystem: Battle-tested by major projects (tiktoken, tokenizers, pydantic-core, ruff, uv)

What Makes It “Solved”

1. Minimal configuration: Maturin’s opinionated defaults mean most projects need almost no setup
2. Full CI/CD support: Automated wheel building for all platforms (Linux, macOS, Windows, multiple Python versions)
3. Type safety: Rust’s type system catches errors at compile time
4. Zero-copy where possible: Efficient data transfer between Python and Rust
5. Bidirectional: Call Rust from Python and Python from Rust
6. Mature packaging: Direct PyPI upload, manylinux compliance, free-threaded Python 3.13t support

Remaining Friction Points

• Call overhead: PyO3 has slightly higher call overhead than Cython for very frequent small calls
• Learning curve: Developers need proficiency in both Rust and Python
• Build times: Rust compilation is slower than C/Zig

Verdict: For teams willing to use Rust, this is production-ready with no significant gaps.

2. Python↔Zig: The Current Landscape

2.1 Ziggy Pydust

Ziggy Pydust is currently the most mature framework for Python-Zig integration. It provides:

• Wrappers for CPython Stable API
• Poetry integration for wheel building
• pytest plugin for running Zig tests
• Buffer Protocol support

Current Status (December 2025): Actively Maintained, One Version Behind

• Supports Zig 0.14 (current stable is 0.15.1, released ~4 months ago)
• Release 0.26.0 published September 2025
• 698 stars, 481 commits, 13 contributors
• Requires CPython ≥3.11
• Documentation at pydust.fulcrum.so

What Pydust Provides

• Zig-native syntax for defining Python modules (comptime decorators)
• Automatic type conversion between Python and Zig types
• Poetry integration for wheel building
• pytest plugin for running Zig tests

The Trade-off Question

Pydust’s benefits are primarily ergonomic - nicer syntax for defining Python bindings. However, this comes at the cost of being locked to an older Zig version. For a language evolving as rapidly as Zig, this is a significant constraint.

With the HPy + C shim approach (see Section 4), a thin C layer (~50-100 lines) handles Python interaction, while Zig code exports standard extern functions. This means:
- Use any Zig version
- C shim rarely needs updates
- Works on PyPy/GraalPy, not just CPython

When Pydust Makes Sense

• You’re starting a new project and can accept Zig 0.14
• Ergonomics matter more than using latest Zig features
• You only need CPython support
• You want the most “Zig-native” experience

When to Skip Pydust

• You need Zig 0.15+ features
• You need PyPy/GraalPy support
• You’re already comfortable writing C shims
• You want to avoid framework version dependencies

2.2 PyOZ: Promising but Very New

PyOZ is a new project (started November 2025) targeting Zig 0.15.0+.

Current Status (December 2025):

• Very early stage - project is ~3 weeks old
• No PyPI release yet - must install from source
• Supports Zig 0.15.0+ and Python 3.8-3.13
• 32 stars, active development
• Documentation at pyoz.dev

Advertised Features:

• Declarative API for defining modules and functions
• Automatic type conversion between Zig and Python
• Full class support with magic methods and operators
• Zero-copy NumPy array integration
• Error handling: Zig errors → Python exceptions
• Automatic .pyi stub generation for IDE support
• CLI tooling: pyoz init, pyoz build, pyoz publish

Considerations:

• Too new to recommend for production use
• No proven track record
• API may change significantly
• But: worth watching as it matures
• Only option currently targeting Zig 0.15+

Wait for a stable release before adopting. Monitor for progress.

2.3 py.zig: Lightweight Alternative

py.zig takes a minimalist approach as a “lighter alternative to ziggy-pydust.”

Current Status:

• Alpha status with breaking changes expected
• v0.13.0 released March 2025
• 42 commits, no recent updates
• Uses setuptools-zig for building

Interesting Design Ideas:

• Dual-access pattern: Type wrappers have the same memory representation as C equivalents, allowing safe @ptrCast to raw C API when needed
• Minimal abstraction: Exposes both wrapped methods and raw C API via impl field
• Protocol composition: Uses usingnamespace for Object/Sequence protocol support

Considerations:

• Best suited for developers comfortable with lower-level code
• Useful if you need to mix Zig wrappers with direct C API calls
• May contain ergonomic ideas worth borrowing even if not used directly

2.4 Framework Comparison

Framework	Zig Version	Status	Ergonomics	Maturity
Pydust	0.14 only	Active	High	Established
PyOZ	0.15.0+	Active	High	Very new (~3 weeks)
py.zig	Unknown	Alpha	Medium	Experimental

2.5 The cffi/ctypes Escape Hatch

The most reliable Python-Zig integration today uses the C ABI:

1. Write Zig code with export functions using C-compatible types
2. Compile to a shared library (.so/.dylib/.dll)
3. Call from Python via cffi or ctypes

Advantages:
- Works with any Zig version
- No framework dependencies
- cffi has good performance (faster than ctypes)
- Battle-tested approach

Disadvantages:
- Limited to C-compatible types at the boundary
- No automatic type conversion or error handling
- Manual memory management across the boundary
- No Python object integration (can’t return Python lists, dicts, etc.)
- Requires function signature declarations in Python

This approach works for computation-heavy libraries with simple interfaces, but doesn’t scale to complex APIs.

3. Experimental Findings: Four FFI Approaches Tested

We conducted experiments comparing four different FFI approaches for Python-Zig integration, using a performance-critical workload (regex matching, string transformations, phrase matching).

3.1 Approaches Tested

Approach	Status	Overhead	Notes
cffi	Working	~0.5µs	High string marshalling overhead
Pydust	Working	~0.25µs	PyO3-like ergonomics, now supports Zig 0.14
C-API	Working	~0.3µs	Direct Python C extension, complex
HPy + C Shim	Working	~0.22µs	Stable ABI, works on PyPy/GraalPy

Note: Our experiments were conducted when Pydust was stalled at Zig 0.11. With Pydust now supporting Zig 0.14, it should be re-evaluated as a primary option.

3.2 Critical Finding: FFI Overhead is NOT the Bottleneck

Rust FFI call (PyO3):     0.234µs
Zig FFI call (HPy shim):  0.222µs
Ratio:                    0.95x (essentially identical)

Both Rust (PyO3) and Zig (HPy + C shim) achieve similar FFI overhead. The performance gap in real benchmarks comes from algorithm and library choices, not FFI efficiency.

3.3 Algorithm Choice Matters More Than Language

Benchmark	Python	Rust	Zig	Root Cause
regex_complex_long	2.38µs	0.17µs	12.79µs	DFA vs PCRE2 backtracking
phrase_match_large	134µs	49µs	182µs	Aho-Corasick vs linear search
transform_lowercase	0.12µs	0.12µs	3.75µs	Python builtins are C-optimized

The Rust implementation used:
- DFA-based regex (the regex crate) - O(n) guaranteed
- Aho-Corasick for phrase matching - O(n + m)

The Zig implementation used:
- PCRE2 (backtracking) - O(n) to O(2^n) worst case
- Linear search for phrase matching - O(n × p)

Lesson: To match Rust performance, a Zig implementation needs equivalent algorithms, not just a different language. The Rust ecosystem provides battle-tested crates (regex, aho-corasick) that would need to be reimplemented or wrapped in Zig.

4. The HPy-Zig Incompatibility Issue

When attempting to use HPy directly from Zig via @cImport, we discovered a fundamental incompatibility that affects any Zig project trying to use HPy.

4.1 The Problem

HPy defines handles using intptr_t (signed):

typedef struct _HPy_s { intptr_t _i; } HPy;

HPy’s conversion function casts this to a pointer:

static inline PyObject* _h2py(HPy h) {
    return (PyObject*) h._i;  // Valid C: cast signed int to pointer
}

Zig’s @cImport translates this to:

return @as([*c]PyObject, @ptrFromInt(h._i));  // Error: @ptrFromInt requires usize

But @ptrFromInt requires usize (unsigned), while h._i is isize (signed). Compilation fails.

This is valid C (per C99 §7.18.1.4), but Zig’s stricter type system rejects the automatic translation.

4.2 Workarounds

1. Preprocessor Override (works):

const hpy = @cImport({
    @cInclude("stdint.h");
    @cDefine("intptr_t", "uintptr_t");  // Shadow with unsigned
    @cInclude("hpy.h");
});

The bit patterns are identical; we’re just telling Zig to treat the value as unsigned.

2. translate-c + Manual Patch (works):

zig translate-c hpy.h > hpy_binding.zig
# Manually change: @ptrFromInt(h._i)
# To: @ptrFromInt(@as(usize, @bitCast(h._i)))

3. C Shim Wrapper (recommended):

Python → HPy Module (C) → extern functions → Zig implementation

The C shim handles all HPy interaction, exposing simple C functions to Zig:

// hpy_shim.c - compiled as C
 <hpy.h>

extern bool zig_regex_match(const char* pattern, const char* text);

HPyDef_METH(regex_match, "regex_match", HPyFunc_VARARGS)
static HPy regex_match_impl(HPyContext *ctx, HPy self, const HPy *args, size_t nargs) {
    // Extract strings, call Zig, return result
    bool result = zig_regex_match(pattern, text);
    return HPyBool_FromLong(ctx, result);
}

This approach:
- Bypasses translation issues completely
- Has minimal overhead (~0.22µs)
- Uses HPy as intended (compiled C)
- Is maintainable (shim rarely changes)

This issue should be reported to both Zig (requesting @bitCast insertion for signed→pointer casts) and HPy (for awareness).

5. HPy as a Foundation

What is HPy?

HPy is a redesigned C API for Python extensions, developed collaboratively by the GraalPy and PyPy teams. The “H” stands for “handle”—the core innovation is using opaque handles instead of raw PyObject* pointers.

Key Features

1. Implementation-agnostic: One binary works on CPython, PyPy, GraalPy
2. ABI stability: Extensions don’t need recompilation for new Python versions
3. Zero overhead on CPython: When compiled in “CPython ABI mode”, performance equals traditional C extensions
4. Better on alternative Pythons: 3x faster than C API on PyPy (measured with ujson)
5. Future-proof: Hides implementation details that may change

Current Status (HPy 0.9, approaching stable)

• Active development by Oracle (GraalPy) and PyPy teams
• Major ports underway: numpy, kiwi-solver (Matplotlib dependency)
• API considered stable enough for binary compatibility promises
• Production-ready for new projects

HPy vs PyO3: Different Goals

Aspect	HPy	PyO3
Source Language	C (usable from any language with C FFI)	Rust only
Target	Any Python implementation	CPython only
ABI Stability	Universal binary across Python versions	Per-version builds
Performance on CPython	Equal to C API (CPython mode)	Slight overhead vs Cython
Performance on PyPy/GraalPy	Excellent	N/A
Ecosystem Maturity	Approaching stable	Production-ready

6. Is HPy Good Enough for Zig?

The Case For HPy

1. C-compatible API: HPy is a C API, and Zig has excellent C interop
2. No comptime magic needed: Unlike Pydust, HPy doesn’t require compile-time introspection of Python types
3. Stable target: HPy’s API is stabilizing, unlike Zig’s evolving comptime semantics
4. Portable results: A Zig library built against HPy would work on CPython, PyPy, and GraalPy

The Case Against (Current Limitations)

1. No Zig bindings exist: Someone needs to create hpy.zig wrapper headers
2. Manual work: HPy reduces boilerplate compared to raw C API, but still requires more ceremony than PyO3
3. Handle management: HPy’s handle semantics require explicit lifetime management (close handles when done)
4. Ecosystem size: HPy has fewer examples, tutorials, and Stack Overflow answers than PyO3
5. The intptr_t issue: Requires workaround (see Section 4)

Viability Assessment

HPy is a viable foundation for Python-Zig interop, but significant work is needed:

1. Zig bindings for HPy headers (~1-2 weeks for core API)
2. Zig-idiomatic wrapper layer (~2-4 weeks for ergonomic API)
3. Build system integration (Meson, setuptools, or custom)
4. Documentation and examples

This approach is worth considering if you need PyPy/GraalPy support, as HPy provides a stable ABI across Python implementations.

7. What Needs to Be Done

Option A: HPy + C Shim (Recommended)

Effort: Low-Medium (write a thin C shim)
Risk: Low (no Zig version constraints)

This approach decouples Python binding from Zig version:

1. Write a C shim (~50-100 lines) that uses HPy to define the Python module
2. The C shim calls extern functions exported by Zig
3. Zig code is pure computation, version-independent

Advantages:
- Use any Zig version, including 0.15.1+
- Works on CPython, PyPy, and GraalPy
- ABI stability across Python versions
- C shim rarely needs updates

Recommended for most projects.

Option B: Use Ziggy Pydust

Effort: Low (it’s ready to use)
Risk: Medium (Zig version lag, CPython-only)

Pydust offers the best developer experience if you can accept constraints:

1. Follow the getting started guide
2. Use the template repository for new projects
3. Leverage Poetry integration for packaging

Considerations:
- Currently supports Zig 0.14 (0.15.1 is current as of December 2025)
- Expect ~3-6 month lag when new Zig versions are released
- CPython ≥3.11 only (no PyPy/GraalPy)

Choose if ergonomics outweigh version constraints for your project.

Option C: cffi

Effort: Low (per-project basis)
Risk: Low

Suitable when:
- Interface is simple (numbers, arrays, strings)
- No need to manipulate Python objects from Zig
- Computation dominates (call overhead doesn’t matter)

Acceptable for specific use cases, but doesn’t solve the general problem.

Option D: Direct Python C-API

Effort: Medium-High
Risk: Medium (API changes between Python versions)

Required:
- Manual reference counting
- Version-specific builds
- Complex error handling

Not recommended unless you need features unavailable in HPy.

8. Community and Contributions

Contributing to Ziggy Pydust

Pydust welcomes contributions via GitHub. Key areas:
- Tracking new Zig releases
- Expanding Python version support
- Documentation improvements
- Example projects

HPy-Zig Bindings (Future Work)

If PyPy/GraalPy support becomes important, HPy-Zig bindings would be valuable.

Ideal contributors:
- Zig Foundation or community members interested in Python interop
- Python alternative implementation teams (GraalPy, PyPy) wanting broader HPy adoption
- Companies needing cross-implementation support

Funding possibilities:
- Sovereign Tech Fund (funds critical open source infrastructure)
- Python Software Foundation grants
- Corporate sponsors

9. Strategic Recommendations: Zig vs Rust

Based on our experimentation, here’s guidance on when to use each approach:

When to Choose Rust (via PyO3)

Use Case	Rationale
Maximum performance NOW	Ecosystem is mature (regex, aho-corasick, serde, etc.)
Team already knows Rust	No learning curve
Complex Python object manipulation	PyO3’s ergonomics are excellent
Long-term production use	Battle-tested, large community

When to Choose Zig

Use Case	Rationale
WASM/edge deployment	Excellent WASM compilation, smaller binaries
Comptime code generation	Patterns baked into binary at compile time
Learning/exploration	Educational value, simpler than Rust
C library integration	First-class C interop without bindgen
Minimal runtime	No allocator required for many operations

The Hybrid Option

Use Rust’s algorithms via their C APIs, with Zig for glue:

Python → HPy → Zig glue → rure (Rust regex C API)
                        → aho-corasick (Rust C API)

This gives Rust’s optimized, battle-tested algorithms with Zig’s build system and comptime capabilities. The rure crate specifically provides a stable C API for Rust’s regex engine.

10. Practical Recommendations

Short Term (Now)

1. For simple interfaces: Use cffi with C-ABI Zig libraries
2. For complex APIs: Consider whether Rust/PyO3 is acceptable instead
3. For research/experimentation: Try py.zig or PyOZ, report issues

Medium Term (2026)

1. Invest in HPy-Zig bindings: This is the most sustainable path
2. Engage with HPy community: Ensure Zig use cases are considered
3. Document cffi patterns: Create templates for common Zig-Python patterns

Long Term

1. Stabilize HPy-Zig: Aim for a “hpy-zig” package and proper documentation
2. Integration with build tools: zig build → wheel pipeline
3. Lobby for Zig comptime stability: The current churn blocks sophisticated tooling

11. Conclusion

Python↔Zig interoperability has matured significantly in 2025. Ziggy Pydust now supports Zig 0.14 and is actively maintained, making it a viable option for production use.

Key findings from our experiments:

1. FFI overhead is not the bottleneck - Zig and Rust achieve similar call overhead (~0.22µs)
2. Algorithm choice dominates performance - Use DFA regex and Aho-Corasick, not PCRE2 and linear search
3. Multiple viable approaches exist - Pydust, HPy + C shim, and cffi all work

Recommended approaches:

Use Case	Recommendation
Any Zig version, PyPy/GraalPy support	HPy + C shim - most flexible
CPython-only, can accept Zig 0.14	Ziggy Pydust - best ergonomics
Simple interfaces, minimal dependencies	cffi - battle-tested
Maximum ecosystem maturity	Rust/PyO3 - still the gold standard

Note: PyOZ targets Zig 0.15+ but is too new (3 weeks, no PyPI release) to recommend. Monitor its progress.

For most projects, HPy + C shim offers the best trade-off: use any Zig version, support alternative Python implementations, and avoid framework version lock-in.

Python↔Zig is no longer “problematic” - it’s now a reasonable choice when Zig’s strengths (comptime, C interop, small binaries, WASM) align with project requirements. For most projects prioritizing ecosystem maturity and developer availability, Rust/PyO3 remains the safer choice.

References

Python-Rust Ecosystem

• PyO3 User Guide
• Maturin Documentation
• rure - Rust regex C API

Python-Zig Frameworks

• PyOZ GitHub
• PyOZ Documentation
• Ziggy Pydust GitHub
• Ziggy Pydust Documentation
• py.zig GitHub

HPy

• HPy Project
• HPy Documentation
• HPy: Better Python C API in Practice

General Resources

• How to Boost Python Performance with Zig (InfoWorld)
• Optimizing Python with Zig for Numerical Calculations
• cffi Documentation

Page last modified: 2025-12-07 20:53:18