Rust is a remedy, not a diagnosis
Rust earns its place in production systems where predictable latency, low memory use and safe concurrency matter, and it has become a common way to speed up the CPU-heavy parts of Python AI stacks. That makes it an easy answer to propose whenever something is slow. It is not always the right one. A rewrite costs months, splits your codebase across two languages, and sometimes moves the bottleneck rather than removing it.
This engagement starts from the other end. Before anyone writes Rust, we reproduce the problem, measure it and find out where the time and memory actually go. Then the decision is made on evidence. My published guide to Rust in production AI systems sets out the same reasoning in more depth.
What the work involves
Reproducing the workload. A benchmark that exercises the slow path with production-shaped inputs and measures the metric you care about. Without it, nobody can say whether a change helped.
Profiling. CPU sampling and flame graphs to find hot functions, allocation profiling for memory growth, and lock and scheduler analysis for concurrent services. For Python services, profiling shows whether the time is in Python code, in native libraries already, or waiting on I/O, and only the first case is a candidate for Rust.
Deciding. The decision record lists the options the profile supports, with expected effect and cost. Typical outcomes:
- a fix in the existing language (better algorithm, batching, caching, a faster library);
- a narrow Rust extension module called from Python for one hot path;
- a Rust library or service behind an existing interface;
- or no change, because the bottleneck is elsewhere.
Building. If Rust is warranted, I implement the narrowest change: typically an extension built with PyO3 or a small service, behind the interface the rest of your system already uses, with a fallback to the old path during rollout.
Proving. An equivalence test and benchmarks that run in CI. For existing Rust services, the same profile-first approach applies to async runtime configuration, allocation patterns, data layout and contention.
My recent work has been in this area: performance-critical components of an AI agent stack in regulated financial services, and Rust-accelerated drop-in components for production AI stacks at Neul Labs.
The signature deliverable
You receive Rust systems engineering and profiling: the benchmark, the profiles, the decision record and, where warranted, the change with its equivalence test. Illustrative example of a decision record summary:
| Option | Profile evidence | Expected effect | Cost and risk | Recommendation |
|---|---|---|---|---|
| Batch tokeniser calls | Most of request time in per-item calls with fixed overhead | Large reduction in call overhead | Small change, no new language | Do first |
| Rust extension for scoring loop | Pure-Python loop dominates remaining CPU time | Substantial reduction in CPU per request | Small Rust module, needs one owner | Do if batching is not enough |
| Rewrite service in Rust | Most remaining time is network and database wait | Little effect on latency | Months of work, two codebases | Do not do |
Illustrative example. Not taken from a client engagement.
How acceptance is judged
Acceptance is set before the build: the benchmark and target metric, the equivalence test and its tolerance, and the operational checks (memory under load, behaviour when the extension fails). Your platform lead accepts the decision record and, if commissioned, the build against those measures.
Ownership and handover
All code, benchmarks and profiles live in your repository. Benchmarks run in CI so regressions are caught after I leave. The engineer who will own the Rust code pairs with me on the build and receives a walk-through of the design, the build tooling and the release process.
When to choose something else
If the bottleneck is model serving, GPU use or provider latency, see AI infrastructure, inference and cost optimisation or reducing LLM cost and latency. For ongoing Rust capacity under your own manager, see hire a contract AI engineer.