Rust Mastery Notes
This is my learn-by-doing knowledge base — the distilled lessons from each
Rust concept I drill in rust-scratch.
For every concept I finish a ladder of 7–9 problems (foundations → mechanics →
footguns → real-world patterns → a build-it-from-scratch capstone), living in
src/bin/<concept>.rs. Once a ladder is done, I write the lasting takeaways here —
the mental model, the signatures worth memorizing, the footguns I hit, and the
“explain it back” prompts that prove I actually own it.
The code is the gym. This is the notebook.
How to read a note
Each concept page follows the same shape:
| Section | What’s in it |
|---|---|
| TL;DR | The one-paragraph mental model. |
| Why it exists | The problem this concept solves. |
| The ladder | The 7–9 rungs I worked, what each one taught. |
| Signatures to know | The std types/bounds worth memorizing. |
| Footguns | The traps I hit (or that the ladder deliberately set). |
| Explain it back | Questions I should be able to answer cold. |
Completed concepts
| Concept | Note | Source ladder |
|---|---|---|
| Modules & visibility | ✅ note | src/bin/modules.rs |
Cargo features & cfg | ✅ note | src/bin/features_cfg.rs |
| Testing | ✅ note | src/bin/testing.rs + practice/testing_lab/ |
Cow | ✅ note | src/bin/cow.rs |
Box & the heap | ✅ note | src/bin/box_heap.rs |
Rc / Arc | ✅ note | src/bin/rc_arc.rs |
Cell / RefCell | ✅ note | src/bin/cell_refcell.rs |
Rc<RefCell<T>> patterns | ✅ note | src/bin/rc_refcell.rs |
| Conversion traits | ✅ note | src/bin/conversions.rs |
| Lifetimes in depth | ✅ note | src/bin/lifetimes_depth.rs |
HRTB — for<'a> | ✅ note | src/bin/hrtb.rs |
Borrow / ToOwned | ✅ note | src/bin/borrow_toowned.rs |
Drop & ordering | ✅ note | src/bin/drop_ordering.rs |
| Associated types vs generic params | ✅ note | src/bin/assoc_vs_generic.rs |
Generic bounds & where clauses | ✅ note | src/bin/generic_bounds.rs |
| Blanket impls & coherence | ✅ note | src/bin/blanket_coherence.rs |
| Static vs dynamic dispatch | ✅ note | src/bin/dispatch.rs |
Closures & Fn/FnMut/FnOnce | ✅ note | src/bin/closures.rs |
impl Trait & RPIT | ✅ note | src/bin/impl_trait.rs |
| Marker & auto traits | ✅ note | src/bin/marker_auto_traits.rs |
| Error handling architecture | ✅ note | src/bin/error_arch.rs |
| Custom error types | ✅ note | src/bin/custom_errors.rs |
| Builder pattern | ✅ note | src/bin/builder.rs |
| The typestate pattern | ✅ note | src/bin/typestate.rs |
| Newtype & zero-cost wrappers | ✅ note | src/bin/newtype.rs |
| API evolution & semver | ✅ note | src/bin/semver.rs |
| Threads & scoped threads | ✅ note | src/bin/threads.rs |
Send & Sync deeply | ✅ note | src/bin/send_sync.rs |
Mutex / RwLock | ✅ note | src/bin/mutex_rwlock.rs |
| Channels | ✅ note | src/bin/channels.rs |
Data parallelism with rayon | ✅ note | src/bin/rayon_parallel.rs |
| Shared state vs message passing | ✅ note | src/bin/concurrency_models.rs |
| Collections deep-dive | ✅ note | src/bin/collections.rs |
| Strings & text | ✅ note | src/bin/strings_text.rs |
| Iterators end-to-end | ✅ note | src/bin/iterators.rs |
New notes get added under Concepts as each ladder is finished — see Adding a new note.
Modules & visibility
Ladder:
src/bin/modules.rs· Run:cargo run --bin modules· Phase 0 · 9 rungs
TL;DR
A crate is a tree of modules rooted at the crate root. mod declares a node
in that tree — it imports nothing. Everything is private by default, and
privacy is tree-relative: an item is visible to its own module and all of that
module’s descendants; a parent can only see into a child what the child marks
pub. Paths walk the tree (crate:: from the root, super:: up one, self::
here). use is just a local alias for a path; pub use is a re-export that
adds a brand-new public path. Master those three facts and everything else —
field privacy, pub(crate), facades, sealed traits — is a corollary.
Why this exists (from first principles)
Without modules, every name in a crate lives in one flat namespace, and every function is callable from everywhere. Two problems follow immediately:
- No encapsulation. If any code can call any function and read any field,
you can’t enforce invariants. A
Celsiuscould be set to -1000; a half-built value could be observed. “It’s an internal detail” becomes a comment, not a guarantee. - No stable API. If your internal organization is your public surface, you can’t rename a helper or move a file without breaking everyone who depends on you. Refactoring becomes a breaking change.
Modules + visibility solve both. The module tree gives you namespaces and a
place to hide things; pub and its restricted variants let you publish
exactly the surface you intend, and nothing more. The compiler enforces it —
privacy is a checked rule, not advice. That’s the whole point: the things you
don’t mark pub are things you are free to change.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | Foundations | Module tree & paths | mod builds a tree; crate::/super::/self:: walk it |
| 2 | Foundations | pub opens a door | Privacy is tree-relative; keep helpers private |
| 3 | Mechanics | Field privacy | pub struct ≠ pub fields; private field + smart constructor guards an invariant |
| 4 | Mechanics | use & pub use | Local alias vs re-export; flatten a deep tree |
| 5 | Footgun | Leaking a private type | A pub fn can’t honestly expose a less-visible type |
| 6 | Footgun | Restricted visibility | pub(crate) / pub(super) / pub(in path): pick the narrowest reach |
| 7 | Real-world | Facade pattern | Private internals, curated public surface via pub use |
| 8 | Real-world | Sealed trait | A private-module supertrait gates who can impl |
| 9 | Capstone | inventory mini-library | A full module tree with a real front door |
The ideas, built up
1. A module is a node; paths walk the tree
mod kitchen { ... } does not “import” anything. It creates a node named
kitchen under the current module, and the items inside it live at
kitchen::.... To reach an item you spell a path, and the prefix you choose
says where the path starts:
fn oven_temp() -> u32 { 220 } // lives at the crate ROOT
mod kitchen {
pub mod pantry {
pub fn flour_grams() -> u32 { 500 }
}
pub mod stove {
pub fn bake() -> (u32, u32) {
// pantry is a SIBLING of stove → go up one (super), then down
// oven_temp is at the ROOT → start from the root (crate)
(super::pantry::flour_grams(), crate::oven_temp())
}
}
}
Three prefixes, three starting points:
| Prefix | Starts at | Use when |
|---|---|---|
crate:: | the crate root (absolute) | the item lives near the root / “shared, top-level” |
super:: | the parent module (relative, up one) | reaching a tight sibling or parent helper |
self:: | the current module (relative) | disambiguating a local name; rarely needed |
Key insight:
super::pantryandcrate::kitchen::pantryresolve to the same item here. The difference is robustness to change.super::survives the whole subtree being moved or renamed;crate::survives local shuffling. Reach forsuper::for tight siblings,crate::for things that live near the root.
2. pub opens a door — and privacy is about the tree
By default an item is private to its module. Privacy is relative: an item is
visible to its defining module and every descendant of that module. A parent
sees a child’s item only if it’s pub.
mod billing {
fn tax_rate() -> u32 { 8 } // PRIVATE: internal detail
pub fn total_with_tax(price: u32) -> u32 { // PUBLIC entry point
price + price * tax_rate() / 100 // can call the private helper
}
}
// at the crate root:
billing::total_with_tax(100); // OK — it's pub
// billing::tax_rate(); // E0603: function `tax_rate` is private
Two subtleties worth internalizing:
mod billingitself needed nopubbecausebillingand its caller both live at the crate root — same module, already visible. Ifbillingwere nested inside another module, the outside would needpub mod billing.E0603is a hard error. Privacy isn’t a lint you can ignore; the compiler refuses to let unrelated code reach in.
3. pub struct is not pub fields
Marking a struct pub makes the type nameable. Its fields stay private unless
each one is individually pub. That asymmetry is the single most useful tool in
the whole topic: publish the type, hide the data, force everyone through your
methods.
mod temperature {
#[derive(Debug, PartialEq)]
pub struct Celsius {
degrees: i32, // PRIVATE — note: no `pub`
}
impl Celsius {
pub fn new(d: i32) -> Option<Celsius> { // smart constructor
if d >= -273 { Some(Celsius { degrees: d }) } else { None }
}
pub fn get(&self) -> i32 { self.degrees }
}
}
// let bogus = temperature::Celsius { degrees: -1000 }; // E0451: field is private
Because the field is private, the struct literal Celsius { degrees: ... } is
forbidden outside the module (E0451), and reading .degrees directly is too
(E0616). The only way to obtain a Celsius is new, which checks the
invariant. This is parse, don’t validate enforced by the module system: a
Celsius existing is proof it’s valid — downstream code never re-checks.
Inside the defining module you can still use the literal freely — privacy only bites when you cross the module boundary.
4. use aliases; pub use re-exports
These look similar and do fundamentally different things.
mod deep { pub mod nested { pub mod core {
pub struct Engine { pub power: u32 }
impl Engine { pub fn new(power: u32) -> Engine { Engine { power } } }
}}}
fn start_deep() -> u32 {
use crate::deep::nested::core::Engine; // LOCAL alias — only this fn sees it
Engine::new(9000).power
}
pub use deep::nested::core::Engine; // RE-EXPORT — adds crate::Engine
useis private plumbing: it shortens a path for the current scope and changes nobody else’s view of the tree.pub useis API surface: it makes the item reachable through a new path (crate::Engine), in addition to its real one.
This is how every mature crate is laid out: deep folders internally for
organization, a thin layer of pub use at the root so users write tokio::spawn
instead of tokio::runtime::task::spawn. The internal tree is an implementation
detail; the re-exports are the contract.
5. You can’t leak a private type through a public API
If a pub fn returns (or accepts) a type less visible than the function
itself, that’s a leak: an outside caller would receive a value of a type they
can’t name or use. The compiler stops you.
mod widget {
pub struct Inner { pub id: u32 } // must be pub to be honestly returned
pub fn make(n: u32) -> Inner { Inner { id: n } }
}
let w: widget::Inner = widget::make(7); // naming the type needs it pub (else E0603)
assert_eq!(w.id, 7); // reading the field needs it pub (else E0616)
Where the error shows up depends on the crate kind. In a library crate, exposing a private type in a
pubsignature fires theprivate_interfaceslint (and historically the hard errorE0446). In a binary like this ladder, there’s no external consumer, so the leak instead bites at the use site: callers can’t name the private type (E0603) or read its private fields (E0616).
The fix isn’t to silence the error — it’s to decide the type’s honest visibility. If you return it, you must publish it (this rung). If it should have stayed internal, don’t return it — hide it behind a facade (rung 7).
6. The visibility dial: pub(crate), pub(super), pub(in path)
pub and private are the two extremes. Between them sits a dial: “public, but
only up to here.”
mod engine {
pub(crate) const fn version() -> u32 { 3 } // anywhere in THIS crate
pub mod fuel {
pub(super) const fn secret_formula() -> u32 { 42 } // only the parent `engine`
}
pub mod electrical {
pub(in crate::engine) const fn calibrate() -> u32 { 7 } // the engine subtree
}
pub fn diagnostics() -> u32 {
fuel::secret_formula() + electrical::calibrate() // engine can see both
}
}
// engine::fuel::secret_formula(); // privacy error: pub(super) doesn't reach the root
| Marker | Visible to | Typical use |
|---|---|---|
pub(crate) | anywhere in this crate, not downstream | the workhorse: “shared internal API” |
pub(super) | the parent module and its descendants | a child exposing something to just its parent |
pub(in path) | within the named ancestor subtree | precise scoping when crate-wide is too loose |
Pick the narrowest visibility that satisfies the actual callers. In the ladder,
VERSIONcould even stay fully private, because the only thing reading it (version()) lives in the same module.pub(crate)would only earn its keep if some other module needed to readVERSIONdirectly. Don’t reach for a wider marker than the call sites demand.
Footguns
- Forgetting fields aren’t auto-
pub.pub struct Foo { x: i32 }exposes the type but notx. Outside code can’t build it with a literal (E0451) or readx(E0616). Usually that’s what you want — but it surprises people who expectedpub structto mean “all public.” - Returning a private type from a
pub fn.private_interfaceslint in libs,E0603/E0616at use sites in bins. The cure is to decide the type’s real visibility, not to paper over the symptom. pub(super)doesn’t reach the crate root unless the item’s parent is the root. It’s exactly one level of upward reach (plus descendants of that parent).usevspub usemix-up. A plainusein your lib root does not expose anything to downstream crates — it only aliases for your own code. If you meant to re-export, you needpub use.- Over-widening visibility to “make it compile.” The compiler will happily
accept
pubeverywhere; then your entire internal structure becomes API you can’t change. Tighten to the minimum the callers actually need.
Real-world patterns
The facade: private internals, curated surface
This is how production crates are organized. Implementation lives in private
modules; a thin layer of pub use exposes only the handful of names that form
the public API.
mod api {
mod internal { // PRIVATE — no `pub`
struct RawSocket; // guts the public API must NOT expose
pub struct Client { pub name: &'static str }
impl Client {
pub fn connect(name: &'static str) -> Self { Self { name } }
pub fn ping(&self) -> &'static str { "pong" }
}
}
pub use internal::Client; // re-export ONLY Client
}
api::Client::connect("db-1"); // OK
// api::internal::RawSocket; // E0603 — `internal` is private, path sealed
The payoff: you can rename RawSocket, split internal into ten files,
restructure freely — and nothing downstream breaks, because the only public
path is the curated pub use. The module tree stops being part of your API
contract.
The sealed trait: a private-module supertrait
Privacy can gate not just calling but implementing. Make your public trait
require a supertrait that lives in a private module. Outsiders can see and
call your trait, but can’t impl it — satisfying it requires impl’ing the
supertrait, which they can’t even name.
mod format {
mod sealed { pub trait Sealed {} } // private module → unnameable downstream
pub trait Encoder: sealed::Sealed { // supertrait bound is the seal
fn encode(&self, input: &str) -> String;
}
pub struct Json;
impl sealed::Sealed for Json {} // only possible INSIDE `format`
impl Encoder for Json {
fn encode(&self, input: &str) -> String { format!("json:{input}") }
}
}
// Downstream:
// struct Rogue;
// impl format::Encoder for Rogue { ... } // error: Rogue: format::sealed::Sealed
// // not satisfied — and you can't impl it
The “sealed trait” you meet in typestate and blanket_coherence is, mechanically,
just this: privacy applied to a supertrait. Because no downstream impl can ever
exist, you’re free to add methods to Encoder later without breaking anyone — a
genuine API-evolution tool.
Capstone insight
Rung 9 assembles everything into an inventory “library in a file”:
inventory (the library root)
├── util (PRIVATE) pub(crate) normalize() — shared by submodules
├── model (PRIVATE) Sku (private field + smart constructor), Item
├── store (PRIVATE) Warehouse (private items: Vec<Item>)
└── (facade) pub use model::{Sku, Item}; pub use store::Warehouse;
The structural “aha”: every visibility decision is driven by who the actual caller is, and the public surface is a deliberate, tiny re-export layer.
util::normalizeispub(crate)— bothmodelandstorecall it, so crate-wide is the right reach, but it’s not part of the public API. (Apub(super)would have been too narrow once two different submodules needed it.)Skuispubwith a privatecodefield: the only way to get one is the normalizing, length-checkingnew, so an existingSkuis provably valid.- The three submodules (
util,model,store) are all private. The only way in is the facade —inventory::Skuworks,inventory::model::Skudoes not. - Two seal probes prove it:
inventory::store::Warehouse::new()is unreachable (the submodule is private) andinventory::Sku { code: ... }is forbidden (the field is private). The invariant and the encapsulation are both enforced, not hoped for.
Build this once and the mental model locks in: the module tree is your private
workshop; pub use is the storefront window; and the narrowest pub(...) that
satisfies the real callers is always the right answer.
Explain it back
- Why does
mod foo;not “import” anything? What does it actually do? super::barvscrate::foo::barresolve to the same item — when would you prefer each, and why?- Why does marking a struct
pubnot make its fields public, and how is that the foundation of “parse, don’t validate”? - What’s the difference between
use path::Thingandpub use path::Thing? Which one is part of your crate’s public API? - A
pub fnreturns a private struct. What happens in a library crate vs a binary crate, and what are the two honest fixes? - When would you choose
pub(crate)overpub(super)? Over leaving it private? - How does a private module turn a public trait into a sealed trait, and why is that an API-evolution tool?
- In the facade pattern, what exactly are you free to change without breaking downstream code, and why?
See also
- The typestate pattern — sealed
Statetrait via a private supertrait. - Blanket impls & coherence — the sealed extension-trait pattern.
- API evolution & semver — sealed traits and
#[non_exhaustive]as evolution levers. - Newtype & zero-cost wrappers — private fields + smart constructors for invariants.
Cargo features & cfg
Ladder:
src/bin/features_cfg.rs· Run:cargo run --bin features_cfg· Phase 0 · 9 rungs
TL;DR
Conditional compilation lets the compiler decide which code even exists before
it type-checks anything, based on cfg predicates (target_os, feature = "x",
test, debug_assertions, …). Two faces of one idea:
#[cfg(...)]— an attribute that includes or excludes an item entirely. Excluded code is never compiled. It can be broken Rust and nobody notices until the predicate flips.cfg!(...)— a macro that evaluates to a plainboolat runtime. Both branches around it are always compiled.
A feature is a named cfg flag you declare in Cargo.toml [features] and turn
on with --features. The single law that governs all feature design:
features must be additive — turning one on may only add behavior, never
remove or change it, because Cargo compiles your crate once with the union of
every feature anyone in the dependency graph requested.
Why this exists (from first principles)
One source tree has to compile for many worlds: Linux and Windows, 64-bit and
32-bit, debug and release, “with JSON support” and “without”. You could maintain
separate files or if everything at runtime, but both are bad: runtime ifs
still force you to compile and link code (and its dependencies) you’ll never run
on this target, and some code is literally uncompilable elsewhere (a Windows API
call on Linux).
Conditional compilation solves this by moving the decision before type
checking. #[cfg(target_os = "windows")] on a function means: on Linux, that
function does not exist — it is never parsed for types, never linked, costs
nothing. The compiler enforces only what survives the cfg filter.
Features generalize this from “facts about the target” to “knobs the user picks”. But features carry a hazard that platform cfgs don’t: they are shared. If crate A and crate B both depend on your crate and each asks for a different feature, Cargo does not build your crate twice. It builds it once with the union. That single fact is the source of every feature footgun and the reason for the additivity law.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | foundations | cfg!() macro | Runtime bool; both arms compile |
| 2 | foundations | #[cfg(...)] attribute | Item exists or not; twin defs would be E0428 |
| 3 | mechanics | First feature flag | Declare in [features], gate with #[cfg(feature)] |
| 4 | mechanics | cfg_attr | Conditionally apply an attribute (a derive) |
| 5 | footgun | Additivity | Mutually-exclusive features collide under unification |
| 6 | footgun | Missing-symbol trap | cfg! keeps both arms → E0425; gate the call site |
| 7 | real-world | Optional dependency | optional = true + dep:rand; off → not compiled |
| 8 | real-world | Feature graphs | default, umbrella features, --no-default-features |
| 9 | capstone | Mini config module | cfg(target_os) + additive renderers + cfg_attr + cfg! |
The ideas, built up
1. cfg!(...) — the macro: a compile-time bool
cfg!(predicate) evaluates to true or false at compile time, but it is used
at runtime like any other boolean. The key property: it does not delete code.
Both arms of an if cfg!(...) are fully compiled and type-checked.
fn build_profile() -> String {
let mut s = String::new();
if cfg!(debug_assertions) { // both the "debug" and "release" arms
s.push_str("debug"); // are compiled; one is chosen at runtime
} else {
s.push_str("release");
}
if cfg!(target_os = "linux") {
s.push_str(" on linux");
}
s
}
On a debug Linux build this returns "debug on linux". Because both arms compile,
cfg! is only safe when every arm is valid Rust on every target. That’s its
limitation and the reason rung 2 exists.
2. #[cfg(...)] — the attribute: an item either exists or it doesn’t
The attribute form is the exclusion tool. #[cfg(pred)] on an item means the
item is compiled only if pred holds; otherwise it vanishes before name
resolution. That lets you write two definitions of the same function gated by
mutually exclusive predicates:
#[cfg(target_pointer_width = "64")]
fn platform_tag() -> &'static str { "64-bit" }
#[cfg(not(target_pointer_width = "64"))]
fn platform_tag() -> &'static str { "non-64-bit" }
There is no “defined multiple times” error because, on any given target, only one
of these exists. The proof: change the second predicate to also be
target_pointer_width = "64" and you get E0428: the name 'platform_tag' is defined multiple times — the collision only appears once both items survive the
cfg filter. The cfg deletes one before the compiler ever sees a conflict.
This is the deep difference from
cfg!: the attribute can guard code that wouldn’t even compile on the other target. The macro cannot — both its arms must always be valid.
3. Your first feature flag
A feature is a cfg flag you invent. Unlike target_os, the name feature = "demo"
is unknown until you declare it in Cargo.toml:
[features]
demo = [] # the empty list = "enables no other features / optional deps (yet)"
Then #[cfg(feature = "demo")] works like any other cfg, and --features demo
turns it on:
#[cfg(feature = "demo")]
fn feature_line() -> &'static str { "demo enabled" }
With the feature off, feature_line does not exist. The same binary now compiles
two different ways depending on the flag:
cargo run --bin features_cfg # demo off — fn not compiled in
cargo run --bin features_cfg --features demo # demo on — fn exists
Since Rust 1.80, an undeclared feature name in a
#[cfg(feature = "...")]triggers theunexpected_cfgslint (“consider addingdemoas a feature in Cargo.toml”) instead of silently evaluating to false. That lint catches typo’d feature names — a class of bug that used to fail silently.
4. cfg_attr — conditionally applying another attribute
#[cfg(...)] includes or excludes an item. cfg_attr includes or excludes an
attribute:
#[cfg_attr(PREDICATE, attr1, attr2, ...)]
// reads as: if PREDICATE holds, expand to #[attr1] #[attr2] ...; else nothing
The canonical use is gating a derive:
#[derive(Debug, Clone)] // always wanted
#[cfg_attr(feature = "demo", derive(PartialEq))] // PartialEq only under `demo`
struct Point { x: i32, y: i32 }
Without demo, Point has Debug and Clone but no PartialEq, so p == q
won’t compile (and the test that does the comparison is itself behind
#[cfg(feature = "demo")]). This is exactly how every serde-supporting crate works:
#[cfg_attr(feature = "serde", derive(serde::Serialize, serde::Deserialize))]
One line; no duplicated struct definition.
5. The additivity law (the rung the whole concept exists for)
Cargo compiles your crate once with the union of every feature any crate in the graph requested. Therefore a feature turning on may only add behavior. The anti-pattern is two features that assume they are mutually exclusive:
// ANTI-PATTERN — collides under unification
#[cfg(feature = "metric")]
fn unit_label() -> &'static str { "meters" }
#[cfg(feature = "imperial")]
fn unit_label() -> &'static str { "feet" }
Each builds fine alone. But --features metric,imperial produces
E0428: defined multiple times — and that combination is exactly what unification
can force on you: crate A enables metric, crate B enables imperial, and your
build breaks through no fault of either. The victim never asked for the conflict.
The fix is to accumulate instead of choosing:
// ADDITIVE — enabling both just yields both
fn enabled_units() -> Vec<&'static str> {
let mut units = Vec::new();
if cfg!(feature = "metric") { units.push("meters"); }
if cfg!(feature = "imperial") { units.push("feet"); }
units
}
Now --features metric,imperial returns ["meters", "feet"]. The design rules
that fall out of this:
- Never gate the default behavior behind
#[cfg(not(feature = "x"))]— that’s a feature that disables something, which is removal in disguise. - Never make two features mutually exclusive — unification can turn both on.
- Enabling a feature must never break a consumer who didn’t ask for it.
6. The missing-symbol trap
If an item is gated behind a feature, every use of it must be gated too. This
is where the cfg! vs #[cfg] distinction becomes a correctness issue, not a
style choice. Consider a function that exists only under demo:
#[cfg(feature = "demo")]
fn pretty_print(label: &str, n: i32) -> String { format!("[ {label} = {n} ]") }
The tempting wrong solution uses the macro:
// WRONG — fails to compile when demo is OFF
fn describe(label: &str, n: i32) -> String {
if cfg!(feature = "demo") { pretty_print(label, n) } // E0425: cannot find `pretty_print`
else { format!("{label}={n}") }
}
cfg! keeps both arms compiled, so the off-build still tries to resolve
pretty_print, which doesn’t exist → E0425. The only correct solution gates the
call site with the attribute form, so each build names only what it has:
// OK — each build compiles only its own arm
fn describe(label: &str, n: i32) -> String {
#[cfg(feature = "demo")]
{ pretty_print(label, n) }
#[cfg(not(feature = "demo"))]
{ format!("{label}={n}") }
}
The same discipline applies to use imports — use rand::RngExt; must carry
#[cfg(feature = "dice")] or the off-build hits E0432: unresolved import. The
rule in one line: gate the definition AND every reference (calls and imports),
with the attribute, not the macro.
7. Optional dependency = a feature
A dependency you only sometimes need is marked optional = true. Cargo then does
not compile it by default; a feature pulls it in via the dep: syntax:
[dependencies]
rand = { version = "0.10", optional = true }
[features]
dice = ["dep:rand"]
#[cfg(feature = "dice")]
fn roll_die() -> u32 { rand::rng().random_range(1..=6) }
Two subtleties worth owning:
- An optional dependency implicitly creates a same-named feature (
rand) you could enable directly — unless you reference it asdep:randsomewhere, which suppresses that implicit feature and keeps your dependency names out of your public feature surface.dep:(Rust 1.60+) is the modern idiom. - In the default build,
randis not merely unused — it is not downloaded or compiled at all. That’s the real payoff: feature-gated bloat costs nothing when off, both in compile time and binary size.
8. Feature graphs
Features form a directed graph, not a flat list. Three mechanisms:
[features]
default = ["metric"] # on automatically; --no-default-features strips it
color = [] # a leaf feature
full = ["color", "demo"] # umbrella: enabling `full` transitively enables both
defaultis the set Cargo turns on for a plaincargo build.--no-default-featuresremoves it (common forno_std/embedded builds).- A feature listing other features enables them transitively. Turning on
fullis guaranteed to also turn oncoloranddemo. - The combination
--no-default-features --features fullmeans “minimal build, plus exactly this subtree”.
The active set behaves like a closure under “enables”:
| build | active features |
|---|---|
| default | ["metric"] |
--no-default-features | [] |
--features full | ["metric", "color", "demo"] |
--no-default-features --features full | ["color", "demo"] |
Footguns
| Trap | What bites | Fix |
|---|---|---|
| Mutually-exclusive features | E0428 when unification turns both on | Make features additive — accumulate, don’t choose |
| Feature that disables behavior | A consumer enabling it silently breaks another | On may only add; never gate defaults behind not(feature) |
cfg! to guard a gated symbol | E0425 in the off-build — both cfg! arms compile | Use #[cfg] on the call site, not cfg! |
Ungated use of a gated crate | E0432: unresolved import in the off-build | #[cfg(feature = "...")] on the use too |
| Typo’d feature name | #[cfg] silently always-false (pre-1.80) | Declare every feature; heed the unexpected_cfgs lint |
Real-world patterns
- serde support behind a feature — the universal
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))], withserdeas an optional dependency pulled in bydep:serde. Costs nothing when off. stdvsno_std— libraries expose astdfeature (often indefault) and use#![cfg_attr(not(feature = "std"), no_std)]so embedded users can opt out via--no-default-features.- Umbrella
fullfeature — big crates (e.g. tokio’sfeatures = ["full"]) ship one feature that enables a curated subtree, so users don’t have to know the whole list. - Platform shims —
#[cfg(target_os = "...")]selects one of several same-named functions, so callers write portable code over an OS-specific impl.
Capstone insight
The capstone builds a mod config that exercises the whole matrix at once — the
shape of a real crate’s config/output layer:
#[cfg(target_os = "linux")]
pub fn config_dir() -> &'static str { "/etc/app" } // platform path (attribute form)
#[cfg(not(target_os = "linux"))]
pub fn config_dir() -> &'static str { "/tmp/app" }
#[derive(Debug)]
#[cfg_attr(feature = "json", derive(serde::Serialize))] // serde derive only under json
pub struct Report { pub name: &'static str, pub level: u8 }
pub fn render(r: &Report) -> Vec<String> {
let mut lines = Vec::new();
#[cfg(feature = "json")] { lines.push(serde_json::to_string(r).unwrap()); }
#[cfg(feature = "pretty")] { lines.push(format!("{} @ level {}", r.name, r.level)); }
#[cfg(not(any(feature = "json", feature = "pretty")))]
{ lines.push(format!("{:?}", r)); } // fallback when neither is on
lines
}
The “aha” is that all four tools compose cleanly because each respects its lane:
cfg(target_os) picks exactly one config_dir; cfg_attr adds the Serialize
impl only when the JSON path needs it; the per-branch #[cfg] in render makes
the output additive — --features json,pretty emits both lines, and the
not(any(...)) fallback guarantees exactly one line when neither is on. The whole
module is a microcosm of the additivity law: every feature only ever adds a line.
Explain it back
- What’s the difference between
cfg!(...)and#[cfg(...)], and when does only one of them work? - Why does
if cfg!(feature = "x") { gated_fn() }fail to compile when the feature is off, while#[cfg(feature = "x")] { gated_fn() }succeeds? - Why must features be additive? Construct the two-crate scenario where a non-additive feature breaks an innocent consumer.
- What does
dep:randdo that barerandin a feature list does not? What is the implicit feature, and why suppress it? - Predict the active feature set for
--no-default-features --features fullgivendefault = ["metric"]andfull = ["color", "demo"].
See also
- Modules & visibility — what gets gated lives in modules;
#[cfg]andpubboth shape the surface a crate exposes. - Newtype & zero-cost wrappers —
#[cfg_attr(..., derive(...))]is the same conditional-derive machinery newtypes lean on for serde support.
Testing
Ladder:
src/bin/testing.rs
practice/testing_lab/· Run:cargo test --bin testingandcargo test -p testing_lab· Phase 0 · 9 rungs
TL;DR
A test in Rust is just a function tagged #[test]. The compiler bundles all
of them into a separate test harness binary; cargo test runs each one in
isolation and catches its panic. A test fails iff it panics — and every
assertion macro is just a fancy if !cond { panic!(...) }.
There are three places tests live, distinguished by scope:
| Kind | Lives in | Sees | Tests it from |
|---|---|---|---|
| Unit | #[cfg(test)] mod tests inside the source file | private items (white-box) | the inside |
| Integration | top-level tests/ dir (separate crate) | only pub items (black-box) | the outside, like a user |
| Doctest | /// examples in doc comments | only pub items | the docs, kept honest |
Everything else — assert_eq!, #[should_panic], Result-returning tests,
#[ignore], doctest fences — is detail layered on those two facts.
Why this exists (from first principles)
Most languages bolt testing on as a library: import a framework, register test
classes, run a separate tool. Rust builds it into the compiler and cargo,
and that design choice explains every quirk on this page.
Because the harness is part of the build, a test is an ordinary function the
compiler already type-checks. There is no “test runner reflection” — #[test]
is an attribute the compiler collects at compile time. And because the harness
reports a result by observing whether the function returned or panicked, the
entire assertion vocabulary reduces to “panic on failure.” Once you internalize
“failing == panicking,” the rest follows: #[should_panic] simply inverts that
rule, and a Result-returning test lets an Err stand in for a panic.
The one thing that trips everyone up — why integration tests and doctests need a
library — also falls out of the build model. A tests/foo.rs file compiles
as its own crate that links your code as a dependency. A dependency only
exposes its pub surface. Binaries (src/main.rs, src/bin/*.rs) have no
linkable public API, so there’s nothing for an external test crate to call. Hence
this ladder spins up practice/testing_lab/ (a real src/lib.rs) the moment it
needs tests/ and doctests.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | foundations | First test | #[cfg(test)] mod tests, #[test], assert_eq!, use super::* |
| 2 | foundations | Assertion toolbox | assert! / assert_eq! / assert_ne! + custom messages |
| 3 | mechanics | Result-returning test | fn() -> Result<(), E> lets you use ? instead of .unwrap() |
| 4 | mechanics | #[should_panic] | assert a panic happens; tighten with expected = "..." |
| 5 | footgun | Execution control | #[ignore], name filter, --nocapture, parallelism |
| 6 | footgun | Assertion footguns | loose should_panic, float equality, private reach |
| 7 | real-world | Integration tests | tests/ = separate crate, public API only, common/mod.rs |
| 8 | real-world | Doctests | runnable /// examples, hidden # lines, ?, fences |
| 9 | capstone | A Ledger tested every way | unit + integration + doctests in one suite |
The ideas, built up
1. A test is a #[test] fn in a #[cfg(test)] module
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn add_works() {
assert_eq!(add(2, 3), 5);
}
}
Two attributes carry the whole foundation:
#[cfg(test)]is conditional compilation. The module exists only when building undercargo test; acargo build --releasecompiles it away to nothing. That’s why you can write as many tests as you like with zero binary cost.#[test]marks a function the harness should collect and run. It must take no arguments and return()(orResult, see rung 3).
use super::* matters because mod tests is a child module. Child modules
don’t automatically see their parent’s items, so you import them. And because the
test module is a child of the very module it tests, it can see private items —
the defining privilege of unit tests, which rung 6 and 7 turn into a contrast.
2. The assertion toolbox, and why assert_eq! beats assert!
assert!(classify(0) == "zero"); // boolean: prints only "false"
assert_eq!(classify(-4), "negative"); // prints left AND right on failure
assert_ne!(classify(7), "zero");
assert_eq!(classify(2), "positive", "ctx: {}", 2); // trailing custom message
All three panic on failure. The difference is the failure message:
// assert!(a == b) on failure:
assertion failed: classify(0) == "ZERO"
// assert_eq!(a, b) on failure:
assertion `left == right` failed
left: "zero"
right: "ZERO"
assert_eq! captures both operands and prints them, so a red test tells you what
the value actually was, not merely that a boolean came out false. Reach for
assert_eq!/assert_ne! whenever you’re comparing two values; save bare
assert! for genuine booleans. Any of them takes a trailing format string for
extra context.
3. Result-returning tests and ?
A #[test] fn may return Result<(), E>. Ok(()) passes; Err(e) fails and
prints e. That unlocks ? in test bodies:
#[test]
fn parse_pair_ok() -> Result<(), Box<dyn std::error::Error>> {
let (a, b) = parse_pair("3,4")?; // ? compiles ONLY because of the return type
assert_eq!((a, b), (3, 4));
Ok(())
}
The Box<dyn std::error::Error> is the catch-all error type any ?-converted
error can flow into. Why prefer this over .unwrap()? Both fail the test on an
unexpected error, but the intent differs: ? says “an error here is a test
failure, surface it,” while .unwrap() says “this can’t happen, panic if it
does.” When the error is part of the path you’re exercising, ? reads better and
keeps the happy path uncluttered.
4. #[should_panic] — inverting the pass/fail rule
Sometimes the correct behavior is to panic, and you want to assert it does.
#[should_panic] inverts the harness rule: the test passes iff the body
panics.
#[should_panic] // passes on ANY panic
#[test]
fn seat_too_high() { seat(99); }
#[should_panic(expected = "seat 99 out of range")] // passes only on THIS panic
#[test]
fn seat_too_high_expected() { seat(99); }
The runner even labels them: test tests::seat_too_high - should panic ... ok.
The bare form is dangerously loose — it can’t tell a correct panic from an
unrelated one (rung 6 weaponizes this). expected = "..." requires the panic
message to contain that substring, pinning the test to the panic you actually
mean. Always add expected.
5. Driving the runner
Knobs after a -- go to the test harness, not to cargo:
cargo test --bin testing # everything (ignored ones skipped)
cargo test --bin testing classify # only tests whose name contains "classify"
cargo test --bin testing -- --ignored # run ONLY the #[ignore] tests
cargo test --bin testing noisy -- --nocapture # let println! through
cargo test --bin testing -- --test-threads=1 # run serially, not in parallel
#[ignore]marks a test skipped by default (slow/flaky/manual). It shows asN ignoredand runs only with-- --ignored.- Captured output:
cargo testswallows the stdout of passing tests (it only surfaces output on failure, to keep noise down).-- --nocaptureletsprintln!through. - Name filter is a substring match on the full test path.
6. Tests run in parallel — the footgun that defines this rung
By default the harness runs tests on multiple threads at once. Independent tests get faster; tests that share mutable state (a global, a file, an env var, the current working dir) can interleave and clobber each other — green alone, flaky in the suite. Two fixes:
- Isolate — give each test its own state. The real fix.
- Serialize —
-- --test-threads=1. A crutch that hides the coupling.
The add / classify / seat tests share zero mutable state, which is precisely
why they’re safe in parallel. Keep tests that way by construction.
Footguns
The whole footgun tier (rung 6) is a catalogue of “passing” tests that lie.
Loose #[should_panic] passes for the wrong reason. A panic anywhere in the
body satisfies a bare #[should_panic]:
// LIE: the unwrap panics FIRST, so seat(0) is never reached, yet the test is green
#[should_panic]
#[test]
fn footgun_loose() {
let _bogus: u32 = "not a number".parse::<u32>().unwrap(); // panics HERE
seat(0); // never runs
}
// HONEST: pin the message, and make the line under test the one that panics
#[should_panic(expected = "seat 0 out of range")]
#[test]
fn footgun_fixed() { seat(0); }
Adding expected fails the lying version (the parse panic doesn’t contain “seat
0 out of range”), exposing the bug. The same trap reappears in doctests: a
should_panic doctest still containing a todo!() will pass, because todo!()
panics.
Never assert_eq! raw floats. IEEE-754 makes 0.1 + 0.2 != 0.3:
// WRONG: fails with right: 0.30000000000000004
assert_eq!(0.1_f64 + 0.2, 0.3);
// OK: compare within a tolerance
assert!((0.1_f64 + 0.2 - 0.3).abs() < 1e-9);
Private reach is a unit-test-only superpower. A unit test can call a private
fn secret_key() -> u32 because it lives inside the crate. Hold that thought —
the next rung shows an integration test getting a compile error for the same
call.
Real-world patterns
Integration tests: tests/ is a separate crate
The testing_lab library exposes pub fn normalize backed by a private
squeeze_spaces. An integration test links the crate from outside:
// practice/testing_lab/tests/normalize.rs — its own crate, black-box
mod common;
#[test]
fn normalizes_messy_input() {
assert_eq!(testing_lab::normalize(" Hello WORLD "), "hello world");
}
Try to reach a private helper and the compiler stops you — this is the wall that defines black-box testing:
error[E0603]: function `squeeze_spaces` is private
--> tests/_probe.rs:3:18
|
3 | testing_lab::squeeze_spaces("a b");
| ^^^^^^^^^^^^^^ private function
A unit test reaches privates; an integration test sees only pub. Different
scope, different job.
Shared helpers go in tests/common/mod.rs, not tests/common.rs. Every file
directly under tests/ compiles as its own test binary — so tests/common.rs
would run as a confusing empty test target. A file in a subdirectory
(tests/common/mod.rs) is a plain module other test files pull in with
mod common;, and is not itself a test target:
// tests/common/mod.rs
use testing_lab::normalize;
pub fn assert_normalizes(input: &str, expected: &str) {
assert_eq!(normalize(input), expected, "input: {input:?}");
}
Doctests: examples that can’t rot
A fenced code block in a /// comment is compiled and run by cargo test. If the
example stops matching the API, the doctest fails — your docs can never silently
drift:
/// ```
/// use testing_lab::normalize;
/// # let _unused = 1; // hidden `#` line: runs, not rendered
/// assert_eq!(normalize(" Hi THERE "), "hi there");
/// ```
pub fn normalize(s: &str) -> String { /* ... */ }
Doctest mechanics worth knowing:
-
Hidden lines. A line starting with
#runs but is omitted from the rendered docs — perfect for boilerplate setup that would distract the reader. -
?needs aResult-returning example. rustdoc wraps your snippet in afn main() -> (), so?won’t compile until the example returns aResult. End it with a hidden# Ok::<(), SomeError>(())and rustdoc switches the wrapper’s return type to match:/// let p = parse_port("8080")?; /// assert_eq!(p, 8080); /// # Ok::<(), std::num::ParseIntError>(()) -
Fence attributes change what “pass” means:
```should_panic— passes only if the example panics.```compile_fail— passes only if the example fails to compile (great for proving an API rejects misuse, e.g.nth_word(42, 0)where a&stris required).```no_run— compiles but doesn’t execute (network/side-effects).```ignore— skip entirely.
Capstone insight
Rung 9 tests one small library — a money Ledger (balance in integer cents, a
private entries counter, and a private check_invariant) — every way at once,
and the structural lesson is how the techniques partition by scope:
cargo test -p testing_lab
unittests src/lib.rs ... white-box: assert on private `entries`, call check_invariant()
tests/ledger.rs ... black-box: public deposit/withdraw + custom assert_balance helper
Doc-tests testing_lab... runnable docs: a basic one, a should_panic, a `?`-using one
- A unit test inside
mod ledger::testsassertsledger.entries == 3— a field an integration test literally cannot name. - An integration test in
tests/ledger.rsdrives onlydeposit/withdrawand checks results through a sharedcommon::assert_balance, exactly as a downstream user would. - A doctest on every public method doubles as documentation and a test,
including
deposit’sshould_panic(negative amount) andwithdraw’s?-using example.
The “aha”: these aren’t three competing styles, they’re three altitudes. Unit
tests verify internal invariants you can only see from inside; integration tests
verify the contract you ship; doctests verify the contract as documented. A
mature crate runs all three from one cargo test, and the harness reports each as
its own section.
Explain it back
- Why does a test “fail”? What single mechanism underlies
assert_eq!,#[should_panic], and aResult-returning test? - What does
#[cfg(test)]cost a release build, and why? - Why can a unit test call a private function but an integration test gets E0603 for the same call?
- Why do integration tests and doctests require a library crate, but unit tests don’t?
- Your
#[should_panic]test is green. Name two ways it could be lying, and the fix for each. - Why does
assert_eq!(0.1 + 0.2, 0.3)fail, and what do you write instead? - Why is
tests/common/mod.rscorrect for shared helpers buttests/common.rswrong? - A doctest uses
?and won’t compile. What’s missing?
See also
- Modules & visibility —
pub, privacy, and the crate boundary that decides what an integration test can see. - Cargo features &
cfg—#[cfg(test)]is the same conditional-compilation machinery as#[cfg(feature = "...")]. - Error handling architecture —
Box<dyn Error>as the catch-all for?in tests.
Cow — Clone-on-Write
Ladder:
src/bin/cow.rs· Run:cargo run --bin cow· Phase 1 · 9 rungs
TL;DR
Cow<'a, B> (“clone on write”) is an enum with two states — Borrowed(&'a B)
or Owned(B::Owned) — that lets one value be either a cheap borrow or a
heap-owned thing, behind a single type. You hand back Borrowed when the data is
already fine, and pay for an Owned allocation only when you actually have to
change something. It Derefs to B, so callers use it like a plain &str /
&[T] and never care which variant it’s holding.
Mental model:
Cowis a maybe-allocation. “Here’s your string back. I only made a new one if I had to.”
Why this exists (from first principles)
Imagine a function ensure_https(url) -> ???. Most URLs already start with
https:// — for those you’d love to just return the input untouched. But some
don’t, and for those you must build a new string "https://" + url.
Now: what’s the return type?
- Return
&str(a borrow)? Impossible for the fix-up case — the new string is a local; you can’t return a reference to data that dies at function end. - Return
String(owned)? Works, but forces an allocation + copy on every call, even for the 90% of inputs that were already correct. Wasteful.
You’re stuck because the two cases want different types. Cow is the type that
says “either of those, decided at runtime”:
fn ensure_https(url: &str) -> Cow<'_, str> {
if url.starts_with("https://") {
Cow::Borrowed(url) // zero cost
} else {
Cow::Owned(format!("https://{}", url)) // allocate only here
}
}
That’s the whole reason Cow exists: a function that usually borrows but
sometimes must own, without committing every caller to the cost of owning.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | foundations | ensure_https | Borrow when correct, own only when you must build. |
| 2 | foundations | sanitize (spaces to _) | Decide before allocating — .replace() always allocates. |
| 3 | mechanics | Cow as a struct field | One Config<'a> type holds a literal or a runtime string. |
| 4 | mechanics | clamp_negatives + .to_mut() | The actual clone-on-write: upgrade on first mutation. |
| 5 | footgun | greeting | You can only borrow inputs, never locals. |
| 6 | mechanics | first_word via Deref | A Cow<str> is usable as a &str — no variant matching. |
| 7 | real-world | normalize batch | Borrow the clean entries, allocate only the dirty ones. |
| 8 | real-world | serde #[serde(borrow)] | Zero-copy deserialize; own only when decoding escapes. |
| 9 | capstone | hand-rolled MyCow | Build the borrow/own/upgrade machine yourself. |
The ideas, built up
The defining discipline: inspect before you allocate
The naive way to replace spaces is input.replace(' ', "_") — but replace
always returns a fresh String, even if there were no spaces to replace. That
throws away the entire point of Cow. The fix is to check first:
fn sanitize(input: &str) -> Cow<'_, str> {
if input.contains(' ') {
Cow::Owned(input.replace(' ', "_")) // allocate only on the dirty path
} else {
Cow::Borrowed(input) // clean input: zero allocation
}
}
The pattern that repeats all ladder long: ask “is any work actually needed?” before you reach for an allocation.
Cowonly pays off if the borrowed path is genuinely free.
Cow as a field: one type, two origins
Cow isn’t just a return type — as a struct field it lets one type absorb
both a borrowed literal and an owned runtime value:
struct Config<'a> { name: Cow<'a, str> }
Config { name: Cow::Borrowed("default") } // from a &'static literal — no alloc
Config { name: Cow::Owned(format!("user-{id}")) } // from a runtime String
Both are the same Config<'a> type, and name(&self) -> &str reads either one
uniformly (via .as_ref()). The lifetime 'a is the price: the struct can’t
outlive whatever the borrowed variant points at.
The heart: .to_mut() and lazy upgrade
This is where “clone-on-write” earns its name. cow.to_mut() returns a
&mut to the owned data — and here’s the mechanism:
- If the cow is
Ownedalready: hands back the ref, no clone. - If it’s
Borrowed: clones intoOwnedfirst, swaps itself, then gives you the mutable ref.
So you call to_mut() exactly at the moment you first need to mutate, and the
allocation happens then — and only then:
fn clamp_negatives(input: &[i32]) -> Cow<'_, [i32]> {
let mut cow: Cow<[i32]> = Cow::Borrowed(input); // start free
for i in 0..input.len() {
if input[i] < 0 {
cow.to_mut()[i] = 0; // first negative upgrades Borrowed -> Owned
}
}
cow // all-positive input is returned still-Borrowed
}
An all-positive slice never calls to_mut(), so it’s returned borrowed for free.
One negative anywhere triggers a single clone, and every later write reuses it.
Note this also shows Cow is not string-only — here B = [i32], owned
form Vec<i32>. Anything that is ToOwned works.
Ergonomics: Deref makes a Cow act like its target
Cow<str> implements Deref<Target = str>, so every &str method works on it
directly — no match, no .as_ref(), no caring about the variant:
fn first_word(c: &Cow<str>) -> &str {
c.split_whitespace().next().unwrap_or("") // str methods, called straight on the Cow
}
c.len(), c.starts_with(..), &**c == "hello" — all Just Work. This is why
Cow is pleasant to consume, not just to produce.
Footguns
You cannot borrow a local (rung 5)
This is the defining Cow compile error. This does not compile:
fn broken(name: &str) -> Cow<'_, str> {
let local = format!("hi {name}");
Cow::Borrowed(&local) // WRONG: cannot return value referencing local variable
}
Cow::Borrowed ties its lifetime to data that must outlive the call. A
String built inside the function dies at the closing brace, so you literally
cannot hand it back borrowed. The correct version owns what it builds:
fn greeting(name: &str) -> Cow<'_, str> {
if name == "hi there" {
Cow::Borrowed(name) // OK: borrowing an INPUT is fine
} else {
Cow::Owned(format!("hi {}", name)) // OK: built locally -> must be Owned
}
}
Rule:
Borrowed= “I’m pointing at someone else’s data that lives long enough” (inputs,'staticliterals).Owned= “I made this myself.” You can never borrow a local.
.replace() / .to_lowercase() always allocate
These produce a fresh String unconditionally. If you call them on the borrowed
path “just in case”, you’ve silently defeated Cow. Gate them behind a
.contains(..) / .any(..) check (rungs 2 and 7).
Signatures to know
// The enum itself — B is the borrowed form, B::Owned is the owned form
enum Cow<'a, B: ?Sized + ToOwned> {
Borrowed(&'a B),
Owned(<B as ToOwned>::Owned),
}
// Upgrade: clone into Owned on first write, then hand back &mut
fn to_mut(&mut self) -> &mut <B as ToOwned>::Owned
// Consume the Cow, producing an owned value either way
fn into_owned(self) -> <B as ToOwned>::Owned
// Transparent access: Cow<str> derefs to &str
impl<B: ?Sized + ToOwned> Deref for Cow<'_, B> {
type Target = B;
}
Real-world patterns
Borrow most, own a few (rung 7)
Normalize a batch of words to lowercase, allocating only for the ones that actually had uppercase:
fn normalize<'a>(words: &'a [&'a str]) -> Vec<Cow<'a, str>> {
words.iter().map(|w| {
if w.chars().any(|c| c.is_uppercase()) {
Cow::Owned(w.to_lowercase()) // dirty -> allocate
} else {
Cow::Borrowed(*w) // already clean -> free
}
}).collect()
}
A mostly-clean batch costs almost nothing — each clean word still points into the original input.
Zero-copy deserialization with serde (rung 8)
This is the marquee payoff. Give a serde struct a Cow<'a, str> field and tag it
#[serde(borrow)]:
#[derive(Deserialize)]
struct Msg<'a> {
#[serde(borrow)]
text: Cow<'a, str>,
}
Now when you deserialize:
{"text":"hello world"}—Borrowed, pointing straight into the JSON input buffer. Zero copy.{"text":"line1\nline2"}— the\nescape must be decoded, so serde has no choice but to build a freshString—Owned.
One field, both outcomes, decided by the data. Drop the #[serde(borrow)] and
serde defaults to always Owned — watch the borrowed assertion fail. That
contrast is the lesson.
Capstone insight
Re-implementing MyCow from scratch makes the whole thing click. It’s just two
variants plus three methods — and to_mut is the only interesting one, because it
performs the in-place state transition from borrowed to owned:
fn to_mut(&mut self) -> &mut String {
match self {
Self::Borrowed(s) => {
*self = Self::Owned(s.to_string()); // clone + replace SELF
match self { // now re-match to hand out the owned ref
Self::Owned(s) => s,
_ => unreachable!(),
}
}
Self::Owned(s) => s, // already owned: no clone
}
}
That *self = ...; re-match dance is exactly how std does it. Once you’ve written
it, “clone on write” stops being a slogan and becomes a concrete line of code: the
borrow becomes an allocation right here, and nowhere else.
Explain it back
- Why can’t
ensure_httpsjust return&str? Why not justString? - What does
.to_mut()do differently for aBorrowedvs anOwnedcow? - Why does
Cow::Borrowed(&local)fail to compile, butCow::Borrowed(input)is fine? - What makes
c.split_whitespace()work directly on aCow<str>? - In the serde rung, why does
"line1\nline2"come backOwnedbut"hello world"comes backBorrowed? Cow<'a, B>requiresB: ToOwned— why? (What couldn’t it do without it?)
See also
- Borrow / ToOwned — the two traits
Cowis built on;B: ToOwnedis what lets theOwnedvariant exist, and rung 8 there closes this exact loop. - Drop & Ordering —
mem::replace(used byto_mutinternally) is covered in depth there.
Box & the Heap
Ladder:
src/bin/box_heap.rs· Run:cargo run --bin box_heap· Phase 1 · 9 rungs
TL;DR
Box<T> is Rust’s simplest heap allocator: a single owning pointer to a
value on the heap. It is exactly one pointer wide, implements Deref<Target = T>
so you use the inner value transparently, and drops the heap allocation when it
goes out of scope. Box exists because some values cannot live on the stack —
recursive types have infinite size without indirection, and trait objects need a
uniform container for heterogeneous values. When you see Box, think: “owned
heap value, pointer-sized, cleaned up automatically.”
Mental model:
Box<T>is aTthat lives on the heap instead of the stack. You reach through it with*, but auto-deref means you rarely have to.
Why this exists (from first principles)
Every local variable in Rust lives on the stack, and the compiler must know its size at compile time. This creates two hard problems:
Problem 1 — Recursive types. An enum List { Cons(i32, List), Nil } is
infinitely sized: a List contains a List contains a List… The compiler
needs a finite size_of::<List>() and can’t get one. Wrapping the recursion in
Box<List> breaks the cycle — a Box is always one pointer wide, regardless of
what it points to.
Problem 2 — Heterogeneous collections. A Vec<Shape> doesn’t work when
Shape is a trait — different implementors have different sizes. But
Vec<Box<dyn Shape>> does: every element is the same size (a fat pointer), and
the actual data lives on the heap at whatever size it needs.
Problem 3 — Large values. Moving a 10 MB array into a function copies 10 MB
on the stack. Box::new(big_array) puts it on the heap; moving the Box copies
just one pointer.
Box is the answer to “I need a value whose size isn’t known at compile time”
or “I need a value that outlives this stack frame’s layout constraints.”
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | foundations | boxed_sum | Box is an owning, pointer-sized handle; dereference with *. |
| 2 | foundations | drop_order | Heap value drops when its Box scope ends, in reverse declaration order. |
| 3 | mechanics | cons list | Recursive types require Box to get a finite size (E0072). |
| 4 | mechanics | greet_len / rename / unbox | Deref/DerefMut + moving the owned value out of a Box. |
| 5 | footgun | Expr tree | The infinite-size error — read it, understand it, fix it with Box. |
| 6 | edge cases | into_parts / steal_items | Moving out of an owned Box vs. mem::take through &mut Box. |
| 7 | real-world | Box<dyn Shape> | Heterogeneous trait objects; fat pointer = data ptr + vtable ptr. |
| 8 | real-world | Box<dyn Error> + Box::leak | Dynamic error unification; intentional leak for 'static refs. |
| 9 | capstone | generic LinkedList<T> | Owning pointer chain from scratch: push/pop/len/iter + iterative Drop. |
The ideas, built up
Box basics: allocate, deref, pointer-sized
A Box<T> allocates T on the heap and gives you an owning pointer. You read
the heap value by dereferencing:
fn boxed_sum(b: Box<i64>, n: i64) -> i64 {
*b + n // explicit deref — but arithmetic would auto-deref too
}
The critical size fact: a Box<T> is always exactly size_of::<usize>()
bytes — one pointer — regardless of how large T is. A Box<[u8; 1024]> is
8 bytes on a 64-bit machine, same as Box<i64>. The heap holds the data; the
stack holds just the address.
assert_eq!(size_of::<Box<i64>>(), size_of::<usize>());
assert_eq!(size_of::<Box<[u8; 1024]>>(), size_of::<usize>());
Drop timing: Box follows stack rules
When a Box goes out of scope, it drops the heap value — exactly like a stack
value’s destructor runs at scope end. And when multiple Boxes are declared in
sequence, they drop in reverse declaration order (LIFO), because the Box
itself is on the stack and stack drops are LIFO:
let a = make("a"); // declared first
let b = make("b"); // declared second
drop(b); // force "b" to drop first
drop(a); // then "a"
// log == ["b", "a"]
Without explicit drop() calls, locals drop in reverse order at scope end
anyway. The Noisy struct with a shared log (via Rc<RefCell<Vec<String>>>)
proves this observationally — the drop impl pushes the label, and you assert the
log matches. The takeaway: heap ownership follows stack scoping rules; the Box
is on the stack, so it obeys stack destruction order.
Recursive types: why Box is required
This is the poster-child use case. A cons list:
// WRONG — infinite size
enum List { Cons(i32, List), Nil }
// OK — Box breaks the recursion
enum List { Cons(i32, Box<List>), Nil }
Without Box, the compiler tries to compute size_of::<List>() and finds that
Cons contains a List which contains a Cons which contains a List…
It’s turtles all the way down — error E0072, “recursive type has infinite size.”
Box<List> is the fix because a Box is one pointer regardless of its target.
Now size_of::<List>() = max(size_of::<i32>() + size_of::<Box<List>>(), 0) + tag,
which is finite and known. The recursion still exists at runtime — the heap chain
can be arbitrarily long — but each type-level nesting is just “a pointer to
more of the same.”
Summing the list is straightforward pattern-matching with recursion:
fn sum_list(list: &List) -> i32 {
match list {
List::Nil => 0,
List::Cons(v, rest) => v + sum_list(rest),
}
}
Auto-deref means rest (a &Box<List>) coerces to &List when passed
recursively — no explicit * needed.
Deref, DerefMut, and moving out
Box<T> implements Deref<Target = T> and DerefMut, which gives three
capabilities:
Auto-deref lets you access fields directly:
fn greet_len(p: &Box<Person>) -> usize {
p.name.len() // no *, no (**p).name — auto-deref handles it
}
DerefMut lets you mutate through the box:
fn rename(mut p: Box<Person>, new_name: &str) -> Box<Person> {
p.name = new_name.to_string(); // writes straight through
p
}
*box on an owned Box moves the value out of the heap:
fn unbox(p: Box<Person>) -> Person {
*p // consumes the Box, returns the owned Person
}
This last one is unique to Box — you can’t * a Rc or Arc to move out,
because they might have other owners. A Box is the sole owner, so moving
out is always safe. The box is consumed, the heap is freed, and you hold the
value on the stack.
The infinite-size error: expression trees
The cons list had one recursive field. An expression tree has two:
// WRONG — Add(Expr, Expr) is doubly-infinite
enum Expr { Num(i64), Add(Expr, Expr), Mul(Expr, Expr) }
// OK — box each child
enum Expr {
Num(i64),
Add(Box<Expr>, Box<Expr>),
Mul(Box<Expr>, Box<Expr>),
}
The error message (E0072) tells you exactly which type is recursive and
suggests Box. The fix is mechanical: every recursive field gets wrapped.
Evaluating the tree is the natural recursive pattern:
fn eval(e: &Expr) -> i64 {
match e {
Expr::Num(n) => *n,
Expr::Add(a, b) => eval(a) + eval(b),
Expr::Mul(a, b) => eval(a) * eval(b),
}
}
Why Box and not &Expr? A reference would need a lifetime and wouldn’t own
the children. The tree owns its nodes — Box is the ownership-preserving
indirection.
Moving out of a Box and partial moves
When you own a Box<T>, you can destructure the contents by moving them out:
fn into_parts(b: Box<Config>) -> (String, Vec<String>) {
let Config { name, items } = *b; // move all fields out, box freed
(name, items)
}
But when you only have &mut Box<Config>, you cannot move a field out —
that would leave a hole in a value someone else still holds a reference to:
fn steal_items(b: &mut Box<Config>) -> Vec<String> {
b.items // WRONG: "cannot move out of `b.items` which is behind a mutable reference"
}
The fix is std::mem::take, which swaps in a default value as it takes the
old one out — no hole left behind:
fn steal_items(b: &mut Box<Config>) -> Vec<String> {
std::mem::take(&mut b.items) // items becomes Vec::new(), config stays valid
}
This is the same mem::take / mem::replace technique from the Drop & ordering ladder. It shows up constantly with linked structures — any time you
need to rewire a pointer through &mut, you swap in a placeholder.
Box<dyn Trait>: heterogeneous collections
A Vec<Circle> can only hold circles. A Vec<Box<dyn Shape>> can hold
anything that implements Shape — circles, squares, triangles, side by side:
trait Shape {
fn area(&self) -> f64;
fn name(&self) -> &'static str;
}
let shapes: Vec<Box<dyn Shape>> = vec![
Box::new(Circle { radius: 1.0 }),
Box::new(Square { side: 2.0 }),
];
Each element is a fat pointer: one word for the data on the heap, one word
for the vtable (the table of function pointers for that concrete type’s Shape
impl). This is dynamic dispatch — s.area() looks up the right function
pointer at runtime.
The size difference tells the story:
assert_eq!(size_of::<Box<dyn Shape>>(), 2 * size_of::<usize>()); // fat: data + vtable
assert_eq!(size_of::<Box<Circle>>(), size_of::<usize>()); // thin: data only
A concrete Box<Circle> is one pointer — the compiler knows the type statically.
A Box<dyn Shape> needs the extra vtable pointer because the type is erased.
Summing areas just iterates and calls through the vtable:
fn total_area(shapes: &[Box<dyn Shape>]) -> f64 {
shapes.iter().map(|s| s.area()).sum()
}
Auto-deref means s.area() works directly on &Box<dyn Shape> — you don’t
need (**s).area().
Box<dyn Error> and Box::leak
Box<dyn Error> unifies error types. When a function can fail in multiple
unrelated ways (parse error, IO error, custom validation), you need a single
return type. Box<dyn std::error::Error> is the catch-all: any type implementing
Error converts into it via ?, and you can build one from a plain string with
.into():
fn parse_and_double(s: &str) -> Result<i32, Box<dyn Error>> {
let n = s.parse::<i32>()?; // ParseIntError -> Box<dyn Error>
if n < 0 {
Err(Box::new(std::io::Error::new(
std::io::ErrorKind::Other,
"negative number", // custom error from string
)))
} else {
Ok(n * 2)
}
}
Box::leak gives you 'static from runtime data. It consumes a Box<T>
and returns &'static mut T by deliberately never freeing the memory — an
intentional leak. The use case is turning runtime-built data into a reference
that lives forever (global config, interned strings, etc.):
fn leak_static(s: String) -> &'static str {
Box::leak(s.into_boxed_str())
}
String::into_boxed_str() gives a Box<str>, and Box::leak on it yields
&'static mut str (which coerces to &'static str). The heap allocation will
never be freed — that’s the trade. Use it for data that genuinely must live for
the whole program, not as a lifetime escape hatch.
Footguns
-
Infinite-size enum (E0072). Every recursive field in an enum needs
Box(or some other indirection). Two recursive fields means twoBoxes —Veccan’t save you when each node has a fixed number of children. -
Moving out through
&mut. You can destructure an ownedBox<T>, but through a mutable reference you can’t leave a hole. Reach formem::takeormem::replaceto swap in a default. -
Recursive Drop on long chains. A linked list where each node’s
Dropcalls the next node’sDropwill blow the stack on long chains. You must write an iterativeDropthat walks and frees one node at a time. -
Box::leakis permanent. The memory is never freed. It’s correct for process-lifetime data, but a leak in a loop is a real memory leak.
Signatures to know
// Allocate on the heap
fn Box::new(x: T) -> Box<T>
// Transparent access — Box<T> acts like T
impl<T: ?Sized> Deref for Box<T> {
type Target = T;
}
impl<T: ?Sized> DerefMut for Box<T> { ... }
// Move the value out of the heap (consumes the Box)
let val: T = *boxed;
// Intentional leak -> 'static reference
fn Box::leak<'a>(b: Box<T>) -> &'a mut T
// String -> Box<str> (for leak or other unsized boxing)
fn String::into_boxed_str(self) -> Box<str>
Real-world patterns
Trait objects for plugin/strategy patterns
Box<dyn Trait> is how Rust does runtime polymorphism. Any time you’d reach for
an interface/abstract class in another language — a logging backend, a storage
driver, a rendering strategy — you store a Box<dyn Backend> and dispatch
through the vtable.
Error propagation with ?
In application code (not libraries), Result<T, Box<dyn Error>> is the quick
path when you don’t want to define a custom error enum. Every ? auto-converts
the concrete error into the box. For libraries, prefer thiserror for typed
errors; for applications, anyhow wraps this pattern with better ergonomics.
Recursive data structures
Trees, expression ASTs, and linked lists all use Box (or Rc/Arc for
shared ownership) to break the infinite-size cycle. The pattern is always the
same: Option<Box<Node<T>>> for an optional link.
Capstone insight
Building a generic LinkedList<T> from scratch makes every Box idea concrete.
The link type is Option<Box<Node<T>>>: Some(box) is a node, None is the
end. Every operation revolves around Option::take() — the same mem::take
trick from rung 6, applied to rewire links through &mut self:
fn push(&mut self, elem: T) {
let old_head = self.head.take(); // take the old head out (leaves None)
let new_node = Box::new(Node {
elem,
next: old_head, // old head becomes next
});
self.head = Some(new_node); // new node becomes head
}
pop is the mirror: take the head, set head to node.next, return node.elem.
The iterative Drop is the non-obvious piece. Without it, dropping a 100,000-
node list recurses 100,000 deep and overflows the stack — each Box<Node>
drops its next field, which drops its next, and so on. The fix is a while let loop that takes each link one at a time:
impl<T> Drop for LinkedList<T> {
fn drop(&mut self) {
let mut current = self.head.take();
while let Some(mut node) = current {
current = node.next.take(); // free one node per iteration
}
}
}
Each iteration drops one Box<Node> (the Some(mut node) binding owns it and
drops it at the end of the loop body), but crucially that node’s next has
already been take()-ed out to None, so no recursive drop fires. The chain
unwinds iteratively, not recursively.
The iterator is the final piece: an Iter<'a, T> that holds
Option<&'a Node<T>> and advances by following node.next.as_deref(). It
borrows the list without consuming it, yielding &T references:
impl<'a, T> Iterator for Iter<'a, T> {
type Item = &'a T;
fn next(&mut self) -> Option<Self::Item> {
self.next.take().map(|node| {
self.next = node.next.as_deref();
&node.elem
})
}
}
Once you’ve built this, Box stops being “the heap pointer” and becomes a
concrete ownership link — a chain of sole-owner pointers you can rewire,
iterate, and drop by hand.
Explain it back
- Why is
Box<T>exactly oneusizewide, no matter how bigTis? - What error do you get if you write
enum List { Cons(i32, List), Nil }, and why doesBox<List>fix it? - How does
*boxed_valuediffer from dereferencing an&T? (Hint: ownership.) - Why can you destructure
*owned_boxbut notborrowed_box.fieldthrough&mut? - What is the size of
Box<dyn Shape>vsBox<Circle>, and why? - Why does the linked list need an iterative
Drop, and what would happen without one on 100,000 nodes? - What does
Box::leakactually do, and when is it appropriate to use?
See also
- Cow — uses
Boxinternally for theOwnedvariant’s heap allocation;into_boxed_str()appears in the leak rung. - Drop & Ordering —
mem::takeandmem::replace, the tools that make Box-based linked structures work through&mut. - Rc<RefCell<T>> patterns — shared ownership via
RcwhereBox’s sole ownership isn’t enough; the doubly-linked list there contrasts with this singly-linked capstone.
Cell & RefCell — Interior Mutability
Ladder:
src/bin/cell_refcell.rs· Run:cargo run --bin cell_refcell· Phase 1 · 9 rungs
TL;DR
Rust enforces many &T XOR one &mut T at compile time. Interior mutability
lets you mutate through a shared &T by upholding that same rule a different
way. Cell<T> never hands out references at all — it copies values in and out,
so no aliasing can occur. RefCell<T> hands out real &T / &mut T, but checks
the borrow rule at runtime (and panics if you break it). Both are !Sync —
single-threaded only; the multi-threaded counterparts are Mutex and RwLock.
Mental model:
Cellis a slot you can only peek at or swap.RefCellis a slot with a borrow-checker bouncer who works the night shift (runtime) instead of the day shift (compile time).
Why this exists (from first principles)
The borrow checker is conservative. It enforces “many readers XOR one writer” at
compile time by tracking & and &mut through the type system. This is sound
and zero-cost — but it rejects programs that are actually safe:
struct Stats { count: u32 }
impl Stats {
fn record(&mut self) { self.count += 1; }
// ^^^^^^^^^ requires exclusive access
}
If two parts of your program hold &Stats, neither can call record — the
compiler can’t prove they won’t alias. But you know you’re single-threaded and
the mutation is fine. The compiler won’t budge.
Interior mutability is the escape hatch: wrap the field in Cell or RefCell,
and the type itself enforces the aliasing rule (by copying or by runtime
checks), so the compiler can accept &self methods that mutate.
Without Cell/RefCell, you’d need &mut all the way up the call chain for
any mutation — which is often impossible when multiple owners (Rc) or callbacks
need to write.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | foundations | bump via Cell | Mutate a Copy value through & with get/set. |
| 2 | foundations | log via RefCell | borrow_mut() to push into a Vec through &. |
| 3 | mechanics | Cell toolbox | replace, take, update, into_inner; Cell<Option<T>> for non-Copy. |
| 4 | mechanics | RefCell toolbox | &self methods that mutate; many coexisting borrows; try_borrow. |
| 5 | footgun | borrow panic | Overlap borrow_mut with borrow – runtime panic. Fix by scoping. |
| 6 | footgun | !Sync + re-entrancy | RefCell can’t cross threads; callback that re-borrows panics. |
| 7 | real-world | Rc<RefCell<Node>> | Shared mutable tree; mutate through one handle, see it through another. |
| 8 | real-world | Ref::map projection | Borrow a single field out of a RefCell without losing the guard. |
| 9 | capstone | MyRefCell from scratch | UnsafeCell + borrow flag + RAII guards. |
The ideas, built up
Cell: mutate by copying, never by reference
Cell<T> provides interior mutability for Copy types with zero runtime
overhead. The API is deliberately narrow — you can get() a copy of the value
and set() a new one, but you never get a reference to the contents:
fn bump(counter: &Cell<u32>, by: u32) {
counter.set(counter.get() + by);
}
The signature is &Cell<u32>, not &mut Cell<u32> — two shared references can
both drive mutations because no aliasing reference to the inner u32 ever
exists. The value is copied out, modified, and copied back in. This is why get
requires T: Copy — it can’t hand you a reference (that would create aliasing),
so it must copy.
let counter = Cell::new(0u32);
let r1 = &counter;
let r2 = &counter;
bump(r1, 5);
bump(r2, 3); // both shared refs can mutate — no &mut anywhere
assert_eq!(counter.get(), 8);
The Cell toolbox: replace, take, update
get/set handle Copy types, but what about a String in a Cell? You
can’t copy it out. The toolbox fills the gap with swapping operations:
| Method | What it does | Requires |
|---|---|---|
replace(new) -> old | Store new, return the previous value | nothing |
take() -> T | Store T::default(), return the previous value | T: Default |
update(f) | set(f(get())) — read-modify-write in one shot | T: Copy |
into_inner() -> T | Consume the Cell, extract the value | ownership |
The classic trick for non-Copy types: Cell<Option<T>>. You can take() the
Option, which replaces it with None (the Default for Option), giving you
the owned value without needing Copy:
fn steal(slot: &Cell<Option<String>>) -> Option<String> {
slot.take() // moves the String out, leaves None behind
}
let name = Cell::new(Some(String::from("ferris")));
assert_eq!(steal(&name), Some(String::from("ferris")));
assert_eq!(steal(&name), None); // already taken
RefCell: runtime borrow checking
Cell can’t help when you need a reference to the contents — you can’t get() a
Vec and push to it. RefCell<T> solves this by handing out real references,
guarded by a runtime borrow flag:
borrow() -> Ref<T>: shared read borrow (many allowed)borrow_mut() -> RefMut<T>: exclusive write borrow (only one, no readers)
The returned Ref/RefMut are RAII guards. While they live, the borrow flag is
held. When they drop, the flag resets.
fn log(entries: &RefCell<Vec<String>>, msg: &str) {
entries.borrow_mut().push(msg.to_string());
}
Again: &RefCell, not &mut RefCell. The RefCell enforces exclusivity at
runtime, so the compiler accepts the shared reference.
The “&self that mutates” pattern
This is the real reason RefCell exists in practice. A struct wraps its mutable
state in RefCell and exposes all-&self methods — callers see a read-only
interface, but the struct mutates internally:
struct Stats {
samples: RefCell<Vec<i32>>,
}
impl Stats {
fn add(&self, n: i32) { // &self, NOT &mut self
self.samples.borrow_mut().push(n);
}
fn len(&self) -> usize {
self.samples.borrow().len()
}
fn sum(&self) -> i32 {
self.samples.borrow().iter().sum()
}
}
This is how caches, loggers, lazy-init fields, and counters work in safe Rust
when &mut self isn’t available.
Multiple simultaneous read borrows are fine — borrow() can be called many
times while other Ref guards are alive:
let a = s.samples.borrow();
let b = s.samples.borrow(); // both Refs alive — OK, many readers
assert_eq!(a.len(), b.len());
assert!(s.samples.try_borrow_mut().is_err()); // but a writer is refused
try_borrow / try_borrow_mut return Result instead of panicking — useful
when you’re unsure whether a borrow is already active.
Ref::map — projecting a borrow to a single field
A common need: borrow one field out of a RefCell<Struct>. You can’t return a
plain &str — the Ref guard would drop at function end, resetting the borrow
flag, and the reference would dangle. The compiler won’t let you.
Ref::map solves this by transforming the guard while keeping it alive:
fn borrow_name(c: &RefCell<Config>) -> Ref<'_, str> {
Ref::map(c.borrow(), |cfg| cfg.name.as_str())
}
fn borrow_retries_mut(c: &RefCell<Config>) -> RefMut<'_, u32> {
RefMut::map(c.borrow_mut(), |cfg| &mut cfg.retries)
}
The returned Ref<str> still holds the borrow flag down — a try_borrow_mut
will fail while it lives. When it drops, the flag releases. This lets you expose
fine-grained borrows of individual fields without leaking the whole struct.
Footguns
The runtime borrow panic (rung 5)
This is the defining RefCell hazard. Overlap a read borrow with a write
borrow and you get a panic at runtime, not a compile error:
fn trigger_panic(v: &RefCell<Vec<i32>>) {
let _r = v.borrow(); // Ref alive for the rest of the scope
v.borrow_mut().push(1); // PANIC: "already borrowed"
}
The fix is scope the borrow — end the read borrow before taking the write
borrow. Copy what you need out, drop the Ref, then mutate:
fn duplicate_first(v: &RefCell<Vec<i32>>) {
let first = v.borrow()[0]; // temporary Ref dropped at semicolon
v.borrow_mut().push(first); // now safe — no outstanding borrows
}
The trap is subtle: v.borrow()[0] creates a temporary Ref that lives only
for the expression. But let r = v.borrow(); ... r[0] keeps the Ref alive
until r goes out of scope. The difference between a temporary and a binding
is the difference between working code and a panic.
Re-entrant borrow through a callback (rung 6)
The most insidious variant: a read borrow held during iteration, and a callback
that tries to write to the same RefCell:
fn each<F: FnMut(i32)>(v: &RefCell<Vec<i32>>, mut f: F) {
for &x in v.borrow().iter() { // Ref alive for the whole loop
f(x); // if f() borrows v mutably -> PANIC
}
}
fn double_into_buggy(v: &RefCell<Vec<i32>>) {
each(v, |x| {
v.borrow_mut().push(x * 2); // re-entrant: panics
});
}
The borrow() in each holds a Ref for the entire loop body. The closure
calls borrow_mut() on the same RefCell — boom. The two borrows aren’t
adjacent in the source; the mutable one is buried in a closure. This is why
re-entrancy is the real danger with RefCell.
The fix: snapshot and release. Collect what you need, drop the read borrow, then mutate:
fn double_into_fixed(v: &RefCell<Vec<i32>>) {
let doubles = v.borrow().iter().map(|x| x * 2).collect::<Vec<_>>();
v.borrow_mut().extend(doubles);
}
The borrow() is a temporary — it lives for the collect() expression and
drops before borrow_mut() is called.
RefCell is !Sync
RefCell’s borrow flag is a plain Cell<isize> with no atomics. Sharing
&RefCell across threads would race on the flag. The compiler prevents this:
RefCell<T> is !Sync, so std::thread::scope with a shared &RefCell is a
compile error. The thread-safe equivalents are Mutex (one writer, blocks) and
RwLock (many readers or one writer, blocks).
Real-world patterns
Rc<RefCell<T>> — shared mutable state
Rc gives multiple owners. RefCell gives mutation through &. Together:
multiple handles to the same data, any of which can mutate it. This is how
graphs, trees, and observer state work in single-threaded Rust:
fn new_node(value: i32) -> Rc<RefCell<Node>> {
Rc::new(RefCell::new(Node { value, children: vec![] }))
}
fn add_child(parent: &Rc<RefCell<Node>>, child: Rc<RefCell<Node>>) {
parent.borrow_mut().children.push(child);
}
The payoff: mutate through one handle, observe through another — they share the
same underlying RefCell:
let root = new_node(1);
let a = new_node(2);
add_child(&root, Rc::clone(&a));
a.borrow_mut().value = 20; // mutate through `a`
assert_eq!(sum_tree(&root), 1 + 20); // see it through `root`
The threaded counterpart is Arc<Mutex<T>>.
Caches and lazy fields
A struct with a RefCell<Option<ExpensiveResult>> can lazily compute and cache
a value through &self:
fn get_result(&self) -> Ref<'_, ExpensiveResult> {
if self.cache.borrow().is_none() {
*self.cache.borrow_mut() = Some(expensive_compute());
}
Ref::map(self.cache.borrow(), |opt| opt.as_ref().unwrap())
}
For single-init cases, OnceCell / OnceLock are simpler; RefCell shines
when the cached value can be invalidated and recomputed.
Capstone insight
Building MyRefCell<T> from scratch reveals that the whole mechanism is just
three pieces:
1. UnsafeCell<T> — the only legal way to get a *mut T from a shared
&T. Any other route to &T -> &mut T is instant UB. UnsafeCell is the
compiler-blessed primitive that says “I know what I’m doing; don’t optimize
based on immutability.”
2. A borrow flag — a Cell<isize> tracking the state:
| Flag value | Meaning |
|---|---|
0 | Free — no borrows outstanding |
> 0 | That many shared borrows are out |
-1 | One exclusive (mutable) borrow is out |
The rules:
borrow(): panic if flag < 0 (writer out), else flag += 1.borrow_mut(): panic if flag != 0 (anyone out), else flag = -1.
3. RAII guard types — MyRef and MyRefMut. They Deref to the data
(via the UnsafeCell’s raw pointer), and their Drop impl restores the flag.
This is why borrows auto-release — when the guard goes out of scope, the
destructor runs and the flag resets:
impl<T> Deref for MyRef<'_, T> {
type Target = T;
fn deref(&self) -> &T {
unsafe { &*self.cell.value.get() }
}
}
impl<T> Drop for MyRef<'_, T> {
fn drop(&mut self) {
self.cell.flag.set(self.cell.flag.get() - 1);
}
}
impl<T> Drop for MyRefMut<'_, T> {
fn drop(&mut self) {
self.cell.flag.set(0);
}
}
The unsafe in Deref is sound because the flag guarantees the aliasing
invariant: if a MyRef exists, no MyRefMut can exist (flag would be -1, but
it’s > 0), and vice versa. The flag is the proof obligation — get it right and
the unsafe is justified; get it wrong and you have UB.
Once you’ve written this, RefCell stops being magic. It’s a Cell<isize>
counter plus two RAII types that hold it. The borrow checker didn’t go away — it
moved into your flag arithmetic.
Explain it back
- Why does
Cell::getrequireT: Copy? What would go wrong if it handed out a&Tinstead? - What is the exact runtime cost of
RefCellcompared to a plain&mut? (Hint: it’s a flag check, not a lock.) - When a
Refdrops, what happens to the borrow flag? Why is this an RAII pattern? - Why does
let _r = v.borrow(); v.borrow_mut().push(1);panic, butlet x = v.borrow()[0]; v.borrow_mut().push(x);doesn’t? - What makes
RefCell!Sync? What’s the thread-safe replacement? - In the re-entrant callback rung, where is the read borrow still alive when the write borrow fires? Why can’t the compiler catch this at compile time?
- What is
UnsafeCelland why is it the foundation of all interior mutability in Rust? - In
MyRefCell, why is theunsafeinDeref for MyRefsound?
See also
- Rc<RefCell<T>> patterns — the full treatment of the
Rc<RefCell<T>>combo: cycles,Weak, observer pattern, doubly-linked list. - Box & the Heap — sole ownership on the heap;
Box<dyn Trait>is the owned trait-object counterpart toRc<RefCell<dyn Trait>>. - Drop & Ordering — RAII guards and
mem::take/replace, the same patterns that makeRefCellguards andCell::takework.
Conversion traits — From / Into, TryFrom / TryInto, AsRef / AsMut
Ladder:
src/bin/conversions.rs· Run:cargo run --bin conversions· Phase 1 · 9 rungs
TL;DR
Type conversion in Rust is a small family of traits, split on two questions: can it fail? and do you consume or just borrow?
| infallible | fallible | |
|---|---|---|
| take ownership | From / Into | TryFrom / TryInto |
| just borrow | AsRef / AsMut | — |
The unlock that makes the whole family small: you only ever implement From
and TryFrom. The Into and TryInto directions are handed to you for free
by blanket impls. And the ? operator converts error types through From, so
making heterogeneous errors collapse into one type is also just writing From
impls. Almost everything on this page falls out of those two facts.
Why this exists (from first principles)
A conversion is a function A -> B. You could just write free functions
(celsius_to_fahrenheit, string_from_char, …) and be done. The reason Rust
lifts conversions into traits is that traits are how you write code generic
over “anything convertible.” Once conversion is a trait, a function can say
“give me anything that becomes a String” and the compiler wires up the right
conversion at each call site. Free functions can’t do that.
But one trait isn’t enough, because conversions differ along two independent axes that the type system has to respect:
-
Can it fail? Turning a
Celsiusinto aFahrenheitalways succeeds — the result type can hold any value. Turning ani32into au8cannot always succeed:300doesn’t fit. An infallible conversion returnsB; a fallible one must returnResult<B, E>. You cannot model both with one signature, so the family splits intoFrom(returnsSelf) andTryFrom(returnsResult<Self, Self::Error>). -
Do you need to own the input? Producing a
Stringfrom a&strallocates and consumes nothing it can’t recreate — that’sFrom/Into, which take the value by move. But a function that only reads text shouldn’t demand ownership or force a clone. It just needs a&strview of whatever you have. That’sAsRef: a cheap, non-consuming “give me a&Tof yourself.”
What the compiler guarantees, given these traits: conversions are explicit and
type-directed. There’s no silent coercion between unrelated types — you either
call .into()/.try_into() (and the target type drives which impl runs) or you
get a compile error. The one infamous exception is the as keyword, which is
not a trait and silently truncates — rung 6 is about why you should reach for
TryInto instead.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | foundations | From basics | impl From<Celsius> -> .into() comes free via the blanket impl |
| 2 | foundations | Into bounds | impl Into<String> params accept &str, String, char… convert once at the boundary |
| 3 | mechanics | From powers ? | ? inserts From::from on the error -> many error types collapse into one |
| 4 | mechanics | TryFrom | fallible construction with an associated Error; try_into() comes free |
| 5 | footgun | reflexivity & orphan rule | From<T> for T is identity; you can’t impl a foreign trait for a foreign type -> newtype |
| 6 | footgun | as vs TryInto | as silently wraps (300 as u8 == 44); TryInto<u8> catches the overflow |
| 7 | real-world | AsRef<str> / AsRef<[u8]> | accept many types by reference, no allocation — the stdlib API shape |
| 8 | real-world | AsRef<Path> + AsMut | the File::open trick; AsMut for an in-place mutable view |
| 9 | capstone | mini JSON Value | From in (infallible), AsRef<str> lookup, TryFrom out (fallible) |
The ideas, built up
From is the one you implement; Into is the one you get
Implement From in one direction and the reverse .into() appears for free:
struct Celsius(f64);
struct Fahrenheit(f64);
impl From<Celsius> for Fahrenheit {
fn from(c: Celsius) -> Self {
Fahrenheit(c.0 * 9.0 / 5.0 + 32.0)
}
}
Both of these call the same impl:
let f1 = Fahrenheit::from(Celsius(100.0)); // explicit From
let f2: Fahrenheit = Celsius(0.0).into(); // .into() — free, type-driven
You never write impl Into<Fahrenheit> for Celsius. The stdlib has a blanket
impl that derives it from your From:
impl<T, U: From<T>> Into<U> for T { /* calls U::from(self) */ }
This is the rule to memorize: implement From, callers enjoy Into. The
asymmetry exists because the blanket impl only flows one way — From -> Into —
and (historically) you couldn’t even impl Into for a foreign type. From is
always the right thing to write.
Notice in f2 the conversion is driven by the target type annotation
(let f2: Fahrenheit). .into() is “convert into something”; the compiler
figures out which From impl from the type it’s assigned to. No annotation, no
resolution.
Into bounds make APIs ergonomic — convert once at the boundary
The real reason Into matters at the call site: a parameter typed
impl Into<String> accepts anything that knows how to become a String, and
the function converts exactly once, at the boundary.
struct Tag { name: String }
impl Tag {
fn new(name: impl Into<String>) -> Self {
Self { name: name.into() }
}
}
One function, three different argument types, zero clones written by the caller:
let a = Tag::new("literal"); // &'static str
let b = Tag::new(String::from("owned")); // String — a no-op conversion
let c = Tag::new('x'); // char -> String!
The String -> String case is free because of the reflexive impl (rung 5): it’s
a real but no-op From. So you pay nothing for the flexibility when the caller
already has the owned type.
Rule of thumb: put
impl Into<T>(orT: From<X>) on the caller side of a generic boundary when the function needs to store an ownedT. If it only needs to read the data, useAsRefinstead (rung 7) — don’t take ownership you don’t need.
From powers the ? operator — the most important fact here
This is why From matters more than any other trait on the page. When you write
? on a Result whose error type doesn’t match the function’s return error
type, the compiler inserts .map_err(From::from) for you. So you make
heterogeneous errors flow into one error type just by implementing From for
each source error.
#[derive(Debug, PartialEq)]
enum ConfigError {
NotANumber(ParseIntError),
OutOfRange(i32),
}
impl From<ParseIntError> for ConfigError {
fn from(error: ParseIntError) -> Self {
ConfigError::NotANumber(error)
}
}
fn parse_config(s: &str) -> Result<i32, ConfigError> {
let n: i32 = s.parse()?; // parse() errors with ParseIntError
if !(0..=100).contains(&n) {
return Err(ConfigError::OutOfRange(n)); // returned explicitly
}
Ok(n)
}
The s.parse()? line is the whole lesson. parse() returns
Result<i32, ParseIntError>, but the function returns Result<_, ConfigError>.
The ? desugars roughly to:
let n = match s.parse() {
Ok(v) => v,
Err(e) => return Err(ConfigError::from(e)), // From::from inserted here
};
Because you wrote From<ParseIntError> for ConfigError, that conversion exists
and the code compiles. This is the engine behind anyhow, thiserror, and
every hand-rolled error enum: ? + From turns many failure types into one.
TryFrom — when the conversion can fail
From::from returns Self — it has no way to signal failure. So when a
conversion can fail, From is simply the wrong trait. TryFrom is the fallible
twin: fn try_from(v) -> Result<Self, Self::Error>, with an associated error
type you choose.
struct Percent(i32);
#[derive(Debug, PartialEq)]
enum PercentError { OutOfRange(i32) }
impl TryFrom<i32> for Percent {
type Error = PercentError;
fn try_from(value: i32) -> Result<Self, Self::Error> {
if value < 0 || value > 100 {
return Err(PercentError::OutOfRange(value));
}
Ok(Percent(value))
}
}
Exactly mirroring From -> Into, implementing TryFrom gives you try_into()
for free:
let p: Result<Percent, _> = 100.try_into(); // free from the TryFrom impl
Note you must annotate the target (Result<Percent, _>) so the compiler knows
which TryInto to pick — same type-direction rule as .into(). And TryFrom
composes with ? because the error type already matches:
fn make(n: i32) -> Result<Percent, PercentError> {
let p = Percent::try_from(n)?; // ? works — error type is already PercentError
Ok(p)
}
Here the ? calls From::from on the error, but since it’s PercentError -> PercentError that’s the reflexive identity — no conversion needed. Which leads
straight to rung 5.
Reflexivity and the orphan rule — two coherence facts that bite
(a) impl<T> From<T> for T exists in the stdlib. Every type can “convert” to
itself — a no-op identity. This quietly makes three earlier things work:
?works even when the error types already match (it callsFrom::from, which is identity here).impl Into<String>accepts aStringat zero cost (String: From<String>).u64::from(42u64)is a real, if pointless, conversion.
let same = u64::from(42u64); // identity From — a genuine impl, just a no-op
assert_eq!(same, 42);
(b) The orphan rule (coherence). You may implement a trait for a type only if the trait or the type is local to your crate. So this is rejected:
// WRONG — both From and Duration are foreign to your crate:
// impl From<u64> for std::time::Duration { ... } // E0117
Uncommenting that in the source produces E0117 "only traits defined in the current crate can be implemented for types defined outside of the crate." You
cannot make it compile from this crate — that’s the entire point. The rule
exists so two different crates can’t write conflicting impls for the same
trait/type pair and break each other.
The universal fix: a newtype you own.
struct Timeout(Duration); // a local type -> now you CAN impl From for it
impl From<u64> for Timeout {
fn from(secs: u64) -> Self {
Timeout(Duration::from_secs(secs))
}
}
fn secs_to_timeout(secs: u64) -> Timeout {
secs.into() // resolves to your From<u64> for Timeout
}
Because Timeout is local, the orphan rule is satisfied and the impl is allowed.
This is also a second reason you implement From and never Into: the blanket
impl gives Into for free and historically you couldn’t impl Into for a
foreign type at all.
as truncates silently; TryInto is the checked path
as casts between numeric types and never fails — it silently
truncates/wraps. This is a notorious bug source:
let truncated = 300i32 as u8; // == 44, no error, no warning
assert_eq!(truncated, 44); // 300 - 256 = 44, wrapped around
The safe counterpart is TryFrom/TryInto, which returns Err when the value
doesn’t fit. You can write a generic that narrows anything try-convertible into
a u8:
fn narrow<T: TryInto<u8>>(value: T) -> Result<u8, T::Error> {
value.try_into()
}
Two things to unpack in that signature:
T: TryInto<u8>— the bound is on the caller’s type, accepting any integer type that knows how to try to become au8.T::Error— the error type isn’t named; it’s the trait’s associated type. Different source types may have different error types, and the return type tracks whichever oneTbrings.
assert!(narrow(300i32).is_err()); // doesn't fit -> Err (vs. `as` -> 44)
assert_eq!(narrow(200i32), Ok(200)); // fits
assert_eq!(narrow(200u32), Ok(200)); // different input type, same bound
assert!(narrow(-1i32).is_err()); // negative -> Err
This is exactly how the stdlib downcasts integers safely: u8::try_from(x) and
x.try_into(). Reach for them whenever a numeric narrowing could lose data.
AsRef — cheap reference conversions, no allocation
From/Into consume a value and usually allocate. But often a function only
needs to read the data — it shouldn’t demand ownership or force a clone.
AsRef<T> is the answer: a zero-cost “give me a &T view of myself.” &str,
String, &String, Box<str> all impl AsRef<str>, so a single bound accepts
all of them by reference:
fn shout<S: AsRef<str>>(s: S) -> String {
s.as_ref().to_uppercase()
}
fn byte_len<B: AsRef<[u8]>>(b: B) -> usize {
b.as_ref().len()
}
The caller passes whatever it has, and a borrowed input stays usable afterward:
let owned = String::from("hi");
assert_eq!(shout(&owned), "HI"); // &String -> &str view
assert_eq!(owned, "hi"); // still usable: shout only borrowed it
AsRef<[u8]> is even broader — it unifies &str, String, &[u8], Vec<u8>,
and arrays as a byte view:
assert_eq!(byte_len("abc"), 3); // &str
assert_eq!(byte_len(vec![1u8, 2, 3]), 3); // Vec<u8>
assert_eq!(byte_len([0u8; 5]), 5); // [u8; 5]
AsRefvsInto, the decision: useimpl Into<String>when you need to store an ownedString(rung 2). Useimpl AsRef<str>when you only need to look at the text (rung 7). Taking ownership you don’t need forces needless clones on the caller.
AsRef<Path> (the File::open trick) and AsMut
The most famous AsRef in the stdlib is the signature of File::open:
fn open<P: AsRef<Path>>(path: P) -> io::Result<File>
That single bound is why File::open("f.txt"), File::open(string), and
File::open(path_buf) all work — &str, String, PathBuf, and &Path all
impl AsRef<Path>. You write the bound once; callers pass whatever path-like
thing they hold. The ladder mirrors it:
fn extension<P: AsRef<Path>>(p: P) -> Option<String> {
p.as_ref()
.extension() // Option<&OsStr>
.and_then(|e| e.to_str()) // Option<&str>
.map(String::from) // Option<String>
}
AsMut is the mutable mirror: as_mut() hands back a &mut T view, so one
function can mutate a Vec, an array, or a &mut slice in place:
fn double_all<T: AsMut<[i32]>>(mut data: T) -> T {
data.as_mut().iter_mut().for_each(|x| *x *= 2);
data
}
assert_eq!(double_all(vec![1, 2, 3]), vec![2, 4, 6]); // Vec<i32>
assert_eq!(double_all([10, 20]), [20, 40]); // [i32; 2]
AsMut<[i32]> abstracts over “anything that can lend a mutable i32 slice,”
so the in-place algorithm is written once and works across container types.
Capstone insight: data flows in infallibly, out fallibly
The capstone builds a mini serde_json::Value and wires the whole family
together — and the structural “aha” is the asymmetry:
Data flows into a dynamic type infallibly (
From— aboolalways makes a validValue). Data flows out fallibly (TryFrom— aValuemight not be the type you asked for). That asymmetry is the entire reason both traits exist.
enum Value {
Null, Bool(bool), Num(f64), Str(String),
Array(Vec<Value>), Object(Vec<(String, Value)>),
}
In, infallibly — every Rust value maps to some valid Value:
impl From<bool> for Value { fn from(b: bool) -> Self { Value::Bool(b) } }
impl From<i64> for Value { fn from(n: i64) -> Self { Value::Num(n as f64) } }
impl From<&str> for Value { fn from(s: &str) -> Self { Value::Str(s.to_string()) } }
impl From<Vec<Value>> for Value { fn from(v: Vec<Value>) -> Self { Value::Array(v) } }
This makes construction ergonomic, even for heterogeneous nested data — every
element just .into()s:
let arr: Value = vec![1i64.into(), "two".into(), true.into()].into();
Lookup, by AsRef<str> — the key bound lets callers pass a &str or a
String:
fn get<S: AsRef<str>>(&self, key: S) -> Option<&Value> {
let key = key.as_ref();
if let Value::Object(object) = self {
object.iter().find(|(k, _)| k == key).map(|(_, v)| v)
} else {
None // not an object -> no key
}
}
obj.get("name"); // &str key
obj.get(String::from("age")); // String key — same function
Out, fallibly — extraction can disagree with the stored variant, so it
returns Result:
impl TryFrom<Value> for f64 {
type Error = WrongType;
fn try_from(v: Value) -> Result<Self, Self::Error> {
if let Value::Num(n) = v { Ok(n) } else { Err(WrongType) }
}
}
impl TryFrom<Value> for String {
type Error = WrongType;
fn try_from(v: Value) -> Result<Self, Self::Error> {
if let Value::Str(s) = v { Ok(s) } else { Err(WrongType) }
}
}
let name = String::try_from(obj.get("name").unwrap().clone()).unwrap(); // "ada"
let age: f64 = obj.get("age").unwrap().clone().try_into().unwrap(); // 36.0
assert_eq!(f64::try_from(Value::Bool(true)), Err(WrongType)); // wrong type -> Err
Once you see this, every dynamic/serialization boundary in Rust reads the same
way: From to build the loose representation, TryFrom to safely pull typed
values back out, AsRef to keep the read-side flexible.
Footguns
-
assilently truncates/wraps numeric casts.300i32 as u8 == 44, no warning. Useu8::try_from(x)/x.try_into()whenever a narrowing could lose data — they returnErrinstead of corrupting the value. -
The orphan rule blocks
impl ForeignTrait for ForeignType. You can’timpl From<u64> for Durationfrom your crate (E0117). Wrap the foreign type in a newtype you own and impl on that. -
Implement
From, neverInto. The blanket impl derivesIntofrom yourFrom. WritingIntoby hand is redundant and was historically impossible for foreign types. -
.into()/.try_into()need a known target type. They convert “into something”; if the target isn’t pinned by an annotation or the surrounding context, the compiler can’t pick an impl. Annotate the binding or the return. -
Taking ownership when you only read. Using
impl Into<String>whereimpl AsRef<str>would do forces callers to give up (or clone) their data. Match the bound to what the function actually needs: store ->Into, read ->AsRef. -
Fromcan’t fail. If a conversion has any invalid inputs, it must beTryFrom. Reaching forFromand panicking inside is a code smell — return aResultinstead.
Real-world patterns
| Pattern | Trait | Example |
|---|---|---|
| Ergonomic constructor | impl Into<String> param | Tag::new("x"), builder APIs |
| Collapse many errors into one | From<E> + ? | anyhow, thiserror, custom error enums |
| Validated construction | TryFrom<Raw> | Percent::try_from(150) -> Err, Ipv4Addr::try_from(bytes) |
| Safe numeric narrowing | u8::try_from / TryInto<u8> | downcasting integers without as |
| Read-only string/byte arg | impl AsRef<str> / AsRef<[u8]> | str helpers, hashing, parsing |
| Path-like argument | impl AsRef<Path> | File::open, fs::read, Path::join |
| In-place mutation over containers | impl AsMut<[T]> | generic slice transforms |
| Dynamic value boundary | From in, TryFrom out, AsRef lookup | serde_json::Value, config trees |
Explain it back
- Why do you only ever implement
From, and where does.into()come from? - What does
?insert on the error path, and which trait must you implement to make a foreign error type flow into your error enum? - When is
Fromthe wrong choice, and what’s the fallible replacement? What does its associatedErrortype let you control? - Why does
?compile even when the error types already match? (Which std impl?) - State the orphan rule in one sentence. Why can’t you
impl From<u64> for Duration, and what’s the standard fix? 300i32 as u8is what, and why? What should you write instead, and what does it return on overflow?- You have a function that only needs to read a string.
impl Into<String>orimpl AsRef<str>— which, and why does it matter to the caller? - Why is
File::open’sP: AsRef<Path>bound so convenient? Name three types that satisfy it. - In the JSON
Valuecapstone, why is constructionFrombut extractionTryFrom? What does that asymmetry reflect about the data?
See also
Borrow/ToOwned—AsRef’s cousins;Borrowadds anEq/Hashcontract thatAsRefdoesn’t, which is whyHashMapkeys useBorrow, notAsRefCow— Clone-on-Write — pairs withInto/AsReffor APIs that borrow when they can and own when they mustBox& the Heap —Box<dyn Error>is the other half of the?/Fromerror-conversion story
Borrow / ToOwned
Ladder:
src/bin/borrow_toowned.rs· Run:cargo run --bin borrow_toowned· Phase 1 · 9 rungs
TL;DR
ToOwned and Borrow are the two traits that sit underneath Cow and
HashMap-key lookups.
ToOwnedis a generalizedClonefor when the borrowed and owned types differ:&str -> String,&[T] -> Vec<T>.Cloneis&T -> T(same type), so it can’t expressstr -> String;ToOwnedcan, via an associatedOwnedtype.Borrow<B>is the other direction — view an owned value as a borrowed&B(String -> &str) — but with a contract: the view must hash, compare, and order identically to the owner. That contract is exactly what lets aHashMap<String, V>be queried by&strwithout allocating.
Why this exists (from first principles)
A HashMap<String, V> stores owned String keys. You want to look something
up with a cheap &str literal — without building a throwaway String every call.
That’s only sound if &str hashes to the same bucket the String went into.
Borrow<str> for String is the promise that it does.
But the problem is deeper than just HashMap. Consider str and String: they
are different types, yet they represent the same data in different ownership
modes. Standard Clone can’t express this — Clone is &T -> T, same type in,
same type out. You can’t impl Clone for str to produce a String. So Rust
needs a trait that says “given a borrowed &str, produce its owned counterpart
String” — that’s ToOwned. And it needs the reverse: “given an owned String,
produce a borrowed &str view” — that’s Borrow.
Together, these two traits form a round-trip contract between borrowed and
owned forms. Cow is built directly on top of them: its Owned variant is
<B as ToOwned>::Owned, and Borrow is how it hands out &B from that variant.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | foundations | &str -> String, &[i32] -> Vec via .to_owned() | The owned type is a different type than the input. |
| 2 | foundations | Borrow a &str out of a &String; borrow_sum<T: Borrow<[i32]>> | “View owned as borrowed”; one fn takes Vec or slice. |
| 3 | mechanics | HashMap<String,_>::get("key") + hand-written contains_key2 | Read and write the K: Borrow<Q> bound — the payoff. |
| 4 | mechanics | owned_pair<T: ToOwned> returning (T::Owned, T::Owned) | Name the associated Owned type; why you can’t return T. |
| 5 | footgun | CiString (case-insensitive) — AsRef yes, Borrow no | Borrow needs Eq/Hash transparency; AsRef makes no promise. |
| 6 | footgun | Cache::get<Q> instead of .to_string() per lookup | Borrow the lookup key — don’t allocate to query. |
| 7 | real-world | TagSet: add<S: Into<String>> + has<Q: Borrow> | Own at insert, borrow at query. |
| 8 | real-world | make_owned (= Cow::into_owned) + pick (Cow producer) | Why Cow<B> requires B: ToOwned. |
| 9 | capstone | Hand-rolled MyBorrow + MyToOwned + MyCow | The whole machine, from scratch. |
The ideas, built up
ToOwned: Clone across type boundaries
Clone is &T -> T — same type. That works fine for i32 or Vec<String>,
where the owned form and the borrowed form are the same type. But str and
String are fundamentally different types. str is unsized (a [u8] with a
UTF-8 invariant), living behind references. String is a Vec<u8> on the
heap. You can’t clone a str into a str — there’s nowhere to put it.
ToOwned bridges the gap with an associated type:
pub trait ToOwned {
type Owned: Borrow<Self>; // the owned form must borrow BACK to Self
fn to_owned(&self) -> Self::Owned;
}
So str: ToOwned<Owned = String> and [T]: ToOwned<Owned = Vec<T>>. The
.to_owned() call on a &str produces a String:
fn duplicate(s: &str) -> String {
s.to_owned()
}
fn duplicate_slice(xs: &[i32]) -> Vec<i32> {
xs.to_owned()
}
The return types are different types than the inputs. That’s the whole point —
Clone can’t do this.
Borrow: the other direction, with a contract
Borrow<B> goes the opposite way: given an owned value, hand out a borrowed
&B view. String: Borrow<str> and Vec<T>: Borrow<[T]>. There’s also a
blanket T: Borrow<T> so every type can borrow as itself.
fn borrow_sum<T: Borrow<[i32]>>(xs: T) -> i32 {
let slice: &[i32] = xs.borrow();
slice.iter().sum()
}
This one function accepts both Vec<i32> and &[i32] — borrow() normalizes
either to &[i32].
But Borrow is not just “give me a reference.” It carries a semantic
contract: x and x.borrow() must produce the same Eq, Ord, and Hash
results. This is critical for HashMap and is what distinguishes Borrow from
AsRef.
The payoff: HashMap lookup without allocation
This is why Borrow exists. The HashMap::get signature is:
fn get<Q>(&self, k: &Q) -> Option<&V>
where
K: Borrow<Q>, // the stored key can be viewed as Q
Q: Hash + Eq + ?Sized // Q = str is unsized; only touched behind &Q
Read it as: “the stored key K can be Borrow’d as Q.” With K = String
and Q = str, String: Borrow<str> holds, so map.get("key") just works —
no String allocation needed.
The contract is what makes this sound: when the map hashes the &str query,
it computes the same hash that the String key produced at insertion time. If
those hashes differed, the lookup would silently miss the bucket.
Writing the bound yourself makes it stick:
fn contains_key2<K, Q>(map: &HashMap<K, u32>, key: &Q) -> bool
where
K: Borrow<Q> + Eq + Hash,
Q: Eq + Hash + ?Sized,
{
map.contains_key(key)
}
The ?Sized on Q is required because Q = str is unsized — it’s only ever
touched behind &Q, so unsized is fine.
The associated type puzzle
When generic over T: ToOwned, the owned value’s type is spelled T::Owned
(or <T as ToOwned>::Owned) — never T. This trips people up. T is the
borrowed type (e.g. str), which is usually unsized and can’t be returned by
value:
fn owned_pair<T: ToOwned + ?Sized>(value: &T) -> (T::Owned, T::Owned) {
(value.to_owned(), value.to_owned())
}
// Called with T = str:
let (a, b): (String, String) = owned_pair("hi");
// Called with T = [i32]:
let (v1, v2): (Vec<i32>, Vec<i32>) = owned_pair(&[1, 2][..]);
The ?Sized bound on T is needed because str and [T] are unsized types —
without it, the compiler demands T: Sized and rejects owned_pair::<str>.
Footguns
Borrow vs AsRef: same shape, different promise
Borrow<T> and AsRef<T> have the same signature: fn(&self) -> &T. So
why two traits?
AsRef<T>: “you can view me as&T.” No other guarantee. Use it for flexible function arguments (accept&str,String,PathBuf, …).Borrow<T>: the view is semantically transparent —xandx.borrow()must produce the sameEq/Ord/Hash. Implement it only when that holds.
The CiString proof (rung 5)
A case-insensitive string hashes "Hello" and "HELLO" identically, but
plain str hashes them differently:
impl Hash for CiString {
fn hash<H: Hasher>(&self, state: &mut H) {
for b in self.0.bytes() {
state.write_u8(b.to_ascii_lowercase());
}
}
}
A Borrow<str> impl for CiString would force str’s hasher on lookup and
silently miss the bucket. The ladder proves this by computing hashes both
ways:
// CiString hashes case-insensitively: "Hello" == "HELLO"
assert_eq!(h(&CiString::new("Hello")), h(&CiString::new("HELLO")));
// but plain str hashes exactly: "Hello" != "HELLO"
assert_ne!(h("Hello"), h("HELLO"));
So CiString implements AsRef<str> (legal — AsRef makes no promise) but
deliberately not Borrow<str>. When the equivalence relations don’t match,
you must honestly allocate a CiString to query:
fn find_ci(map: &HashMap<CiString, i32>, query: &str) -> Option<i32> {
let key = CiString::new(query); // must allocate — no Borrow shortcut
map.get(&key).copied()
}
Needless .to_string() at lookup (rung 6)
The classic wasteful pattern:
fn get_bad(&self, key: &str) -> Option<&str> {
self.0.get(&key.to_string())... // WRONG: allocates per lookup!
}
The fix is one generic method that accepts a borrowed key directly:
fn get<Q>(&self, key: &Q) -> Option<&str>
where
String: Borrow<Q>,
Q: Hash + Eq + ?Sized,
{
self.0.get(key).map(|v| v.as_str()) // OK: zero allocation
}
Reflexively reach for key: &Q where Key: Borrow<Q> instead of taking or
owning a String at query boundaries.
Real-world patterns
Into-in / Borrow-out (rung 7)
A keyed collection has two boundaries that want different traits:
impl TagSet {
// INSERT: you must end up OWNING -> accept impl Into<String> (at most 1 alloc)
fn add<S: Into<String>>(&mut self, tag: S) {
self.tags.insert(tag.into());
}
// QUERY: you only LOOK -> borrow, never allocate
fn has<Q>(&self, tag: &Q) -> bool
where
String: Borrow<Q>,
Q: Hash + Eq + ?Sized,
{
self.tags.contains(tag)
}
}
This is the pattern real APIs use: Into at the ownership boundary (insert,
store, construct), Borrow at the lookup boundary (get, contains, find).
Borrow gives breadth for free
One Borrow<str> bound accepts &str, String, Box<str>, Rc<str>, and
Cow<str>:
fn shout<S: Borrow<str>>(s: S) -> String {
s.borrow().to_uppercase()
}
assert_eq!(shout("hi"), "HI"); // &str
assert_eq!(shout(String::from("yo")), "YO"); // String
assert_eq!(shout(Box::<str>::from("be")), "BE"); // Box<str>
assert_eq!(shout(Rc::<str>::from("rc")), "RC"); // Rc<str>
assert_eq!(shout(Cow::Borrowed("cow")), "COW"); // Cow<str>
Closing the Cow loop (rung 8)
pub enum Cow<'a, B: ToOwned + ?Sized> {
Borrowed(&'a B),
Owned(<B as ToOwned>::Owned),
}
B: ToOwned is mandatory: the Owned variant must name a concrete owned
type (<B as ToOwned>::Owned), and to_owned() is the only way to manufacture
one from a borrow. Re-implementing Cow::into_owned yourself proves this is
the only mechanism:
fn make_owned<B: ToOwned + ?Sized>(c: Cow<'_, B>) -> B::Owned {
match c {
Cow::Borrowed(b) => b.to_owned(), // ToOwned builds the owned form
Cow::Owned(o) => o, // already there
}
}
That’s the full answer to “why does Cow require B: ToOwned?” — without
it, Cow couldn’t name its owned half nor build it on demand.
Signatures to know
// ToOwned — generalized Clone across type boundaries
pub trait ToOwned {
type Owned: Borrow<Self>; // the owned form must borrow BACK to Self
fn to_owned(&self) -> Self::Owned;
}
// Borrow — view owned as borrowed, with Eq/Ord/Hash transparency
pub trait Borrow<Borrowed: ?Sized> {
fn borrow(&self) -> &Borrowed;
}
// HashMap::get — the single most important real-world use
fn get<Q>(&self, k: &Q) -> Option<&V>
where
K: Borrow<Q>, // the stored key can be viewed as Q
Q: Hash + Eq + ?Sized // Q = str is unsized; only touched behind &Q
Capstone insight
The structural insight from building MyBorrow + MyToOwned + MyCow from
scratch: MyToOwned::Owned carries a MyBorrow<Self> bound — the owned type
must borrow back to Self. That round-trip guarantee is exactly what lets
MyCow::borrow() return &B from the Owned variant:
trait MyToOwned {
type Owned: MyBorrow<Self>;
fn my_to_owned(&self) -> Self::Owned;
}
impl<'a, B: MyToOwned + ?Sized> MyCow<'a, B> {
fn borrow(&self) -> &B {
match self {
Self::Borrowed(b) => b,
Self::Owned(o) => o.my_borrow(), // MyBorrow<Self> makes this possible
}
}
}
Without the Owned: MyBorrow<Self> bound, the Owned arm couldn’t produce a
&B — there’d be no trait method to call. And Self: ?Sized being the default
in trait defs is why impl MyToOwned for str (an unsized type) is even legal.
Explain it back
- Why can’t
strjustimpl Cloneto produce aString? (Clone is&T -> T, same type;str -> Stringneeds the differingOwnedassociated type.) - What exactly is the
Borrowcontract, and what breaks if you violate it? - Why is the bound
K: Borrow<Q>and notQ: Borrow<K>? - In
T::Owned, why can’t the return type just beT? - Why does
Cow<B>requireB: ToOwned? Name both reasons (name it / build it). - When do you pick
AsRef<T>overBorrow<T>for a function argument?
See also
- Cow — this note closes the loop opened there;
Cowis built directly onToOwnedandBorrow. - Drop & Ordering —
mem::replace, used internally byCow::to_mut(), is covered in depth there.
Drop & Ordering
Ladder:
src/bin/drop_ordering.rs· Run:cargo run --bin drop_ordering· Phase 1 · 9 rungs
TL;DR
When a value goes out of scope, Rust runs its destructor automatically — no GC,
no free(), no forgetting. The Drop trait gives you a hook into that moment.
The real depth is in ordering: locals drop in reverse declaration order
(LIFO), struct fields drop in declaration order (FIFO), and the compiler inserts
hidden drop flags so a conditionally-moved value is dropped exactly once.
mem::forget, mem::replace, and ManuallyDrop give you escape hatches when
the defaults don’t fit. The payoff is RAII: tie any cleanup action to a
scope, and it runs on every exit path — normal return, early return, or panic.
Why this exists (from first principles)
C gives you malloc/free and hopes you pair them. C++ gives you destructors
but lets you misuse them (double free, use after free). Garbage collectors solve
the pairing problem but add latency spikes and can’t manage non-memory resources
(file handles, locks, network connections) without finalizers that run “sometime,
maybe.”
Rust’s answer: ownership determines cleanup. Every value has exactly one
owner. When that owner’s scope ends, the value is dropped — deterministically,
immediately, in a well-defined order. The Drop trait is the hook that lets you
run code at that moment.
This determinism is what makes RAII (Resource Acquisition Is Initialization) a
first-class pattern: a MutexGuard unlocks on drop, a File flushes and
closes, a TempDir deletes itself. The compiler guarantees the cleanup runs,
and ownership guarantees it runs exactly once.
But “exactly once, in a well-defined order” means you need to know that order. And you need tools for the cases where the default order is wrong, or where you want to skip the destructor entirely. That’s what this ladder teaches.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | foundations | Drop at scope end | impl Drop logs when a value dies — destructor is automatic |
| 2 | foundations | Local drop order | Locals drop in reverse declaration order (LIFO) |
| 3 | mechanics | Struct & nested order | Container’s drop() runs first; fields drop in declaration order |
| 4 | mechanics | Early drop | std::mem::drop(x) ends a value early; x.drop() is E0040 |
| 5 | footguns | Drop flags | Conditional moves tracked at runtime — no double drop, ever |
| 6 | footguns | forget / take / replace | mem::forget leaks; mem::replace moves a value out of &mut |
| 7 | real-world | RAII scope guard | A closure that runs on drop, with .cancel() to disarm |
| 8 | real-world | ManuallyDrop | Suppress auto-drop; choose your own field-drop order |
| 9 | capstone | Rollback-on-drop Transaction | Drop + drop flag + forget = auto-rollback unless committed |
The ideas, built up
Drop fires at scope end — and you don’t call it
The Drop trait has one method:
impl Drop for Noisy {
fn drop(&mut self) {
log(format!("drop {}", self.name));
}
}
The compiler inserts a call to this at the end of the owning scope. You never
call drop() yourself — in fact, x.drop() is a hard compiler error (E0040: explicit use of destructor method). The reason: after your drop(&mut self)
body runs, the compiler still drops each field. If you could call .drop() on a
live binding, the automatic scope-end drop would run the destructor again —
double free. So the compiler forbids the direct call entirely.
To drop early, you use the free function std::mem::drop(x), which takes x
by value. Ownership moves into drop(), the value dies at the end of that
tiny function, and x is now moved-from — using it again is a compile error.
That’s the mechanism that prevents double free: not a runtime check, but a
move.
Two orderings to memorize
Here is where people get confused, because locals and struct fields follow opposite rules:
| What | Drop order | Why |
|---|---|---|
| Locals in a scope | Reverse declaration order (LIFO) | Like a stack: last declared = first cleaned up. This mirrors C++ and ensures that later locals (which might reference earlier ones) die first. |
| Struct fields | Declaration order (FIFO) | Top to bottom, as written in the struct definition. The container’s own Drop::drop() runs before any field drops. |
The ladder makes this concrete with Pair { id, a: Noisy, b: Noisy }:
impl Drop for Pair {
fn drop(&mut self) {
log(format!("drop pair {}", self.id));
}
}
Dropping a Pair produces: ["drop pair P", "drop a", "drop b"]. The
container’s body runs first (while fields are still alive — you can read them in
your drop()), then fields drop in declaration order: a before b.
This is the opposite of locals. If you declared let a; let b; in a function,
you’d get b before a. But if a and b are fields, you get a before
b.
Why the container drops first: Your Drop impl gets &mut self, meaning
it can still read all the fields. If fields dropped first, your drop() body
would be reading dangling references. So the container must go first.
Drop flags: the compiler’s runtime bookkeeping
Consider this:
fn conditional_move(take_it: bool) -> Vec<String> {
let x = Noisy::new("x");
if take_it {
consume(x); // x moved into consume, drops there
}
// scope end: does x need dropping?
}
When take_it is true, x is moved into consume() and drops inside it. When
false, x is still alive at scope end and drops there. Either way, x drops
exactly once. But the compiler can’t know at compile time which branch ran.
The solution: a hidden boolean on the stack — a drop flag — next to x. It
starts as “needs dropping.” When x is moved, the flag is cleared. At scope
end, the compiler checks the flag and only drops if it’s still set.
You never write this flag. You never see it. But it’s there, and it’s how Rust guarantees “exactly once” even across conditional control flow. The cost is one byte and one branch per conditionally-moved value — cheap insurance against double free or leak.
forget, replace, take: bending the rules
Three std::mem functions that give you manual control over when (or whether)
destructors run:
mem::forget(x) — moves x in and does not drop it. The destructor
never runs; the value leaks. This is safe (leaking memory isn’t undefined
behavior in Rust), and it’s how you hand ownership to something that will clean
up later (FFI, ManuallyDrop, or intentional leaks like Box::leak).
let x = Noisy::new("leaked");
std::mem::forget(x);
// log is EMPTY — "drop leaked" never appears
mem::replace(&mut dst, new) — swaps new into the location behind a
mutable reference and returns the old value. This is the only way to move a
non-Copy value out of &mut self. You can’t write let v = self.field; — that
would move out of a borrow (E0507). You have to swap something in to take
something out:
impl Slot {
fn swap_in(&mut self, replacement: Noisy) -> Noisy {
std::mem::replace(&mut self.inner, replacement)
}
}
mem::take(&mut dst) is replace with Default::default() as the
replacement. It’s the idiomatic way to pull a value out of an Option, a
Vec, or anything with a sensible default.
The key insight: replace and take don’t drop anything. They relocate the
old value into your hands. You decide when (or whether) it drops.
RAII scope guard: the reason Drop exists
The killer application of Drop is tying a cleanup action to a scope. A
Guard owns a closure and runs it when dropped — no matter how the scope
exits:
struct Guard<F: FnOnce()> {
action: Option<F>,
}
impl<F: FnOnce()> Drop for Guard<F> {
fn drop(&mut self) {
if let Some(action) = self.action.take() {
action();
}
}
}
There’s a real puzzle here. drop() receives &mut self, but an FnOnce
closure must be called by value (consumed). You can’t move self.action
out of a mutable reference — that’s E0507 again. The solution is the rung-6
trick: store the closure in an Option<F> and .take() it (which is
mem::replace with None). Now you have an owned F you can call.
.cancel() disarms the guard: set self.action = None before the scope
ends, and drop() finds nothing to run.
impl<F: FnOnce()> Guard<F> {
fn cancel(mut self) {
self.action = None;
}
}
This is exactly how MutexGuard, File, scopeguard::defer!, and every
“undo on error” pattern works.
ManuallyDrop: suppressing the compiler’s destructor
ManuallyDrop<T> wraps a value and tells the compiler: do not drop this
automatically. The wrapped value will leak unless you explicitly call the
unsafe ManuallyDrop::drop(&mut md).
Why it exists: it’s the only way to override the fixed field-drop order.
Normally fields a, b drop in declaration order (a then b). With
ManuallyDrop, you take control:
struct Custom {
a: ManuallyDrop<Noisy>,
b: ManuallyDrop<Noisy>,
}
impl Drop for Custom {
fn drop(&mut self) {
// SAFETY: dropping each field exactly once, never used afterward.
unsafe {
ManuallyDrop::drop(&mut self.b); // b first
ManuallyDrop::drop(&mut self.a); // then a
}
}
}
This produces ["drop b", "drop a"] — the reverse of the default. The
unsafe is genuine: calling ManuallyDrop::drop twice on the same field is
undefined behavior (double free). You must uphold the invariant that each field
is dropped exactly once and never read afterward.
ManuallyDrop is also how Vec manages element drops internally — it wraps
its allocation in ManuallyDrop so it can drop elements one by one in its own
Drop impl, rather than relying on the compiler’s default.
Capstone: rollback-on-drop Transaction
The ladder’s synthesis rung combines everything into a pattern used by every database driver, every temp-file-unless-kept, every undo-on-error mechanism:
struct Transaction<'a> {
db: &'a mut Vec<String>,
added: usize,
committed: bool,
}
The pieces:
begin(db)— borrows the database mutably, starts with 0 rows added andcommitted: false.insert(row)— pushes the row ontodband incrementsadded.commit(mut self)— setsself.committed = true. Takesselfby value, so the guard is consumed anddrop()runs with the flag set.Drop— if!self.committed, popsself.addedrows back off and logs"rollback". If committed, does nothing.
impl Drop for Transaction<'_> {
fn drop(&mut self) {
if !self.committed {
for _ in 0..self.added {
self.db.pop();
}
log("rollback");
}
}
}
The committed field is a hand-written drop flag. commit() sets it to
true, disarming the rollback — exactly like Guard::cancel() from rung 7.
The difference: here the state mutation (the inserts) happens eagerly, and
rollback undoes it, whereas the guard defers the action entirely.
The critical test: rollback fires during panic unwinding too. A
catch_unwind around a panicking transaction proves the rows are rolled back
even on the exceptional path. This is the whole point of RAII — cleanup on
every exit, not just the happy path.
Footguns
-
Assuming locals and fields drop in the same order. They don’t. Locals are LIFO (reverse declaration); fields are FIFO (declaration order). Getting this wrong causes subtle resource-ordering bugs (e.g., dropping a lock guard before the data it protects).
-
Calling
x.drop()directly. The compiler forbids it (E0040). Usestd::mem::drop(x)instead — it movesxby value, so ownership transfer prevents double free. -
Forgetting that
mem::forgetis safe. It doesn’t cause UB, but it does leak. Any cleanup you rely on (flushing buffers, releasing locks, temp file deletion) is skipped. Code must be correct even ifDropnever runs — that’s whymem::forgetbeing safe is a design choice, not a bug. -
Moving out of
&mut selfindrop(). You can’t dolet f = self.field;becausedrop()only gets a mutable borrow. The workaround isOption::take()(which ismem::replacewithNone) to get an owned value you can consume. -
Double
ManuallyDrop::drop. Unlike everything else on this list, this is undefined behavior. Once you callManuallyDrop::drop(&mut md), the inner value is gone. Calling it again is a double free. There’s no compiler protection here — you’re inunsafeterritory.
Signatures to know
// The Drop trait — one method, &mut self, no return
trait Drop {
fn drop(&mut self);
}
// Free function: takes ownership, value dies at end
fn std::mem::drop<T>(x: T) {}
// Leak: takes ownership, destructor is skipped
fn std::mem::forget<T>(x: T) {}
// Swap a new value in, get the old one back
fn std::mem::replace<T>(dest: &mut T, src: T) -> T
// replace with Default::default()
fn std::mem::take<T: Default>(dest: &mut T) -> T
// Wrapper that suppresses automatic drop
struct ManuallyDrop<T> { /* ... */ }
impl<T> ManuallyDrop<T> {
fn new(value: T) -> Self;
unsafe fn drop(slot: &mut ManuallyDrop<T>);
}
Real-world patterns
| Pattern | Uses | Example |
|---|---|---|
| RAII guard | Drop runs cleanup on scope exit | MutexGuard unlocks, File closes, TempDir deletes |
| Commit/rollback | Drop flag disarms destructor on success | Database transactions, staged file writes |
| Scope guard with cancel | Option::take() in drop() | scopeguard crate, the Guard<F> from rung 7 |
| Custom field order | ManuallyDrop + unsafe drop | Vec dropping elements before freeing the allocation |
| Intentional leak | mem::forget / ManuallyDrop | Box::leak, handing ownership to FFI |
Move out of &mut | mem::replace / mem::take | Consuming an FnOnce stored behind a borrow |
Explain it back
- Why are locals dropped in reverse order but struct fields in declaration order?
- Why does
x.drop()produce a compiler error, and what do you use instead? - What is a drop flag, and when does the compiler insert one?
- Is
mem::forgetsafe? Why or why not — and what are the consequences? - How do you move a value out of
&mut selfinside adrop()implementation? - What happens if you call
ManuallyDrop::droptwice on the same field? - In the
Transactioncapstone, what happens ifcommit()is never called and the scope exits via panic? - Why does the
Guard’s action field need to beOption<F>rather than justF?
See also
- Cow — uses
mem::replaceinternally for theto_mut()upgrade - Borrow / ToOwned — the
MyCowcapstone also hits the “move out of enum variant” pattern
Lifetimes in depth
Ladder:
src/bin/lifetimes_depth.rs· Run:cargo run --bin lifetimes_depth· Phase 1 · 9 rungs
TL;DR
A lifetime label like 'a does not create, extend, or change the life of
anything. It is a name you attach to references so the compiler can check one
single rule:
A reference must never outlive the data it points to.
<'a> on a function is a generic parameter — just over a lifetime instead of a
type. Its whole job is to describe how an output borrow is connected to the
input borrows, so the compiler can prove the returned reference is still valid
at every call site. You are not telling the compiler how long things live; you
are telling it which borrows share a fate.
Why this exists (from first principles)
Rust has no garbage collector. A reference (&T) is just a pointer — it does not
keep the pointed-to data alive. So the danger is the dangling reference: a
pointer to memory that has already been freed. C and C++ let you write this and
hand you undefined behavior at runtime. Rust refuses to compile it.
To refuse it, the borrow checker needs to reason about spans of validity. Most of the time it figures these out on its own. But at function and struct boundaries the information is lost: a signature is a contract, and the compiler only sees the signature, not the body, when checking a call. Consider:
fn longest(a: &str, b: &str) -> &str { /* ... */ } // WRONG: won't compile
When some other code calls longest(&x, &y), how long is the returned &str
valid? It depends on whether the body returns a or b — but the caller can’t
see the body, and the compiler refuses to peek (that would make signatures
meaningless and changes to a body could silently break callers). So the signature
must carry the answer. Lifetimes are that missing piece of the contract:
fn longest<'a>(a: &'a str, b: &'a str) -> &'a str { // OK
if a.len() > b.len() { a } else { b }
}
Read it as: “for some lifetime 'a, both inputs are borrowed for at least 'a,
and the result is valid for 'a.” At a call site the compiler picks 'a to be
the overlap of the two input borrows, and guarantees the result doesn’t escape
that overlap. No runtime cost — this is all erased before codegen.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | foundations | longest | Annotate a fn returning one of two &str; tie inputs and output to one 'a. |
| 2 | mechanics | first_word / prefix_before | The 3 elision rules — and where they run out. |
| 3 | mechanics | Excerpt<'a> | A struct that holds a reference must declare a lifetime. |
| 4 | mechanics | impl<'a> methods | &self elision (rule 3) and the gotcha when the return can come from a param. |
| 5 | footguns | make_label / after_slash | The dangling-return error: owned vs borrowed-from-input. |
| 6 | footguns | or_default / store | 'a: 'b outlives bounds — when one lifetime must contain another. |
| 7 | patterns | longest_with_announcement / leak_label | Lifetimes + generics + trait bounds; &'static and Box::leak. |
| 8 | patterns | Words<'a> | A borrowing Iterator: why Item = &'a str, not tied to &mut self. |
| 9 | capstone | StrSplit<'h, 'd> | Zero-copy split with two independent lifetimes. |
The ideas, built up
1. One lifetime ties inputs to output (longest)
The annotation <'a>(a: &'a str, b: &'a str) -> &'a str does not mean “a
and b live equally long.” It means: the compiler will choose a single 'a that
is no longer than either input’s actual borrow, and promise the output lives
that long. The check that makes this sound:
let s1 = String::from("a long string");
let result;
{
let s2 = String::from("short");
result = longest(&s1, &s2); // 'a = the shorter of the two borrows = s2's scope
assert_eq!(result, "a long string");
} // s2 dropped here; using `result` past this point would be rejected
'a collapses to the smaller region (here s2’s scope), and the borrow
checker ties result to it. That is the entire mechanism: lifetimes are
constraints the compiler solves, not durations you assign.
2. Elision — the rules that let you omit lifetimes (first_word)
You rarely write 'a because the compiler applies three elision rules, in
order, to fill them in:
- Each elided input reference gets its own fresh lifetime.
- If there is exactly one input lifetime, it is assigned to all outputs.
- If one input is
&self/&mut self, its lifetime goes to all outputs.
If after these an output lifetime is still unknown, you get a hard error.
Rule 1 + rule 2 are why this needs no annotation — one input ref, so the output is unambiguous:
fn first_word(s: &str) -> &str { // elided to fn first_word<'a>(s: &'a str) -> &'a str
let first_space = s.find(' ').unwrap_or(s.len());
&s[..first_space]
}
Where elision runs out: two input refs, no &self. Rule 2 doesn’t apply,
rule 3 doesn’t apply, so the compiler cannot guess which input the output borrows
from:
fn prefix_before(text: &str, marker: &str) -> &str // WRONG: missing lifetime specifier
The fix is the real lesson — only annotate what flows to the output. Only
text is returned, so only text gets the named lifetime. Leave marker with
its own elided one:
fn prefix_before<'a>(text: &'a str, marker: &str) -> &'a str { // OK
let first = text.find(marker).unwrap_or(text.len());
&text[..first]
}
This is a habit worth keeping: tying marker to 'a too would over-constrain
every caller for no reason. Give the output’s source a name; let everything else
elide.
3. Structs that hold references (Excerpt<'a>)
A struct that stores a reference must declare a lifetime and tag the field:
struct Excerpt<'a> {
part: &'a str,
}
The new rule this buys: an Excerpt value may never outlive the str it
borrows from. The lifetime parameter makes the struct generic over how long
its borrow is good for, and the borrow checker enforces that the whole struct
dies before the borrowed data does.
The constructor uses '_, the inferred lifetime:
fn first_sentence(text: &str) -> Excerpt<'_> { // '_ = "infer it" -> tied to text by elision
let dot = text.find('.').unwrap_or(text.len());
let offset = if dot == text.len() { 0 } else { 1 };
Excerpt { part: &text[..dot + offset] }
}
Excerpt<'_> reads as “an Excerpt borrowing for some lifetime the compiler
will infer” — here, by elision, the lifetime of text. Writing Excerpt<'a>
with an explicit <'a> on the fn would mean the same thing.
4. Methods, and the &self gotcha (impl<'a>)
To add methods, you declare the lifetime after impl and use it on the
type — exactly like impl<T> Vec<T>:
impl<'a> Excerpt<'a> {
fn part(&self) -> &str { self.part } // rule 3: return tied to &self, no annotation needed
}
The gotcha appears the moment the return can come from a parameter instead of
self. Elision rule 3 has already tied the elided return to &self, so a
returned parameter is rejected — its lifetime is unrelated to &self:
// WRONG: returning `candidate` fails — its lifetime isn't tied to &self
fn longer_of(&self, candidate: &str) -> &str {
if self.part.len() > candidate.len() { self.part } else { candidate }
}
The fix is to give self.part (which is &'a str — actually &'b self here) and
candidate the same lifetime and return that:
fn longer_of<'b>(&'b self, candidate: &'b str) -> &'b str { // OK
if self.part.len() > candidate.len() { self.part } else { candidate }
}
Now both possible return sources provably share one lifetime, so either branch is
valid. The general principle: when a method’s return may come from a parameter,
elision’s default (tie to &self) is wrong — name a lifetime and share it.
5. The dangling-return footgun (make_label / after_slash)
This is the defining lifetime error. A function may only return a reference to data that outlives the call — i.e. data the caller owns. It cannot return a reference to a local, because the local is dropped at the closing brace and the reference would dangle:
// WRONG (E0515: cannot return reference to local variable `label`)
fn make_label(id: u32) -> &str {
let label = format!("item-{id}");
&label // label dies here; &label can't escape
}
There is no lifetime annotation that fixes this — the data genuinely doesn’t live long enough. The honest fix is to return owned data:
fn make_label(id: u32) -> String { // OK: hand back ownership
format!("item-{id}")
}
The contrast that makes it click: returning a slice of a parameter is fine, because the caller owns it and it outlives the call:
fn after_slash(haystack: &str) -> &str { // OK: result borrows the caller's data
match haystack.find('/') {
Some(i) => &haystack[i + 1..],
None => "",
}
}
Rule of thumb: you can borrow out of what was borrowed in; you cannot borrow out of what you made locally. If you made it, return it (move it out).
6. 'a: 'b outlives bounds (or_default / store)
Syntax: 'a: 'b is a bound meaning “'a outlives 'b” — 'a lasts at least
as long as 'b. It sits in the same slot as a trait bound T: Clone, but for
lifetimes. It is what lets the compiler treat a longer-lived &'a T as a
shorter &'b T (covariance, made explicit).
When you return the 'a reference where a 'b is promised, you must assert the
relationship or the compiler refuses (“lifetime ’a may not live long enough”):
fn or_default<'a: 'b, 'b>(primary: &'a str, fallback: &'b str) -> &'b str { // OK
if !primary.is_empty() { primary } else { fallback }
}
Where the bound is unavoidable — you can’t unify the two lifetimes because
they appear in different positions. store overwrites what a slot points at:
fn store<'a: 'b, 'b>(slot: &mut &'b str, value: &'a str) { // OK
*slot = value; // dropping a &'a into a &'b slot requires 'a to last >= 'b
}
The slot holds a &'b str, so anything written into it must live at least as
long as 'b. Without 'a: 'b the assignment is rejected. The lesson: when one
reference is stored where another lives, you often need to spell out which one
contains the other.
7. Lifetimes + generics + 'static (longest_with_announcement / leak_label)
Lifetime params and type params share one <...> list, lifetimes first. They
don’t interfere — 'a constrains borrows, T constrains a type:
fn longest_with_announcement<'a, T>(x: &'a str, y: &'a str, ann: T) -> &'a str
where
T: Display,
{
println!("Announcing: {}", ann);
if x.len() > y.len() { x } else { y }
}
'static is the lifetime that lasts the whole program. A &'static str borrows
data that never goes away — string literals are baked into the binary. Rung 5
gave one escape from “can’t borrow out a local” (return owned). Here is the
other one: deliberately leak the allocation so it lives forever, yielding a
genuine &'static str:
fn leak_label(id: u32) -> &'static str {
let label = format!("id-{}", id);
Box::leak(label.into_boxed_str()) // never freed -> valid for 'static
}
Caution:
Box::leakis a real, permanent memory leak. It’s the right tool for values that truly live for the program’s duration (config, interned strings), not a way to dodge the borrow checker in a loop.
Worth distinguishing: &'static T (“this reference is valid forever”) vs
T: 'static (“this type contains no borrow shorter than the program” — which
includes all owned types like String). They are not the same constraint.
8. A borrowing iterator (Words<'a>)
Words walks a string and yields each whitespace-separated word as a &str that
borrows the original string — exactly how str::split and slice::iter hand
out references without cloning. The crux is the Item lifetime:
struct Words<'a> { remainder: &'a str }
impl<'a> Iterator for Words<'a> {
type Item = &'a str; // borrows the STRING (lifetime 'a), NOT &mut self
fn next(&mut self) -> Option<Self::Item> {
let trimmed = self.remainder.trim_start_matches(' ');
if trimmed.is_empty() { return None; }
let word = match trimmed.find(' ') {
Some(i) => { self.remainder = &trimmed[i + 1..]; &trimmed[..i] }
None => { self.remainder = ""; trimmed }
};
Some(word)
}
}
Read the signature carefully. &mut self is borrowed only for the duration of
one next() call. But the &str we return points into the underlying string
(lifetime 'a), which lives much longer than a single call. So Item must
be &'a str, tied to the string — not to &mut self.
That choice is what makes this work:
let mut it = Words::new(&text);
let first = it.next().unwrap(); // holds a word...
let second = it.next().unwrap(); // ...across another next() call — compiles!
assert_eq!((first, second), ("the", "quick"));
If Item were tied to &mut self, first would borrow the iterator and the
second next() (another &mut) would be a borrow-conflict — you could never
keep a yielded item past the next iteration, and .collect::<Vec<&str>>() would
be impossible. Tie Item to the data, not the cursor.
9. Capstone: StrSplit with two lifetimes
Build your own "a,b,c".split(",") that never allocates — every yielded piece
is a &str slice into the original haystack. The structural insight is two
distinct lifetimes:
struct StrSplit<'haystack, 'delimiter> {
remainder: Option<&'haystack str>,
delimiter: &'delimiter str,
}
impl<'haystack, 'delimiter> Iterator for StrSplit<'haystack, 'delimiter> {
type Item = &'haystack str; // items borrow the HAYSTACK, never the delimiter
fn next(&mut self) -> Option<Self::Item> {
let remainder = self.remainder.as_mut()?; // None -> already exhausted
match remainder.find(self.delimiter) {
Some(i) => {
let current = *remainder;
let piece = ¤t[..i];
*remainder = ¤t[i + self.delimiter.len()..]; // skip the whole delimiter
Some(piece)
}
None => self.remainder.take(), // last piece; mark exhausted in one move
}
}
}
Why two lifetimes. The yielded pieces borrow 'haystack. They do not borrow
'delimiter — once we’ve located a delimiter we keep no reference to it in the
output. Keeping the lifetimes separate lets a caller pass a short-lived
delimiter (even a temporary that drops before the results are used) and still
keep the pieces:
let haystack = String::from("x-y-z");
let result: Vec<&str>;
{
let delim = String::from("-");
result = StrSplit::new(&haystack, &delim).collect();
} // delim dropped here — result is still valid, because pieces borrow haystack
assert_eq!(result, vec!["x", "y", "z"]);
Collapsing both into one lifetime would over-constrain every caller — the same lesson as rung 2b and rung 6, now at struct scale: give each borrow its own lifetime; only unify when the data flow actually demands it.
Why remainder: Option<&str> and not just &str. To distinguish “more to
yield, possibly an empty final field” from “fully exhausted.” Splitting "a," on
"," must yield ["a", ""] — a trailing empty piece — and then stop. The
Option lets next hand out that last "" once via .take() (which returns
Some(last) and sets the field to None), and return None forever after.
Footguns
- Thinking
'aextends a life. It never does. It only names a borrow so the compiler can relate borrows. The compiler picks'ato be the overlap of the inputs — the shortest region, not the longest. - Over-annotating. Tying every reference to one
'a(e.g.markerinprefix_before, or merging'haystack/'delimiter) compiles but needlessly restricts callers. Name only the borrow that flows to the output. - The
&selfreturn trap. Elision rule 3 ties an elided return to&self. If the method can return a parameter, you must introduce a shared lifetime — the default will reject the parameter branch. - Returning a reference to a local (E0515). No annotation fixes this; the data
dies at the brace. Return owned data, or
Box::leakfor genuine&'static. - Tying an iterator’s
Itemto&mut self. You then can’t hold an item across the nextnext()call, and.collect()breaks. TieItemto the underlying data’s lifetime instead. Box::leakas a borrow-checker dodge. It’s a permanent leak. Fine for program-lifetime data; a bug inside a loop or per-request path.
Real-world patterns
str::split/slice::iterare exactly theWords/StrSplitshape: zero-copy iterators whoseItemborrows the source, not the cursor.- Parsers and tokenizers lean on multi-lifetime structs so tokens can borrow the input buffer while transient state (delimiters, config) lives shorter.
Cow<'a, str>(see Cow) carries a lifetime for its borrowed variant — the samestruct holds a referencerule asExcerpt<'a>.'a: 'boutlives bounds show up whenever you store one reference into a place that already holds another (caches, slot updates, builder APIs).T: 'staticbounds are everywhere instd::thread::spawnand trait objects (Box<dyn Error + 'static>) — “this value carries no borrow that could dangle.”
Capstone insight
A reference-holding iterator has two clocks: the borrow of &mut self during
one next() call, and the borrow of the data it yields. The whole design hinges
on tying Item to the data clock, not the call clock. Generalize that to
StrSplit and you see lifetimes are a dependency graph between borrows: give
each borrow its own lifetime parameter, then add only the edges ('a: 'b, or
shared names) the data flow forces. Under-constrain and it won’t compile;
over-constrain and your callers suffer. Lifetimes are the language for drawing
exactly that graph — no more, no less.
Explain it back
- Why does
longestneed'abutfirst_worddoesn’t? Which elision rules coverfirst_word? - What does the compiler actually pick
'ato be at a call site — the longer or the shorter input borrow, and why? - Why does returning
candidatefromlonger_of(&self, candidate)fail under elision, and what’s the minimal fix? - Why can
after_slashreturn a&strbutmake_labelcannot? What are the two escape hatches? - What does
'a: 'bmean, and give a case where it’s unavoidable. - In
Words, why istype Item = &'a strand not tied to&mut self? What breaks if you get it wrong? - In
StrSplit, why two lifetimes instead of one, and why isremainderanOption? &'static strvsT: 'static— what’s the difference?
See also
- Cow — Clone-on-Write — a lifetime-carrying enum over borrowed/owned.
- Borrow / ToOwned — borrowed-vs-owned views, the
Cowloop. - Drop & Ordering — when the data a reference points at dies.
HRTB — for<'a>
Ladder:
src/bin/hrtb.rs· Run:cargo run --bin hrtb· Phase 1 · 9 rungs
TL;DR
A higher-ranked trait bound moves the quantifier on a lifetime. The two bounds look almost identical but mean opposite things:
fn f<'a, F: Fn(&'a str)>(g: F) // the CALLER picks one 'a; bound holds for THAT one
fn f<F: for<'a> Fn(&'a str)>(g: F) // bound holds for EVERY 'a; the CALLEE picks fresh per call
for<'a> reads literally as “for all lifetimes 'a”. You need it whenever a
value — almost always a closure or a trait impl — must work on a borrow whose
lifetime doesn’t exist yet at the point you write the bound: a reference you’ll
create inside your function and hand off, where the caller could never name its
lifetime.
You have used HRTB for years without seeing it: Fn(&str) -> &str already
desugars to for<'a> Fn(&'a str) -> &'a str. Elision writes the quantifier for
you almost everywhere. This ladder is about the handful of places where it can’t,
and where reading the explicit form is the only way to understand the error.
Why this exists (from first principles)
Start from the lifetime contract. A normal generic lifetime is chosen by the caller, at the call site:
fn longest<'a>(a: &'a str, b: &'a str) -> &'a str { /* ... */ }
When some code calls longest(&x, &y), the compiler picks 'a to be the overlap
of those two specific borrows. 'a is one concrete region, fixed once, from the
outside.
Now flip the direction. Suppose you are the one who will create the reference, deep inside your own function, and you want to accept a callback that operates on it:
fn run_on_local<F>(f: F) -> usize
where
F: Fn(&str) -> &str, // what lifetime is this &str?
{
let s = String::from("hello world"); // born HERE, inside the function
f(&s).len() // f operates on a borrow the caller can't see
}
The borrow &s has a lifetime that lasts only until the closing brace. The caller
of run_on_local has no way to name it — it doesn’t exist at the call site. So a
caller-chosen <'a> is fundamentally the wrong tool: there is no single 'a the
caller could supply that would cover a string born after they called you.
The fix is to demand that f works for all lifetimes, so it certainly works
for the private one you’ll mint internally. That demand is for<'a>:
where F: for<'a> Fn(&'a str) -> &'a str
The mental model in one line: a plain
<'a>is a lifetime the caller fills in;for<'a>is a lifetime the callee fills in, freshly, every time it uses the value.
This is not an exotic corner. It is why Fn traits are defined with it implicitly,
why serde’s DeserializeOwned exists, and why every parser-combinator library in
Rust can compose at all. The bound is the load-bearing wall; you just rarely see it
because elision plasters over it.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | foundations | apply_to_each | Fn(&str) already is for<'a> Fn(&'a str) — feed it borrows of many lifetimes. |
| 2 | foundations | apply_to_each_explicit | Spell the quantifier out; the elided and explicit forms are identical. |
| 3 | mechanics | measure_on_local | Return a borrow of the closure’s arg; the caller can’t name that lifetime. |
| 4 | mechanics | Slicer<'a> / run_slicer | HRTB works on your own lifetime-generic trait, not just Fn. |
| 5 | footgun | apply_str | “implementation of Fn is not general enough”: let-bound closures get one lifetime. |
| 6 | footgun | sum_two_locals | The named-lifetime trap: one <'a> is fixed by the caller and shared by every call. |
| 7 | real-world | DecodeOwned | DeserializeOwned: for<'de> Deserialize<'de> — owners qualify, borrowers don’t. |
| 8 | real-world | StrPipeline | Box<dyn for<'a> Fn(&'a str) -> &'a str> keeps the trait object lifetime-free. |
| 9 | capstone | Parser<T> | Parser combinators stand entirely on for<'i> Fn(&'i str) -> Option<(&'i str, T)>. |
The ideas, built up
1. The quantifier is already there (apply_to_each)
The first surprise is that you have been writing HRTB all along. This bound has no named lifetime at all:
fn apply_to_each<F>(items: &[String], f: F)
where
F: Fn(&str), // implicitly: for<'a> Fn(&'a str)
{
for item in items {
f(item); // each &str lives only for this iteration
}
}
Inside the loop, each item is borrowed for the span of one iteration. The closure
must accept a &str of whatever lifetime each iteration produces — and it does,
because Fn(&str) secretly means for<'a> Fn(&'a str): “works for every input
lifetime.” The check feeds it borrows that only live one loop turn, and a closure
capturing a RefCell records their lengths. Nothing forces you to think about
lifetimes here precisely because the quantifier was inserted for you.
2. Spell it out (apply_to_each_explicit)
for<'a> is a real slot in the grammar — it sits immediately before the trait name
and introduces a lifetime scoped to that one bound:
fn apply_to_each_explicit<F>(items: &[String], f: F)
where
F: for<'a> Fn(&'a str), // identical to Fn(&str) above
{ /* same body */ }
The 'a here is not a generic parameter of the function — notice it does not
appear in <F>. It is bound by the trait bound itself. That scoping is the whole
point: it is a lifetime the function body gets to instantiate, not one the caller
supplies. The elided and explicit forms compile to exactly the same thing.
3. Returning a borrow forces the issue (measure_on_local)
Rung 1 took a borrow; this one returns one, which is where the caller-vs-callee distinction becomes load-bearing:
fn measure_on_local<F>(f: F) -> usize
where
F: for<'a> Fn(&'a str) -> &'a str, // same 'a in and out: output welded to input
{
let s = String::from("hello world");
let result = f(&s); // &s lives only inside this function
result.len()
}
Two things to read carefully:
- The
-> &'a strreuses the same'aas the input. That is what makes returning a borrow sound: the output is allowed to borrow the input and nothing else, so it can’t outlive it. sis born and dies insidemeasure_on_local. The lifetime of&sis private to this call. The caller cannot name it. So the bound must be higher-ranked —fhas to promise it works for every lifetime, including this internal one.
The closures in the check pass inline, which matters (see rung 5):
let n = measure_on_local(|s: &str| s.split(' ').next().unwrap_or("")); // first word
assert_eq!(n, 5); // "hello"
assert_eq!(measure_on_local(|s: &str| s), 11); // identity -> "hello world"
4. HRTB on your own trait (Slicer<'a>)
for<'a> is not special to Fn. It applies to any trait with a lifetime
parameter:
trait Slicer<'a> {
fn slice(&self, input: &'a str) -> &'a str;
}
struct FirstWord;
impl<'a> Slicer<'a> for FirstWord {
fn slice(&self, input: &'a str) -> &'a str {
input.split(' ').next().unwrap_or("")
}
}
The key reframe: Slicer<'a> is not one trait, it’s a family of traits — one per
lifetime. Writing impl<'a> Slicer<'a> for FirstWord implements every member of
that family in a single stroke. And the bound that asks for the whole family is
exactly for<'a> Slicer<'a>:
fn run_slicer<S>(s: S) -> usize
where
S: for<'a> Slicer<'a>, // S implements Slicer<'a> for ALL 'a
{
let word = String::from("green eggs");
s.slice(&word).len() // &word is local -> needs the "for all 'a" guarantee
}
The ladder scaffolds this with a deliberately wrong placeholder bound
(S: Slicer<'static>) that compiles only while the body is todo!(). The moment
you write s.slice(&word) on a local, the compiler rejects 'static and forces
you to generalize to for<'a> Slicer<'a>. The error is the lesson.
5. “implementation of Fn is not general enough” (apply_str)
This is the single most-cursed HRTB error, and it has a precise cause. In rung 3
the closures worked because they were passed inline. Factor a
reference-returning closure into a let binding and it breaks:
fn apply_str<F>(f: F) -> usize
where
F: for<'a> Fn(&'a str) -> &'a str,
{
let s = String::from("scaffold");
f(&s).len()
}
// WRONG: "implementation of `Fn` is not general enough"
let bad = |s: &str| s;
apply_str(bad);
Why does the identical closure fail when named? When a closure is bound to a let
without a guiding context, type inference picks one concrete lifetime for its
signature. A closure inferred as Fn(&'0 str) -> &'0 str for some specific '0
does not satisfy for<'a> — it is general over one lifetime, not all of them.
When passed inline, the expected higher-ranked type propagates into inference and
the closure is inferred higher-ranked from the start.
The fixes, and why they work:
// OK (i): fn-pointer coercion — fn pointers are inherently for<'a>
let good: fn(&str) -> &str = |s| s;
apply_str(good);
// OK (ii): a real fn item — fn items are inherently for<'a> too
fn id(s: &str) -> &str { s }
let good = id;
apply_str(good);
The rule to remember: only closures get a single inferred lifetime that can break HRTB. Function pointers (
fn(&str) -> &str) and namedfnitems are always higher-ranked. Passing a closure inline usually also works, because the expected type guides inference.
6. The named-lifetime trap (sum_two_locals)
Rung 5 was a closure that wasn’t general enough. This is the dual: a bound you wrote that isn’t general enough, because you reached for a single named lifetime where you needed a higher-ranked one.
// WRONG: one named 'a, chosen by the caller, shared across every use of f
fn sum_two_locals<'a, F>(f: F) -> usize
where
F: Fn(&'a str) -> &'a str,
{
let s1 = String::from("ab");
let s2 = String::from("cdef");
f(&s1).len() + f(&s2).len() // ERROR: borrowed value does not live long enough
}
The 'a in <'a, F> is a free parameter chosen by the caller, so it must outlive
the entire function body. But s1 and s2 are locals that die inside it. One
fixed 'a cannot cover either — let alone two borrows in different inner scopes.
The fix is to make each call mint its own lifetime:
fn sum_two_locals<F>(f: F) -> usize
where
F: for<'a> Fn(&'a str) -> &'a str, // OK: callee picks a fresh, short 'a per call
{
let s1 = String::from("ab");
let s2 = String::from("cdef");
f(&s1).len() + f(&s2).len() // each call gets its own 'a
}
The distinction this rung adds over rung 3: a single <'a> isn’t merely
“caller-chosen”, it is one lifetime shared by every call to f. for<'a>
gives each call site its own. Two locals in two scopes makes that concrete — no
single 'a fits both, but “for all 'a” fits each.
7. DecodeOwned = for<'de> Decode<'de> (the serde pattern)
Now the payoff: a higher-ranked bound doing real work in the most-used crate in the
ecosystem. serde has two traits:
pub trait Deserialize<'de> { /* may BORROW from the input (zero-copy) */ }
pub trait DeserializeOwned: for<'de> Deserialize<'de> {}
impl<T> DeserializeOwned for T where T: for<'de> Deserialize<'de> {}
DeserializeOwned is defined as “can be deserialized from input of any lifetime.”
The ladder builds the miniature, where Decode<'a> plays the role of
Deserialize<'de>:
trait Decode<'a>: Sized {
fn decode(input: &'a str) -> Option<Self>;
}
trait DecodeOwned: for<'a> Decode<'a> {}
impl<T> DecodeOwned for T where T: for<'a> Decode<'a> {}
The two contrasting impls are the whole point:
// BORROWS from input -> Decode<'a> only for the ONE matching 'a
struct Borrowed<'a> { first: &'a str }
impl<'a> Decode<'a> for Borrowed<'a> {
fn decode(input: &'a str) -> Option<Self> {
input.split(',').next().map(|first| Borrowed { first })
}
}
// OWNS its data -> Decode<'a> for EVERY 'a
struct Owned { sum: u32 }
impl<'a> Decode<'a> for Owned {
fn decode(input: &'a str) -> Option<Self> { /* parse + sum the csv */ }
}
Borrowed<'a> ties Self to the input lifetime, so it implements Decode<'a> for
exactly one 'a — it is not for<'a> Decode<'a>, therefore not
DecodeOwned. Owned keeps a u32 that borrows nothing, so it implements
Decode<'a> for all 'a and is DecodeOwned.
That distinction is enforced the moment you try to load from data you own internally:
fn load<T: DecodeOwned>(source: String) -> Option<T> {
T::decode(&source) // &source is local -> only a DecodeOwned T can be loaded this way
}
let got: Owned = load("1,2,3,4".to_string()).unwrap(); // OK: sum == 10
// let _: Borrowed = load("a,b".to_string()).unwrap(); // ERROR: Borrowed: DecodeOwned not satisfied
This is precisely why serde_json::from_reader requires DeserializeOwned and
won’t deserialize a struct holding &str: the bytes are owned by the reader and
dropped when it returns, so anything that borrows them would dangle. The for<'a>
bound is what mechanically excludes the borrowing types.
8. HRTB inside a trait object (StrPipeline)
Up to here the higher-ranked thing was a generic parameter F. You can also erase
it behind dyn. Putting for<'a> inside the box is what lets the surrounding
type carry no lifetime parameter:
struct StrPipeline {
steps: Vec<Box<dyn for<'a> Fn(&'a str) -> &'a str>>,
}
Because each boxed step is higher-ranked, one StrPipeline value can be applied to
inputs of any lifetime — the struct itself stays lifetime-free:
impl StrPipeline {
fn add<F>(mut self, f: F) -> Self
where
F: for<'a> Fn(&'a str) -> &'a str + 'static,
{
self.steps.push(Box::new(f));
self
}
fn run<'a>(&self, input: &'a str) -> &'a str {
let mut cur = input;
for step in &self.steps {
cur = step(cur); // &'a str in, &'a str out — same 'a, every iteration
}
cur
}
}
Watch the loop body type-check: cur is &'a str; each step is
for<'a> Fn(&'a str) -> &'a str, so step(cur) returns &'a str — the same 'a —
and the reassignment holds across every iteration.
The contrast that explains the design: if the box were
Box<dyn Fn(&'x str) -> &'x str> for some fixed 'x, the struct would need an
<'x> parameter and could only ever process borrows of that one lifetime. HRTB is
what keeps StrPipeline a plain, storable, lifetime-free type while one value still
serves a 'static string literal and a short-lived local alike.
9. Capstone: a parser combinator (Parser<T>)
A parser is a function: given input, either fail, or return (remaining input, value). The remaining slice is a sub-borrow of the input, so a parser is fundamentally:
for<'i> Fn(&'i str) -> Option<(&'i str, T)>
HRTB is the load-bearing wall of every parser-combinator library — nom, winnow,
chumsky. It’s what lets Parser<T> be a lifetime-free type you can store, pass, and
compose, while each parser still runs on input of any lifetime, and one parser’s
leftover slice feeds straight into the next:
struct Parser<T>(Box<dyn for<'i> Fn(&'i str) -> Option<(&'i str, T)>>);
impl<T: 'static> Parser<T> {
fn new(f: impl for<'i> Fn(&'i str) -> Option<(&'i str, T)> + 'static) -> Self {
Parser(Box::new(f))
}
fn parse<'i>(&self, input: &'i str) -> Option<(&'i str, T)> {
(self.0)(input)
}
}
Notice Parser<T> has no lifetime parameter — the for<'i> lives inside the
box, exactly as in rung 8. The base parser tag matches a literal prefix and yields
it; its value type is &'static str, which never borrows from the input lifetime:
fn tag(prefix: &'static str) -> Parser<&'static str> {
Parser::new(move |input: &str| input.strip_prefix(prefix).map(|rest| (rest, prefix)))
}
The combinators build on it. number parses a leading digit run; map transforms a
parser’s output; and then — the heart of the rung — runs one parser and feeds its
leftover into the next:
fn then<A: 'static, B: 'static>(a: Parser<A>, b: Parser<B>) -> Parser<(A, B)> {
Parser::new(move |input| {
a.parse(input)
.and_then(|(rest, a)| b.parse(rest).map(|(rest, b)| (rest, (a, b))))
})
}
The composition only type-checks because of for<'i>. a.parse(input) hands back
rest: &'i str — a sub-slice of input, lifetime 'i. You then call
b.parse(rest), and only because b is for<'i> can it accept that leftover slice
of the very same 'i. A single-lifetime parser type could not chain to arbitrary
depth without threading an explicit lifetime through every combinator. HRTB makes
the lifetime disappear from the type while staying correct in the body:
let assignment = then(tag("x="), number());
let (rest, (key, value)) = assignment.parse("x=42;").unwrap();
assert_eq!((key, value, rest), ("x=", 42, ";"));
let incremented = map(then(tag("n:"), number()), |(_, n)| n + 1);
assert_eq!(incremented.parse("n:99").unwrap().1, 100);
Footguns
- Thinking
for<'a>is just a fancy<'a>. It is the opposite quantifier.<'a>= caller picks one;for<'a>= holds for all, callee picks per call. let-bound reference-returning closures. They infer one concrete lifetime and failfor<'a>with “implementation ofFnis not general enough.” Pass inline, coerce to afnpointer, or use a namedfnitem.- Reaching for a named
<'a>to accept a callback over locals. The'abecomes caller-fixed and must outlive the whole function; your locals can’t satisfy it. Usefor<'a>so each call mints its own. - Expecting a borrowing type to be
DeserializeOwned/DecodeOwned. A type whoseSelfborrows the input implements the trait for only one lifetime, so thefor<'de>supertrait excludes it. Own your data, or thread the input lifetime. - Adding a lifetime parameter to a struct that stores a callback. If the boxed
Fnis higher-ranked (Box<dyn for<'a> Fn(&'a str) -> &'a str>), the struct needs no lifetime at all. Only a fixed inner lifetime forces one onto the struct.
Real-world patterns
Fn/FnMut/FnOnceover references are all implicitly higher-ranked.Fn(&T) -> &Uisfor<'a> Fn(&'a T) -> &'a U; you only type the quantifier when elision can’t infer it.serde::de::DeserializeOwnedis the canonical HRTB supertrait (for<'de> Deserialize<'de>) — and the reasonfrom_reader/from_slice-into-owned reject borrowing types.- Parser combinators (
nom,winnow,chumsky) are built bottom-to-top onfor<'i> Fn(&'i str) -> ..., keeping theirParsertypes lifetime-free. - Closure-accepting APIs that borrow internal state — iterator adapters,
visitor/callback registries, middleware chains — lean on the implicit
for<'a>so one callback can be invoked on transient internal borrows.
Capstone insight
HRTB is the language’s answer to a quantifier-ordering problem. Ordinary generics
put the caller’s choices outside the function: <'a> is decided at the call site.
But a callback or impl frequently has to operate on data the function makes for
itself, after the call has begun — data whose lifetime is logically inside the
function. for<'a> moves the lifetime’s binder from the outside (caller-chosen,
fixed) to the inside (callee-chosen, fresh each use). Once you see it as “who gets to
fill in this lifetime, and when,” every symptom follows: the let-bound closure that’s
“not general enough” (it committed to one lifetime too early), the named-<'a> that
rejects your locals (it’s fixed from outside), DeserializeOwned (owns its data, so
it qualifies for every lifetime), and the parser combinator that composes without a
lifetime in sight (the binder hides inside each dyn). HRTB is how you write “works
for any borrow I’ll ever hand it” — and most of the time, elision writes it for you.
Explain it back
- What is the difference in who chooses the lifetime between
<'a, F: Fn(&'a str)>and<F: for<'a> Fn(&'a str)>? - Why does
Fn(&str) -> &strneed no annotation, and what does it desugar to? - In
measure_on_local, why can’t the bound be a plain caller-chosen<'a>? - You get “implementation of
Fnis not general enough” from alet-bound closure. What exactly went wrong, and what are three ways to fix it? - Why does a single named
<'a>reject two locals insum_two_locals, whenfor<'a>accepts them? - Why is
OwnedaDecodeOwnedbutBorrowed<'a>is not? Relate it to whyserde_json::from_readerneedsDeserializeOwned. - Why does
Box<dyn for<'a> Fn(&'a str) -> &'a str>letStrPipelineavoid a lifetime parameter, where a fixed'xwould not? - In
then, which line relies onbbeing higher-ranked, and what would break without it?
See also
- Lifetimes in depth — caller-chosen
<'a>, elision, and'a: 'bbounds; HRTB is the next quantifier up. - Conversion traits —
From/TryFromand trait families, the same “one impl, many instantiations” shape asSlicer<'a>. - Cow — Clone-on-Write — the borrowed-vs-owned split that
DecodeOwnedturns into a trait bound.
Rc / Arc
Ladder:
src/bin/rc_arc.rs· Run:cargo run --bin rc_arc· Phase 1 · 9 rungs
TL;DR
Rc<T> is shared ownership by counting. One heap allocation holds your
value plus a counter; every Rc handle is a pointer to that allocation and
owns one unit of the count. clone() bumps the count (cheap — it copies a
pointer, never the data); drop decrements it; when the count hits 0 the
value is freed exactly once. That’s the entire machine. Rc only ever hands out
&T (shared, immutable access), which is what makes the counting sound. Arc
is the same machine with an atomic counter, so it can be shared across
threads; Rc uses a plain integer and is therefore single-threaded only. The
two failure modes to internalize: Rc gives you aliasing but not mutation
(reach for make_mut or RefCell), and a strong reference cycle leaks
because the counts never reach 0.
Why this exists (from first principles)
Rust’s default ownership is a tree: each value has exactly one owner, and
when that owner goes out of scope the value is freed. Box<T> is the canonical
single-owner heap pointer. This is wonderful — it makes “when is this freed?”
decidable at compile time with zero runtime bookkeeping — but it can’t express
every shape.
Some data is a DAG or a graph: one node reachable from two parents, a value several structs all need to keep alive, a string tag shared by thousands of records. There is no single, statically-known owner. So the question “when is this freed?” can’t be answered at compile time. You need to answer it at runtime, and the simplest correct answer is: free it when the last user is gone. That requires counting users.
That is precisely Rc:
| Approach | Owners | Freed when | Cost |
|---|---|---|---|
T (move) | exactly 1 | owner scope ends | none |
Box<T> | exactly 1 (heap) | owner scope ends | one allocation |
Rc<T> | many | last handle dropped | allocation + a counter, bumped per clone/drop |
What the compiler still guarantees, even with shared ownership: no
use-after-free (the value lives as long as any handle does) and no double-free
(only the 0-transition frees). What it gives up: it can no longer prove the
value is uniquely owned, so it refuses to hand out &mut T through an Rc.
That single restriction — shared access only — is the source of everything
interesting in this ladder.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | foundations | two owners | Rc::new + clone() -> two handles, one allocation; Rc::ptr_eq proves it |
| 2 | foundations | the count moves | strong_count rises on clone, falls on scope-end drop; you can watch [1,2,3,1] |
| 3 | mechanics | shared diamond | one node owned by two parents — the shape Box cannot express |
| 4 | mechanics | Rc<str> | intern an immutable string once; N records share one allocation via cheap clones |
| 5 | mechanics | make_mut | clone-on-write: mutate in place when sole owner, copy when shared |
| 6 | footgun | the cycle leak | a <-> b strong cycle: counts never hit 0, Drop never runs, memory leaks |
| 7 | footgun -> fix | Weak breaks it | own down with Rc, point back with Weak; downgrade / upgrade |
| 8 | real-world | Rc is !Send | atomic Arc crosses threads; Arc<Mutex<T>> for shared mutation |
| 9 | capstone | MyRc<T> | build it from scratch: NonNull + Cell<usize> count, last drop frees once |
The ideas, built up
Two owners, one allocation
The foundational move is just new then clone:
fn two_owners(text: &str) -> (Rc<String>, Rc<String>) {
let rc = Rc::new(text.to_string());
(rc.clone(), rc.clone())
}
The original rc is moved out by the time the tuple is built (both elements are
clones), so we return two handles to the same String. The proof is not
that the values are equal — it’s that they share an address:
let (a, b) = two_owners("shared");
assert_eq!(*a, "shared");
assert_eq!(*b, "shared");
assert!(Rc::ptr_eq(&a, &b)); // SAME allocation, not two copies
Rc::ptr_eq compares the raw pointer inside each handle. This is the literal
meaning of shared ownership: not two equal Strings, but two pointers to one
String. clone() here copied 16 bytes of pointer + length + capacity… no,
it copied a single pointer-to-the-Rc-allocation and incremented a counter.
The heap String and its bytes were never touched.
The count is the whole machine
Rc’s entire correctness rests on one number: strong_count. Rung 2 makes it
observable by sampling it at four moments:
fn count_lifecycle(rc: &Rc<String>) -> [usize; 4] {
let a = Rc::strong_count(rc); // 1: just the original
let (b, c) = {
let _rc2 = Rc::clone(rc);
let b = Rc::strong_count(rc); // 2: one clone alive
let _rc3 = Rc::clone(rc);
let c = Rc::strong_count(rc); // 3: two clones alive
(b, c)
}; // _rc2, _rc3 drop here
let d = Rc::strong_count(rc); // 1: back to just the original
[a, b, c, d]
}
The result is [1, 2, 3, 1]. clone() increments; the end of the inner scope
runs the Drop for _rc2 and _rc3, each decrementing. Note the function
takes &Rc<String> — a borrow of a handle, which does not add an owner.
Only clone() does. This distinction (borrowing a handle vs. cloning it) is
worth burning in: passing &Rc lets you read the value or the count without
participating in ownership.
The shared diamond — the shape Box can’t make
This is why Rc exists, drawn out:
top
/ \
left right
\ /
shared <- ONE node, owned by BOTH left and right
With Box, shared would need a single owner — left or right, not both.
Rc lets both branches hold a handle to the same node:
struct Node { name: String, children: Vec<Rc<Node>> }
let shared = Rc::new(Node { name: "shared".into(), children: vec![] });
let left = Rc::new(Node { name: "left".into(), children: vec![Rc::clone(&shared)] });
let right = Rc::new(Node { name: "right".into(), children: vec![Rc::clone(&shared)] });
After building it, the shared node’s strong_count is 2 (held by left’s
and right’s children vectors), and the two paths to it are pointer-equal:
assert!(Rc::ptr_eq(shared_via_left, shared_via_right));
assert_eq!(Rc::strong_count(shared_via_left), 2);
This is a DAG. As long as you only ever follow edges downward (parent to
child), the counts behave and everything frees when the roots go. The moment you
add an edge back upward with a strong Rc, you get rung 6’s leak.
Rc<str> — interning an immutable string the cheap way
Rc<T> shines when T is large and immutable and shared widely. The classic
case: thousands of records all tagged "electronics". Storing a String in
each is one heap allocation per record. Instead, allocate the string once
as Rc<str> and hand each record a clone:
fn tag_all(category: &str, n: usize) -> Vec<Rc<str>> {
let rc: Rc<str> = Rc::from(category); // ONE allocation of the bytes
let mut tags = Vec::with_capacity(n);
for _ in 0..n {
tags.push(Rc::clone(&rc)); // each push: pointer copy + count bump
}
tags
}
All n elements are the same allocation:
let tags = tag_all("electronics", 4);
for t in &tags[1..] {
assert!(Rc::ptr_eq(&tags[0], t)); // every tag clones the SAME Rc<str>
}
assert_eq!(Rc::strong_count(&tags[0]), 4); // the count sees all four
Rc<str>vsRc<String>.Rc<String>is a double indirection:Rc->String(ptr/len/cap on the heap) -> the bytes.Rc<str>stores the length in theRc’s fat pointer and points directly at the bytes — one indirection, noStringheader. For an immutable shared string,Rc<str>is the leaner choice. Build it withRc::from(&str)or.into(). The same logic givesRc<[T]>for shared immutable slices.
make_mut — clone-on-write through a shared handle
Rc won’t give you &mut T directly, because while other handles exist a
mutation would be visible through them and break aliasing. Rc::make_mut
resolves this by checking the count first:
fn push_isolated(rc: &mut Rc<Vec<i32>>, value: i32) {
Rc::make_mut(rc).push(value);
}
- Sole owner (
count == 1): hands you&mut Tto the existing allocation — mutate in place, no copy. - Shared (
count > 1): clones the innerTinto a fresh allocation, points thisRcat the clone, and gives you&mutto that. The other owners keep seeing the original. This is the “write” half of copy-on-write.
The ladder proves both branches. Sole owner mutates in place — same address before and after:
let mut solo = Rc::new(vec![1, 2, 3]);
let addr_before = Rc::as_ptr(&solo);
push_isolated(&mut solo, 4);
assert_eq!(Rc::as_ptr(&solo), addr_before); // no reallocation
Shared owner forces a copy that isolates the writer:
let original = Rc::new(vec![1, 2, 3]);
let mut writer = Rc::clone(&original); // count == 2
push_isolated(&mut writer, 99);
assert_eq!(*writer, vec![1, 2, 3, 99]); // writer sees its push
assert_eq!(*original, vec![1, 2, 3]); // original UNCHANGED
assert!(!Rc::ptr_eq(&original, &writer)); // writer points at a fresh clone
assert_eq!(Rc::strong_count(&original), 1); // the split made each sole again
assert_eq!(Rc::strong_count(&writer), 1);
This is exactly the Cow mental model, but the “am I shared?” test is the
refcount rather than an explicit enum tag. It’s how Rc::make_mut and friends
power cheap, structural-sharing-friendly data structures.
The reference cycle that leaks — the defining Rc failure
Rc frees its value when strong_count reaches 0. So what if two nodes hold
strong handles to each other?
struct Cycle { name: &'static str, link: RefCell<Option<Rc<Cycle>>> }
fn make_leaky_cycle() {
let a = Rc::new(Cycle::new("a"));
let b = Rc::new(Cycle::new("b"));
a.link.borrow_mut().replace(Rc::clone(&b)); // a -> b (strong)
b.link.borrow_mut().replace(Rc::clone(&a)); // b -> a (strong)
} // a and b go out of scope here
(The RefCell is only there because the back-edge must be wired after both
nodes exist — you need interior mutability to mutate a once it’s already in an
Rc.)
Walk the counts. After wiring, a has 2 strong owners (the local a + b’s
link); same for b. When the function returns, the locals a and b drop —
each count falls from 2 to 1, never to 0, because each node’s link still
holds the other. Neither Drop ever fires:
let drops = DROP_COUNT.with(|c| c.get());
assert_eq!(drops, 0, "expected the cycle to LEAK (0 drops)");
This is safe code. Rust guarantees no use-after-free and no double-free — it
does not guarantee no leaks. An Rc cycle is the single-threaded equivalent
of an object graph that’s unreachable but uncollected: the memory is gone for
the rest of the program.
Weak breaks the cycle — the parent/child tree
The fix is Weak<T>: a handle that points at the allocation and bumps the
weak count, but never the strong count. Because it doesn’t touch the
strong count, a Weak can’t keep a value alive, so a chain of weak edges can’t
form a keep-alive cycle. To use one you must upgrade() it — which returns
Option<Rc<T>>, Some if the target is still alive, None if it’s gone.
The ownership rule that makes graphs leak-free:
The direction that owns uses
Rc(strong). The direction that merely refers back usesWeak.
In a tree: parent -> child is strong (the parent owns its children); child -> parent is weak (a child can navigate up but must not pin its parent alive).
struct TreeNode {
name: &'static str,
parent: RefCell<Weak<TreeNode>>, // weak: does NOT own
children: RefCell<Vec<Rc<TreeNode>>>, // strong: owns
}
fn link_parent_child(parent: &Rc<TreeNode>, child: &Rc<TreeNode>) {
parent.children.borrow_mut().push(Rc::clone(child)); // strong down
*child.parent.borrow_mut() = Rc::downgrade(parent); // weak up
}
fn parent_name(child: &Rc<TreeNode>) -> &'static str {
child.parent.borrow().upgrade()
.map(|p| p.name)
.unwrap_or("<no parent>") // None if the parent is gone
}
The counts confirm the weak edge is free:
assert_eq!(Rc::strong_count(&root), 1); // ONLY the `root` binding owns it
assert_eq!(Rc::strong_count(&leaf), 2); // `leaf` binding + root.children
And the payoff — dropping the parent actually frees it, and the child’s weak pointer correctly reports the parent is gone:
drop(root);
assert_eq!(parent_name(&leaf), "<no parent>"); // upgrade() now returns None
When both nodes leave scope, both Drops run (the test asserts 2 drops) —
no leak, unlike rung 6. Rc::downgrade(&rc) makes a Weak from an Rc;
weak.upgrade() tries to promote it back, succeeding only while a strong owner
remains.
Rc is !Send -> Arc across threads
Rc’s counter is a plain usize. If two threads cloned/dropped the same Rc
concurrently, their increments and decrements could interleave and corrupt the
count — leading to a double-free or a leak. Rust forbids this at compile
time by making Rc: !Send: you literally cannot move one into another thread.
// WRONG — won't compile:
// let data = Rc::new(0);
// thread::spawn(move || { let _ = data; });
// error: `Rc<i32>` cannot be sent between threads safely
Arc (“atomic Rc”) is the same machine with an atomic counter. The atomic
increment/decrement is safe under contention, so Arc is Send + Sync and
crosses threads. But Arc, like Rc, still only gives shared access — to
mutate shared state across threads you wrap the data in a lock: Arc<Mutex<T>>.
Arc shares the lock; the Mutex hands out &mut T to one thread at a time.
fn concurrent_count(n_threads: usize, per_thread: usize) -> usize {
let counter = Arc::new(Mutex::new(0usize));
let handles = (0..n_threads).map(|_| {
let counter = Arc::clone(&counter); // each thread gets its own handle
thread::spawn(move || {
let mut counter = counter.lock().unwrap();
*counter += per_thread;
})
}).collect::<Vec<_>>();
for h in handles { h.join().unwrap(); }
*counter.lock().unwrap()
}
assert_eq!(concurrent_count(8, 10_000), 80_000); // no lost updates
Two different counters.
Arc’s atomic counter protects the reference count (how many handles exist). TheMutexprotects the data. Atomicity of the refcount does not make the inner value thread-safe to mutate — that’s theMutex’s job.Arc<T>alone gives shared reads;Arc<Mutex<T>>gives synchronized writes.
Atomic operations cost more than a plain integer bump, which is why Rc exists
at all: when you’re single-threaded, you shouldn’t pay for atomics. Rc and
Arc are otherwise the same API.
Capstone insight: build MyRc<T> from scratch
The capstone strips Rc to its essence and reveals there’s no magic — just one
heap box holding { count, value } and a pointer to it.
struct MyRcInner<T> {
strong: Cell<usize>, // Cell: mutate the count through a shared &self
value: T,
}
struct MyRc<T> {
ptr: NonNull<MyRcInner<T>>,
_marker: PhantomData<MyRcInner<T>>, // "I logically own a T" for drop-check
}
Two design choices encode deep facts about real Rc:
strong: Cell<usize>— the count must be mutable through&self(clone and drop both take shared references), so it needs interior mutability. ACell(non-atomic) is exactly why realRcis!Sync: a non-atomic counter is unsafe to touch from two threads.Arcswaps this forAtomicUsize.PhantomData<MyRcInner<T>>— we hold the value behind a rawNonNull, so the compiler can’t see thatMyRcowns aT. The marker tells dropck “I own aT,” which makes drop-checking correct forTs with lifetimes.
The four operations are the machine:
fn new(value: T) -> MyRc<T> { // allocate inner, strong = 1
let inner = Box::new(MyRcInner { strong: Cell::new(1), value });
MyRc { ptr: NonNull::new(Box::into_raw(inner)).unwrap(), _marker: PhantomData }
}
fn clone(&self) -> MyRc<T> { // bump count, copy the pointer
self.inner().strong.set(self.inner().strong.get() + 1);
MyRc { ptr: self.ptr, _marker: PhantomData }
}
fn deref(&self) -> &T { // SHARED access only
&self.inner().value
}
fn drop(&mut self) {
if self.inner().strong.get() == 1 { // I'm the last one
unsafe { drop(Box::from_raw(self.ptr.as_ptr())); } // free once, runs T's Drop
} else {
self.inner().strong.set(self.inner().strong.get() - 1); // others remain
}
}
The whole correctness argument: new starts at 1, clone adds 1 and shares the
pointer, drop either frees (on the 1-transition, reconstructing the Box so
its destructor runs T’s Drop exactly once) or decrements. The verification
uses a Dropper that logs its own drop to prove the inner value is freed
exactly once — not zero (leak), not twice (double-free):
let a = MyRc::new(Dropper("payload"));
{
let b = MyRc::clone(&a);
assert_eq!(MyRc::strong_count(&a), 2); // clone bumped the shared count
assert_eq!(a.ptr, b.ptr); // same inner, no deep copy
} // b drops: count 2 -> 1, inner still alive
assert_eq!(DROP_COUNT, 0); // nothing freed yet
// ... a drops: count 1 -> 0, Dropper runs once
Once you’ve written these four functions, Rc stops being a black box. It’s a
counter, a pointer, and the discipline of freeing on the last drop — and Arc
is the same four functions with Cell swapped for an atomic.
Reaching for
unsafehere is unavoidable (raw pointer deref, manual free), so this is the rung to validate with Miri:cargo miri run --bin rc_arccatches a leak, a double-free, or use-after-free that a normal run might miss.
Footguns
-
Rcgives you aliasing, not mutation.Rc<T>only ever yields&T. To mutate, either useRc::make_mut(clone-on-write — fine when sharing is rare) or stack aRefCell:Rc<RefCell<T>>(runtime-checked shared mutation). See theRc<RefCell<T>>note. -
Strong cycles leak — silently.
astrong-points atband vice versa -> neither count reaches 0 -> destructors never run. Safe Rust prevents use-after-free and double-free; it does not prevent leaks. Fix: make the back-edgeWeak. -
The ownership rule for back-pointers: the direction that owns is
Rc(strong); the direction that merely navigates back isWeak. Parent -> child strong, child -> parent weak. -
Weak::upgrade()can returnNone. AWeakdoesn’t keep the value alive, so by the time youupgrade()the target may be gone. You must handle theNone— that’s the whole point ofWeak. -
Rcis!Send/!Sync. You cannot move it across threads, by design — its counter isn’t atomic. UseArcfor that. But don’t reach forArcreflexively when single-threaded: you’d pay for atomics you don’t need. -
Arcshares; it doesn’t synchronize the data.Arc<T>gives shared reads. For cross-thread mutation you still need aMutex/RwLock:Arc<Mutex<T>>. The atomic refcount protects the handle count, not the value. -
Rc<String>is a double indirection. PreferRc<str>(orRc<[T]>) for shared immutable strings/slices — one fewer pointer hop and noStringheader.
Real-world patterns
| Pattern | Shape | Example |
|---|---|---|
| Interned immutable data | Rc<str> / Rc<[T]> cloned across many records | Category tags, symbol tables, shared config |
| Shared DAG node | One node held by several parents via Rc | Expression trees with common sub-expressions, scene graphs |
| Copy-on-write | Rc::make_mut mutates in place when unshared, copies when shared | Persistent/immutable data structures, Cow-like APIs |
| Tree with parent pointers | children Rc (own), parent Weak (navigate back) | DOM, file-system models, ASTs with parent links |
| Cross-thread shared state | Arc<Mutex<T>> / Arc<RwLock<T>> | Counters, caches, connection pools, shared registries |
| Cheap immutable snapshots | hand out Arc<T> clones of a config/state | Hot-reloadable config, lock-free read paths |
Explain it back
- What two things live in an
Rc’s heap allocation, and what does a singleRchandle own? - Why is
clone()on anRccheap, and what exactly gets copied? - Give a data shape
Boxcannot express butRccan. Why not? - When does
Rc::make_mutmutate in place, and when does it copy? What decides? - Walk the strong counts through an
a <-> bstrong cycle as the locals drop. Which count stays non-zero, and what’s the consequence? - In a parent/child tree, which edge is
Rcand which isWeak? What leaks if you swap them? - What does
Weak::upgrade()return, and when is itNone? - Why is
Rc!Send? What doesArcchange, and what does it not change about mutating the inner value? - Which two distinct things does
Arc<Mutex<T>>protect, and with which mechanism each? - In
MyRc, why is the count aCell<usize>rather than a plainusize, and what would you change to getArc? Why does the lastdropfree exactly once?
See also
Rc<RefCell<T>>patterns — add interior mutability on top of the shared-ownership layer built here; the cycle/Weakstory in fullCell/RefCell— the interior-mutability layer (themake_mutand capstoneCell<usize>both rely on it)Drop& Ordering — why a cycle means destructors never run, and how the lastRcdrop triggers the freeCow— Clone-on-Write —make_mutis the refcount-driven version of the same copy-on-write idea
Rc<RefCell<T>> patterns
Ladder:
src/bin/rc_refcell.rs· Run:cargo run --bin rc_refcell· Phase 1 · 10 rungs
TL;DR
Rc<RefCell<T>> is the single-threaded “shared mutable state” idiom, built by
stacking two jobs: Rc gives many owners (it lets the value be aliased),
and RefCell lets you mutate through a shared & by moving the borrow
check from compile time to runtime. The whole tension — and every footgun —
lives in that stack: Rc hands out N references to one cell, but RefCell
still allows only one &mut at a time, so the aliasing Rc enables is exactly
what makes RefCell panic at runtime. Reach for it only when you genuinely need
both shared ownership and mutation; the cost is runtime borrow panics and
the ever-present risk of reference-cycle leaks.
Why it exists (from first principles)
Plain Rust ownership is a tree: one owner, borrows flow down. But some data shapes are graphs — a node pointed at from two places (doubly-linked list, DOM, a tree with parent pointers), or one piece of state several objects mutate (an event log, an observer registry).
You can’t express “two owners” with &mut — that’s exclusive. And you can’t
mutate through Rc<T> alone, because Rc only hands out &T (shared
references). Neither layer solves the problem on its own:
| Layer alone | What you get | What’s missing |
|---|---|---|
Rc<T> | Multiple owners of the same allocation | No mutation — Rc gives out only &T |
RefCell<T> | Interior mutability behind a & | Only one owner — no way to alias the cell |
Stack them: Rc provides the shared-ownership topology (multiple handles to one
allocation), and RefCell provides the mutation through those shared handles.
The price is that the borrow check moves from compile time to runtime — two
borrow_mut() calls on the same cell at the same time will panic, not fail
to compile.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | foundations | shared cell | one RefCell, two Rc handles; mutate via A, see it via B; Rc::ptr_eq proves one allocation |
| 2 | foundations | shared owners | two structs each hold a clone of one Rc<RefCell<Vec>>; &self methods mutate via the cell; strong_count counts owners |
| 3 | mechanics | counts & cheapness | &Rc peeks without owning, only .clone() adds an owner, borrow_mut() reaches inside; dropping one owner keeps the value alive |
| 4 | footgun | double borrow_mut | two live borrows of the same cell -> BorrowMutError panic; the overlap (guard staying alive) is what triggers it |
| 5 | footgun | borrow across a call | holding a borrow while calling a method that re-borrows the same cell (reentrancy) -> panic; release the guard before recursing |
| 6 | footgun | the cycle leak | a -> b -> a with strong Rcs: each pins the other, strong_count never hits 0, Drop never runs — a silent leak |
| 7 | real-world | Weak + tree | down = strong (own), up = weak (observe); Rc::downgrade / Weak::upgrade; the tree frees cleanly |
| 8 | real-world | observer/subject | a Subject co-owns observers and fans one event out to all via borrow_mut in a loop |
| 9 | capstone | doubly-linked list | next: Rc / prev: Weak; push both ends, traverse forward and backward, drop with no leak |
| 10 | capstone+ | iterative Drop | the default recursive drop of an Rc-chained list overflows the stack on long chains; .take() each next before the node drops to tear down flat |
The ideas, built up
The shared-cell “aha”
The fundamental move: make one RefCell, wrap it in Rc, clone the Rc.
Now two handles point at the same underlying cell. A mutation through one
is immediately visible through the other — because there is no copy; both
handles dereference to the same allocation.
fn shared_cell(start: i32) -> (Rc<RefCell<i32>>, Rc<RefCell<i32>>) {
let original = Rc::new(RefCell::new(start));
let cloned = original.clone();
(original, cloned)
}
The check proves this:
let (a, b) = shared_cell(10);
*a.borrow_mut() += 5;
assert_eq!(*b.borrow(), 15); // b sees a's mutation
assert!(Rc::ptr_eq(&a, &b)); // same allocation
Rc::ptr_eq is the definitive test — it compares the raw pointer inside each
Rc, confirming they reference the same heap allocation, not just equal values.
This is what “shared ownership” means: not two copies of the data, but two
handles to one copy.
From loose handles to owned structs
Loose locals sharing a cell is a demo. The real pattern is separate structs
each holding a handle to the same shared state, mutating it through &self
methods. The type alias makes the intent clear:
type Log = Rc<RefCell<Vec<String>>>;
struct Logger { log: Log }
struct Auditor { log: Log }
Both Logger::record(&self, msg) and Auditor::count(&self) take &self —
no &mut self needed, because mutation goes through the RefCell, not through
the Rust borrow of self:
impl Logger {
fn record(&self, msg: &str) {
self.log.borrow_mut().push(msg.to_string());
}
}
impl Auditor {
fn count(&self) -> usize {
self.log.borrow().len()
}
}
After two record() calls, the Auditor — a completely separate struct —
sees both entries. And Rc::strong_count(&log) reports 3: the original handle,
the Logger’s clone, and the Auditor’s clone. Three owners, one Vec.
This is why the pattern exists: it decouples ownership from mutability. Each struct holds a shared reference to the cell; the cell enforces exclusive access at runtime.
&Rc peeks, .clone() owns, borrow_mut() reaches inside
Three distinct operations, and confusing them creates bugs:
fn peek_count(h: &Counter) -> usize {
Rc::strong_count(h) // reads the count, no new owner
}
fn bump(h: &Counter, n: i32) {
*h.borrow_mut() += n; // mutates the inner value, no new owner
}
fn make_sibling(h: &Counter) -> Counter {
Rc::clone(h) // creates a new owner (bumps strong_count)
}
Passing &Rc<RefCell<T>> lets you read AND mutate the shared value without
changing the owner count. The &Rc auto-derefs through Rc to reach the
RefCell, and borrow_mut() is a &self method on RefCell — so all you
need is a shared reference to the Rc.
Only Rc::clone() (or the equivalent .clone() on an Rc) bumps
strong_count. The clone is cheap — it copies a pointer and increments a
counter, not the underlying data.
And dropping an owner doesn’t kill the value — it decrements strong_count.
The value survives as long as at least one Rc exists:
let sib = make_sibling(&h);
assert_eq!(peek_count(&h), 2); // two owners
drop(sib);
assert_eq!(peek_count(&h), 1); // back to one; value still alive
assert_eq!(*h.borrow(), 115); // the value is unaffected
Footgun 1: double borrow_mut panics at runtime
This is the defining cost of Rc<RefCell<T>>. The compiler can’t see that two
Rc handles alias the same cell, so it can’t reject a double borrow at compile
time. RefCell re-imposes the rule at runtime: one &mut XOR many &,
enforced by a panic.
fn try_double_mut(x: &Counter, y: &Counter, add: i32) -> Result<(), ()> {
let mut first = x.borrow_mut(); // holds a &mut to the cell
let mut second = y.try_borrow_mut().map_err(|_| ())?; // tries ANOTHER &mut
*first += add;
*second += add;
Ok(())
}
When x and y are different cells, both borrows succeed — they’re independent
RefCells. When they alias (created via Rc::clone), the second borrow finds
the cell already mutably borrowed and fails. The non-panicking
try_borrow_mut() returns Err(BorrowMutError); the panicking borrow_mut()
would crash the thread.
The check also proves the panic version directly:
let _first = h2.borrow_mut();
let _second = alias2.borrow_mut(); // BorrowMutError -> panic
The key detail: it’s the overlap that triggers the panic, not the mere
existence of two handles. If you scope the first borrow so it’s dropped before
the second one starts, no conflict occurs. The RefMut guard returned by
borrow_mut() tracks the borrow’s lifetime — when it drops, the borrow ends.
Footgun 2: borrow held across a call (reentrancy)
The rung-4 double borrow was obvious because both borrows were on adjacent lines. The version that actually bites people in real code is indirect: you hold a borrow, then call a function that — somewhere down the stack — borrows the same cell again. The cell doesn’t know it’s “the same logical operation”; it just sees a second borrow while the first is live, and panics.
The ladder sets up a Bank scenario: an Account can have a backup account
(another Rc<RefCell<Account>>). Withdrawal falls through to the backup if the
primary balance is insufficient. If the backup aliases the primary (a
self-referential backup), the naive implementation holds a borrow_mut() of the
account while recursing into the backup — which tries to borrow_mut() the
same cell again.
The fix: extract what you need from the cell into local variables, drop the guard (by ending its scope), then recurse:
fn withdraw(acct: &Acct, amount: i32) -> Result<i32, &'static str> {
let (shortfall, backup) = {
let mut account = acct.borrow_mut(); // borrow starts
if account.balance >= amount {
account.balance -= amount;
return Ok(account.balance);
}
let shortfall = amount - account.balance;
account.balance = 0;
(shortfall, account.backup.clone()) // clone the Rc handle out
}; // borrow ENDS here
// Now the cell is unborrowed — safe to pass to a recursive call
let Some(backup) = backup else {
return Err("insufficient");
};
if Rc::ptr_eq(acct, &backup) {
return Err("insufficient"); // self-backup: can't double-spend
}
withdraw(&backup, shortfall)?;
Ok(0)
}
The pattern is: read what you need, drop the guard, then call. The curly
braces around the borrow block are the mechanism — when the RefMut guard
goes out of scope, the borrow ends. The Rc::ptr_eq check is an additional
safety net: even after releasing the borrow, recursing into the same cell
would drain an already-zeroed balance, so the function short-circuits.
Footgun 3: the reference cycle that never frees
The runtime borrow panic is loud — you find it fast. This footgun is silent:
it’s a memory leak. Rc frees its value only when strong_count hits 0. If
two nodes hold strong Rc handles to each other, each keeps the other’s count
at >=1 forever — even after every external handle is gone. Destructors never
run.
The ladder builds a Node with a Drop impl that logs into a shared Vec
when it dies:
impl Drop for Node {
fn drop(&mut self) {
self.dropped.borrow_mut().push(self.name.clone());
}
}
Then it creates a cycle:
let a = make_node("a", &log);
let b = make_node("b", &log);
link(&a, &b); // a -> b (strong)
link(&b, &a); // b -> a (strong) — now it's a cycle
Each node starts with 2 strong owners: the local variable and the other node’s
link. When the locals go out of scope, each count drops to 1 — but never to 0,
because the cycle holds. Neither Node::drop ever fires. The drop log is
empty.
This is safe — Rust prevents use-after-free and double-free, but it does
not prevent leaks. Rc cycles are the single-threaded equivalent of a
“GC-proof” leak in a garbage-collected language: the objects are unreachable
but never collected.
Breaking cycles with Weak: the parent-pointer tree
The fix for the cycle leak is Weak<T> — a non-owning handle. Weak does
not increment strong_count, so it cannot pin a value alive. To use the
value behind a Weak, you must upgrade() it to an Option<Rc<T>> — and
you get None if the value was already dropped.
The ownership rule for avoiding cycles:
The direction that owns uses
Rc(strong). The direction that merely observes usesWeak.
In a tree: parent -> child is strong (the parent owns its children); child -> parent is weak (the child can navigate up but must not keep the parent alive).
fn add_child(parent: &Tree, child: &Tree) {
parent.borrow_mut().children.push(Rc::clone(child)); // strong down
child.borrow_mut().parent = Rc::downgrade(parent); // weak up
}
fn parent_value(child: &Tree) -> Option<i32> {
child.borrow().parent.upgrade() // Option<Rc<...>>
.map(|parent| parent.borrow().value)
}
Rc::downgrade(&rc) creates a Weak from an Rc. weak.upgrade() tries
to promote it back to an Rc, succeeding only if the target still has at
least one strong owner.
The counts tell the story:
assert_eq!(Rc::strong_count(&root), 1); // only the local variable
assert_eq!(Rc::weak_count(&root), 1); // the child's parent pointer
assert_eq!(Rc::strong_count(&leaf), 2); // local + parent's children vec
The child’s weak pointer to the root does not bump strong_count. When
the locals go out of scope, the root’s strong count reaches 0 — it drops,
its children Vec drops, the leaf’s strong count reaches 0 — it drops too.
Both TreeNode::drop implementations fire. The drop log confirms both nodes
freed.
The ladder also proves that you can mutate the parent through the child’s
back-pointer — shared mutability across the tree, which is the whole reason
RefCell is in the stack:
if let Some(p) = leaf.borrow().parent.upgrade() {
p.borrow_mut().value = 99;
}
assert_eq!(root.borrow().value, 99);
Real-world pattern: observer / subject fan-out
The other canonical use: one event source (“subject”) pushes updates into many
independent observers, each holding its own mutable state. The subject owns a
list of Rc<RefCell<Observer>> handles; calling publish borrows each one
mutably in a loop:
impl Subject {
fn publish(&self, value: i32) {
for observer in &self.observers {
let mut observer = observer.borrow_mut();
observer.seen += 1;
observer.last = value;
}
}
}
The callers holding their own Rc handles to the same observers see the
mutations — because they’re the same cells:
subject.publish(10);
subject.publish(20);
assert_eq!(a.borrow().seen, 2); // the caller's handle sees the subject's writes
assert_eq!(a.borrow().last, 20);
The borrow discipline from rung 4 matters here: each borrow_mut() must be
scoped to one loop iteration. If you held a borrow across iterations and two
observers aliased the same cell, you’d hit the same double-borrow panic.
This is the shape behind event buses, reactive signals, and GUI data-binding in single-threaded Rust.
Capstone: a doubly-linked list from scratch
The structure that forces everything from this ladder together. A doubly-linked
list can’t be built with plain ownership: a node is pointed at from both
directions (its predecessor’s next and its successor’s prev), so it needs
shared ownership — and you need to mutate those links after the nodes exist,
so it needs interior mutability.
The rung-7 ownership rule maps perfectly:
| Link | Direction | Ownership | Why |
|---|---|---|---|
next | forward | Rc (strong) | the list owns its nodes going forward |
prev | backward | Weak | backward links must not pin nodes, or every adjacent pair forms a rung-6 cycle |
struct DNode {
value: i32,
next: Option<DLink>, // strong: owns the next node
prev: Weak<RefCell<DNode>>, // weak: observes the previous node
dropped: IntDropLog,
}
struct List {
head: Option<DLink>,
tail: Option<DLink>,
dropped: IntDropLog,
}
push_back appends a node. The wiring is: set the new node’s prev to a
weak handle of the old tail, then set the old tail’s next to a strong handle
of the new node. The borrow discipline requires care — you borrow the old tail
mutably to set its next, and borrow the new node mutably to set its prev,
but they are different cells so no conflict:
fn push_back(&mut self, value: i32) {
let new_node = Rc::new(RefCell::new(DNode::new(value, &self.dropped)));
match self.tail.take() {
None => {
self.head = Some(Rc::clone(&new_node));
self.tail = Some(new_node);
}
Some(old_tail) => {
new_node.borrow_mut().prev = Rc::downgrade(&old_tail);
old_tail.borrow_mut().next = Some(Rc::clone(&new_node));
self.tail = Some(new_node);
}
}
}
Forward traversal walks head -> next -> next -> ...:
fn to_vec(&self) -> Vec<i32> {
let mut values = Vec::new();
let mut current = self.head.clone();
while let Some(node) = current {
let node_ref = node.borrow();
values.push(node_ref.value);
current = node_ref.next.clone();
}
values
}
Backward traversal walks tail -> prev.upgrade() -> prev.upgrade() -> ...,
proving the Weak back-links are correctly wired:
fn to_vec_rev(&self) -> Vec<i32> {
let mut values = Vec::new();
let mut current = self.tail.clone();
while let Some(node) = current {
let node_ref = node.borrow();
values.push(node_ref.value);
current = node_ref.prev.upgrade();
}
values
}
The traversal clone()s the Rc handle to advance the cursor, then borrows
the node to read its value and get the next link. The borrow ends when
node_ref goes out of scope at the next iteration — so no borrow overlaps.
The drop test is the proof that the whole structure works: when the List is
dropped, its head drops node 1, whose next drops node 2, and so on — a
cascade of strong-count-reaching-zero. No prev link holds anything alive
because they’re all Weak. The drop log shows all 4 nodes freed in
front-to-back order:
assert_eq!(dropped, vec![1, 20, 3, 4], "front-to-back drop order");
Interior mutability works through the list too — borrowing a node handle and mutating its value is visible via traversal:
n2.borrow_mut().value = 20;
assert_eq!(list.to_vec(), vec![1, 20, 3, 4]);
Footguns
-
Rcdefeats the compile-time aliasing check, soRefCellre-imposes it at runtime. Twoborrow_mut()s on the same cell (reachable via twoRchandles) panic withalready borrowed: BorrowMutError. You traded a compile error for a possible panic. -
Borrow held across a call (reentrancy). The sneaky version: you hold a
borrow_mut(), then call a method that — somewhere down the stack — borrows the same cell. The cell doesn’t know it’s “the same logical operation”; it panics. Fix: read what you need out of the cell, drop the guard (scope it in a{ }block ordrop(guard)), then make the call. -
Strong reference cycles leak.
aholds a strongRctoband vice versa -> neither count reaches 0 -> destructors never run, memory never frees. Safe Rust prevents use-after-free and double-free; it does not prevent leaks. Fix: make one directionWeak. -
The ownership rule for back-pointers: the direction that owns uses
Rc(strong); the direction that merely observes/navigates back usesWeak. Parent -> child strong, child -> parent weak.nextstrong,prevweak. -
Rc-chained structures recurse on Drop. Dropping the head of a longRc-linked list drops itsnext, which drops itsnext… -> stack overflow in the destructor for very long chains. Fix: a manual iterativeDropthat pops nodes in a loop.
Signatures to know
type Shared<T> = Rc<RefCell<T>>;
// Rc — shared ownership (no mutation of its contents)
Rc::new(v) -> Rc<T>
Rc::clone(&rc) -> Rc<T> // cheap: bumps strong_count, same allocation
Rc::strong_count(&rc) -> usize
Rc::weak_count(&rc) -> usize
Rc::ptr_eq(&a, &b) -> bool // same allocation?
Rc::downgrade(&rc) -> Weak<T> // a non-owning handle
// Weak — non-owning; doesn't keep the value alive
Weak::new() -> Weak<T> // points at nothing
weak.upgrade() -> Option<Rc<T>> // None if the target was dropped
// RefCell — interior mutability, borrow-checked at RUNTIME
cell.borrow() -> Ref<'_, T> // panics if a &mut is out
cell.borrow_mut() -> RefMut<'_, T> // panics if ANY borrow is out
cell.try_borrow_mut() -> Result<RefMut, BorrowMutError> // non-panicking
Real-world patterns
| Pattern | Shape | Example |
|---|---|---|
| Shared log / registry | Multiple structs co-own one Rc<RefCell<Vec>> | Event logs, metric collectors, DI containers |
| Observer / subject | Subject owns Vec<Rc<RefCell<Observer>>>; publish fans out via borrow_mut | Event buses, reactive signals, GUI data-binding |
| Tree with parent pointers | Children = Rc (owned), parent = Weak (observed) | DOM trees, scene graphs, file-system models |
| Doubly-linked list | next = Rc, prev = Weak | Caches (LRU), undo stacks, playlist navigation |
| Graph with back-edges | Forward edges Rc, back-edges Weak | Dependency graphs, social graphs |
Explain it back
- Why does
Rc<RefCell<T>>need both layers — what fails if you drop either? - Two
Rchandles to one cell, bothborrow_mut()at once: compile error or runtime panic? Why? - You have a graph traversal that panics with
BorrowMutErroreven though it “looks single-threaded and sequential.” What’s the likely cause? - In the
withdrawfunction, why must the borrow end before the recursive call? What happens if you remove the inner{ }block? - Why does an
a <-> bstrong cycle leak, and exactly which count stays non-zero? - In a tree with parent pointers, which link is
Rcand which isWeak, and what breaks if you swap them? - What does
Weak::upgrade()return, and when is itNone? - In the doubly-linked list, why does the drop cascade proceed front-to-back?
What would happen if
prevwere strong instead ofWeak?
See also
Rc/Arc— the shared-ownership layer on its ownCell/RefCell— the interior-mutability layer and the runtime borrow checkDrop& Ordering — why the cycle leak means destructors never run; iterativeDropfor linked structuresBorrow/ToOwned— theMyCowcapstone also stacks shared-ownership with interior mutability
Associated Types vs Generic Params
Ladder:
src/bin/assoc_vs_generic.rs· Run:cargo run --bin assoc_vs_generic· Phase 2 · 9 rungs
TL;DR
A trait can carry “extra” types in two ways, and the choice is not stylistic — it changes what the type system lets you do:
- Generic parameter (
trait Convert<T>): the type is an input. The caller or the impl picks it, so one type can implement the trait many times, once per choice ofT. - Associated type (
trait Iterator { type Item; }): the type is an output. The implementor determines it once, so there is exactly one impl per type and the compiler can deduce the output instead of asking you.
Rule of thumb: input → generic param, output → associated type.
Why this exists (from first principles)
Say you want a trait whose method returns “some related type”. You need to tell the trait what that type is. There are only two places it can come from:
- The caller supplies it. Then it must be a parameter on the trait:
Convert<T>. Different callers want differentT, so the same type must be allowed to implementConvert<i32>andConvert<String>. - The implementor fixes it. Then it belongs inside the impl as an
associated type:
type Output = i32;. There is one right answer per type, so a second impl with a different answer would be a contradiction.
That single fork — “who chooses the type?” — drives everything else: how many
impls are allowed, whether the compiler can infer the result, whether you can put
it behind dyn, and how the whole iterator-adapter ecosystem resolves element
types at compile time.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | foundations | Two shapes | Same trait written with type Item vs <T>; feel the syntax |
| 2 | foundations | The defining rule | One impl per type (assoc) vs many (generic); E0119 |
| 3 | mechanics | Equality bounds | where I: Iterator<Item = u64> + I::Item projection |
| 4 | mechanics | Your own iterator | impl Iterator with type Item for Countdown |
| 5 | footgun | Inference & turbofish | Generic .into() ambiguity (E0283) vs determined output |
| 6 | footgun | Trait objects | dyn Iterator<Item=..> must pin the assoc type (E0191) |
| 7 | real-world | Add uses both | Rhs generic param + Output associated, in one trait |
| 8 | real-world | Design the split | A Graph trait — decide what’s assoc vs generic |
| 9 | capstone | MyIterator + Map | Thread an associated Item through a generic adapter |
The ideas, built up
1. Two shapes for the same idea
The same “pop an item out” trait, written both ways:
// Shape A: associated type — implementor names the output once.
trait PopAssoc {
type Item;
fn pop_it(&mut self) -> Option<Self::Item>;
}
// Shape B: generic param — output is a parameter on the trait.
trait PopGeneric<T> {
fn pop_it(&mut self) -> Option<T>;
}
The difference shows up at the impl site. With the associated type the chosen type goes inside the impl body; with the generic param it goes in the impl header:
impl PopAssoc for Stack {
type Item = i32; // output: declared inside
fn pop_it(&mut self) -> Option<Self::Item> { self.items.pop() }
}
impl PopGeneric<i32> for Stack { // input: chosen in the header
fn pop_it(&mut self) -> Option<i32> { self.items.pop() }
}
Because Stack now has two pop_it methods (one per trait), a bare
s.pop_it() is ambiguous — the ladder calls them with fully-qualified syntax
(PopAssoc::pop_it(&mut s), PopGeneric::<i32>::pop_it(&mut s)). That ambiguity
is a first hint that generic params multiply impls.
2. The defining rule: one impl vs many
This is the whole concept in miniature. An associated type makes the trait a
function of Self — one input, one answer:
trait Producer { type Output; fn produce(&self) -> Self::Output; }
impl Producer for Counter { type Output = i32; /* ... */ }
// WRONG: a second impl, even with a different Output, is rejected.
// impl Producer for Counter { type Output = String; /* ... */ }
// error[E0119]: conflicting implementations of trait `Producer`
// for type `Counter`
A generic param makes the trait a relation — many answers are fine:
trait Convert<T> { fn convert(&self) -> T; }
impl Convert<i32> for Counter { /* ... */ } // OK
impl Convert<String> for Counter { /* ... */ } // OK — different T
The consequence you feel immediately: produce() needs no annotation (one
answer), but convert() does (the compiler must know which impl):
let p = c.produce(); // i32, deduced
let as_int: i32 = c.convert(); // must say which T
let as_str: String = c.convert();
3. Equality bounds and projection
Associated types unlock two things generic params make clumsy.
Equality bounds pin the output type inside a where clause, keeping the
iterator the only type parameter:
fn sum_items<I>(it: I) -> u64
where
I: Iterator<Item = u64>, // "any iterator whose Item is exactly u64"
{
it.sum()
}
Projection lets you name the output as I::Item in your own signature — again
with no extra type parameter:
fn first<I>(mut it: I) -> Option<I::Item>
where
I: Iterator,
{
it.next()
}
Contrast the generic-trait version. With trait Stream<T> you would be forced to
introduce a separate T that leaks into every signature:
// What you'd be stuck writing with a generic-param iterator trait:
fn first_g<S, T>(s: S) -> Option<T> where S: Stream<T> { /* ... */ }
// ^^^ extra param, and callers must disambiguate T because a type
// could implement Stream<u64> AND Stream<String>.
Associated types turn that T from a parameter-you-must-supply into an
output-the-compiler-deduces.
4. Implementing the real Iterator
Iterator is the canonical associated-type trait:
trait Iterator { type Item; fn next(&mut self) -> Option<Self::Item>; }
Why is Item associated? Because a given iterator yields exactly one type of
value. If it were generic (Iterator<T>), a single type could “be an iterator”
of many T, and then for x in it wouldn’t know what x is, and .map,
.filter, .sum would all be ambiguous. The ladder implements it for a
countdown:
impl Iterator for Countdown {
type Item = u32;
fn next(&mut self) -> Option<Self::Item> {
if self.current == 0 { None }
else { let c = self.current; self.current -= 1; Some(c) }
}
}
The payoff: because you implemented the real std trait, you get .collect(),
.sum(), .map(), and for-loops for free — all keyed off the single Item.
Footguns
Generic params owe you a disambiguation tax (E0283)
Because Counter: Convert<T> holds for more than one T, a function that returns
the generic output can’t be called without help:
fn pull<T>(c: &Counter) -> T where Counter: Convert<T> { c.convert() }
// let oops = pull(&c); // error[E0283]: type annotations needed
let via_annotation: i32 = pull(&c); // fix 1: pin via the binding's type
let via_turbofish = pull::<String>(&c); // fix 2: pin at the call site
This is the same tax you already pay on .into(), .parse(), and
.collect::<Vec<_>>(). Associated outputs (c.produce()) never charge it, because
there is only one answer.
dyn forces you to pin the associated type (E0191)
A trait object must be a concrete, fully-known type behind the pointer. So the associated type has to be nailed down:
fn boxed_counter(n: u32) -> Box<dyn Iterator<Item = u32>> { /* ... */ }
// WRONG:
// fn bad(n: u32) -> Box<dyn Iterator> { /* ... */ }
// error[E0191]: the value of the associated type `Item` must be specified
For a generic-param trait the analogue is simply choosing which object you mean:
dyn Convert<i32> and dyn Convert<String> are two unrelated trait-object types.
The associated type is part of the object’s identity; the generic param selects
the object.
Real-world patterns
Add deliberately uses both
std::ops::Add is the masterclass — it carries a generic param and an
associated type, each chosen for the right reason:
pub trait Add<Rhs = Self> {
type Output;
fn add(self, rhs: Rhs) -> Self::Output;
}
Rhsis a generic param (an input): you might addMeters + Meters, orMeters + f64, orMeters + Vector. Multiple right-hand sides → many impls per type. It even defaults toSelf.Outputis an associated type (an output): once you fix the pair(Self, Rhs), the result type is determined. One answer per impl.
impl Add for Meters { type Output = Meters; /* Meters + Meters */ }
impl Add<f64> for Meters { type Output = Meters; /* Meters + f64 */ }
// The determined output can even be named with projection:
let r: <Meters as Add<f64>>::Output = Meters(3.0) + 1.0;
Designing your own split
When you design a trait, sort each “extra type” into input or output. The ladder’s
Graph trait makes both node-id and weight associated, because a graph has exactly
one of each — they are facts about the graph, not knobs a caller turns:
trait Graph {
type NodeId: Copy + Eq; // one id type per graph → associated
type Weight; // one weight type per graph → associated
fn neighbors(&self, n: Self::NodeId) -> Vec<(Self::NodeId, Self::Weight)>;
}
// A consumer stays clean — one type param, node type via projection:
fn neighbor_count<G: Graph>(g: &G, n: G::NodeId) -> usize { g.neighbors(n).len() }
Had NodeId been a generic Graph<N> param, a single graph type could claim to
be a graph of u32 ids and (i32,i32) ids, and every consumer would need an
extra ambiguous type parameter.
Capstone insight
The capstone rebuilds the iterator-adapter machinery and reveals the deepest move: an adapter’s associated type is computed from its generic parameters.
struct Map<I, F> { iter: I, f: F } // generic over inner iter + closure
impl<I, F, B> MyIterator for Map<I, F>
where
I: MyIterator,
F: FnMut(I::Item) -> B, // F maps inner items to some B
{
type Item = B; // <-- the adapter's output IS the closure's output
fn next(&mut self) -> Option<Self::Item> {
match self.iter.next() {
Some(x) => Some((self.f)(x)),
None => None,
}
}
}
type Item = B is the whole trick. B is a generic parameter of the impl,
constrained by the closure’s return type, and it becomes the associated type of
the resulting iterator. That is how a chain like
Upto { next: 1, end: 4 }
.map_it(|x| x + 10) // u32 -> u32
.map_it(|x| x as usize * 2) // u32 -> usize
threads its element type u32 → u32 → usize entirely through associated-type
projection, resolved statically with zero annotations. Every std iterator chain
you have ever written works exactly this way.
Explain it back
Answer these cold:
- Why can a type implement
From<A>andFrom<B>but not have twoIteratorimpls with differentItems? - Why does
let x: i32 = something.into()need the annotation whileiter.next()does not? - What does
where I: Iterator<Item = u64>give you thatwhere I: Stream<u64>(a generic-param trait) would not? - Why must you write
Box<dyn Iterator<Item = u32>>and notBox<dyn Iterator>? - In
Add<Rhs = Self> { type Output; }, why isRhsgeneric butOutputassociated? - In the
Map<I, F>adapter, where doestype Itemcome from, and why is that the key to compile-time iterator chains?
See also
- Conversion traits —
From/Intoare the archetypal generic-param traits (and the source of.into()ambiguity). - Borrow / ToOwned —
ToOwned::Ownedis an associated type used exactly as an “output determined by the impl”. - Lifetimes in depth — the
IteratorItemlifetime rung is the lifetime-flavored version of projection.
Generic bounds & where clauses
Ladder:
src/bin/generic_bounds.rs· Run:cargo run --bin generic_bounds· Phase 2 · 9 rungs
TL;DR
A generic parameter T arrives as a black box: the compiler knows nothing about it, so you
can’t call anything on it. A bound (T: Trait) is a contract that does two things at once —
it restricts the caller (“you may only pass types that implement Trait”) and empowers the
body (“therefore I’m allowed to use Trait’s methods on a T”). Every method you call on a
generic must be justified by a bound.
A where clause is the same bounds written below the signature instead of inline. It’s not just
cosmetic: the inline <T: Bound> form can only bound a bare type parameter, so anything
structured — a projection like I::Item, or a bound on a derived type like &'a C or Vec<T> —
must live in a where clause. That’s the dividing line.
Why this exists (from first principles)
Rust monomorphizes generics: min_item::<i32> and min_item::<char> compile to separate machine
code. But type-checking happens once, on the generic definition, before any concrete type is
known — not separately per instantiation (that’s C++ templates, where errors surface at the use
site as walls of noise).
So when the compiler sees this generic body, it must decide right now whether it’s legal:
fn min_item<T>(items: &[T]) -> T {
// is `a < b` allowed here? the compiler has NO idea what T is.
}
With nothing known about T, almost nothing is permitted — you can move it, drop it, take its
address, and little else. A bound is how you tell the checker what T is guaranteed to support, so
it can verify the body once and trust it for every future T:
fn min_item<T: PartialOrd>(items: &[T]) -> T {
// now `a < b` typechecks: PartialOrd guarantees it for EVERY T a caller can pass.
}
This is the whole game. Bounds are how you trade away “any type at all” for “the capabilities you actually need.” Too few bounds and the body won’t compile; too many and you needlessly reject callers (the over-constraint footgun in rung 3).
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | foundations | min_item<T: PartialOrd + Copy> | A single bound turns the black box into something comparable. |
| 2 | foundations | dedup_describe w/ 3 bounds | Multiple bounds; where keeps a crowded signature readable. |
| 3 | mechanics | Stack<T> | Bound the method, not the struct — don’t over-constrain. |
| 4 | mechanics | Pair<T>::cmp_display | A method that exists only for some T (conditional method). |
| 5 | footgun | show<T: Display + ?Sized> | The hidden Sized bound, and how ?Sized relaxes it. |
| 6 | footgun | join_display / sum_borrowed | Bounds you can write only in a where clause. |
| 7 | real-world | trait Summary blanket impl | One impl gives every qualifying type a method; coherence is the cost. |
| 8 | real-world | PartialEq/Clone for MyBox<T> | Conditional trait impl — what #[derive] actually emits. |
| 9 | capstone | trait IterExt | Supertrait + blanket impl + per-method where Self::Item: bounds. |
The ideas, built up
1. A bound is a contract in two directions
fn min_item<T>(items: &[T]) -> T
where
T: PartialOrd + Copy,
{
*items
.iter()
.min_by(|a, b| a.partial_cmp(b).expect("items are comparable"))
.expect("items is non-empty")
}
Two bounds, two distinct reasons:
PartialOrdlets the body compare elements (a.partial_cmp(b)). Without it,<andpartial_cmpdon’t exist forT.Copylets the function return aTby value out of a borrowed&[T]. You’re handing back one of the borrowed elements;Copysays “duplicating it is a trivial bit-copy, the original stays put.”
Why
PartialOrd, notOrd? The test passes&[2.5, 0.5, 7.0]. Floats are onlyPartialOrd, neverOrd, becauseNaNmakes them not totally ordered (NaN < x,NaN > x, andNaN == xare all false). Reaching forPartialOrdkeepsf64callers in; demandingOrdwould lock them out. Picking the weakest bound that still compiles is a real API-design instinct — see Associated types vs generic params for the same theme.
2. Multiple bounds, and where where earns its keep
fn dedup_describe<T>(items: &[T]) -> String
where
T: PartialEq + Copy + Debug,
{
let mut result = Vec::new();
for item in items {
if result.last() != Some(item) { // PartialEq: compare neighbours
result.push(*item); // Copy: duplicate out of the borrow
}
}
format!("{:?}", result) // Debug: render with {:?}
}
Each bound again maps to one capability: PartialEq for the !=, Copy for *item, Debug for
{:?}. With three bounds, inline <T: PartialEq + Copy + Debug> already crowds the line; the
where form scales without pushing the return type off-screen. For these bounds it’s pure style —
they’d work inline too. Rung 6 is where where stops being optional.
A subtlety worth noting:
result.last()isOption<&T>andSome(item)isOption<&T>, so the!=compares twoOption<&T>. That works becausePartialEqis lifted throughOptionand&—Option<&T>: PartialEqholds wheneverT: PartialEq. The single bound onTquietly powers a comparison two layers up.
3. Bound the method, not the struct
The single most common beginner mistake:
// WRONG: the bound infects every use site.
struct Stack<T: Debug> { items: Vec<T> }
// Now `Stack<SomethingNotDebug>` won't even compile — you can't store a socket,
// a closure, or any non-Debug type, even if you never print it.
// OK: the struct holds ANYTHING; the capability lives on the impl that needs it.
struct Stack<T> { items: Vec<T> }
impl<T> Stack<T> { // unbounded: available for every T
fn new() -> Self { Self { items: Vec::new() } }
fn push(&mut self, value: T) { self.items.push(value); }
fn len(&self) -> usize { self.items.len() }
}
impl<T: Debug> Stack<T> { // bounded: only when T: Debug
fn dump(&self) -> String { format!("{:?}", self.items) }
}
The ladder proves it: Stack<NotDebug> (a type with no Debug impl) still constructs, pushes, and
reports its length, because those methods live in the unbounded impl<T>. Only dump requires
Debug, and only dump is gated.
This is exactly how Vec<T> is built. Vec<T> stores any T; .contains appears only for
T: PartialEq, .to_vec only for T: Clone, .sort only for T: Ord. The capabilities are
sliced across many impl blocks so the container itself constrains nothing.
Rule of thumb: put a bound at the lowest point that needs it. On a struct definition it’s almost always wrong; on the impl block or the individual method is almost always right.
4. A method that exists only for some T
Push rung 3 one notch further: the bound can gate a single method, and a value whose T doesn’t
satisfy it simply doesn’t have that method.
struct Pair<T> { first: T, second: T }
impl<T> Pair<T> {
fn new(first: T, second: T) -> Self { Self { first, second } }
}
impl<T: PartialOrd + std::fmt::Display> Pair<T> {
fn cmp_display(&self) -> String {
let largest = if self.first > self.second { &self.first } else { &self.second };
format!("the largest is {}", largest) // > from PartialOrd, {} from Display
}
}
Pair<NotDebug> is a perfectly valid, constructible type — it just has a smaller API surface.
Try to call the gated method on it and you get:
error[E0599]: the method `cmp_display` exists for struct `Pair<NotDebug>`,
but its trait bounds were not satisfied
`NotDebug: PartialOrd` is not satisfied
Note the wording: the method exists, but its bounds aren’t met. Method availability is decided
per concrete type, at the call site. This is the literal mechanism behind the Rust Book’s
cmp_display example, and behind every “why doesn’t .sum() work on my Vec<String>” question.
5. The hidden Sized bound, and ?Sized
Here is a bound you never wrote but is always there:
fn show<T>(x: &T) -> String { ... }
// really means:
fn show<T: Sized>(x: &T) -> String { ... }
Every generic parameter has an implicit T: Sized — Rust assumes types have a size known at
compile time, because that’s what you need to put them on the stack, pass them by value, etc. The
consequence bites the moment you try to use a DST (dynamically sized type) like str or [u8]:
fn show<T: std::fmt::Display>(x: &T) -> String { format!("{}", x) }
show(&42); // ok: T = i32, Sized
show("hello str"); // ERROR before the fix
error[E0277]: the size for values of type `str` cannot be known at compilation time
Why? The argument "hello str" is &str, which matches &T with T = str. But str is
unsized, and the implicit Sized rejects it. The fix is the one bound you remove rather than add:
fn show<T: std::fmt::Display + ?Sized>(x: &T) -> String { format!("{}", x) }
// ^^^^^^ opt out of the default Sized bound
?Sized means “T might not be sized.” The price: you may only touch the value behind a
pointer (&T, Box<T>, Rc<T>), never by value — because by-value needs a size. That is the
deep reason you always see &str and never bare str in a signature, and why
impl<T: Display + ?Sized> ToString for T (the impl that gives str a .to_string()) needs that
?Sized.
6. Bounds you can write only in a where clause
This rung is the concrete answer to “when do I actually need where?” Inline <T: Bound> syntax
can only attach a bound to a bare type parameter. The moment your bound is about a type
expression — T::Item, &T, Vec<T> — it has nowhere to go but a where clause.
6a — associated-type projection. You can declare <I: IntoIterator> inline, but the bound that
its items are printable is a fact about I::Item, not I:
fn join_display<I>(iter: I) -> String
where
I: IntoIterator,
I::Item: std::fmt::Display, // a projection — cannot go inline in <...>
{
iter.into_iter().map(|x| x.to_string()).collect::<Vec<_>>().join(", ")
}
6b — a higher-ranked bound on a derived type. To sum a collection by reference (without
consuming it), the capability you need is “I can iterate &C”, which is a bound on &'a C, not on
C:
fn sum_borrowed<'a, C>(collection: &'a C) -> i32
where
&'a C: IntoIterator<Item = &'a i32>, // bound on &'a C — impossible inline
{
let mut sum = 0;
for item in collection { sum += item; } // uses the &C: IntoIterator impl
sum
}
The fully general version of 6b uses a higher-ranked trait bound:
where for<'a> &'a C: IntoIterator<Item = &'a i32>— “for any lifetime,&Cis iterable.” See HRTB —for<'a>for whyfor<'a>is needed and how it differs from a single named'a. Either form proves the same point: the bound is structurally a clause about&C, and onlywhereaccepts clauses about type expressions.
7. Blanket impls — implement a trait for every qualifying type
trait Summary {
fn summary(&self) -> String;
}
impl<T: Debug> Summary for T { // ONE impl covers infinitely many types
fn summary(&self) -> String { format!("{:?}", self) }
}
After this, 42.summary(), vec![1, 2].summary(), and Point { x: 1, y: 2 }.summary() all work
with zero per-type code. This is the mechanism behind ToString (impl<T: Display + ?Sized> ToString for T) and Into (impl<T, U: From<T>> Into<U> for T — implement From, get Into
free).
The cost is coherence. Once a blanket impl covers a set of types, you cannot carve out a special case:
// uncommenting this triggers:
// error[E0119]: conflicting implementations of trait `Summary` for type `i32`
impl Summary for i32 {
fn summary(&self) -> String { format!("the int {}", self) }
}
i32 is already covered by the blanket impl, and stable Rust has no specialization, so the second
impl is an illegal overlap. This trade-off — “implement for all T: Bound” versus “exactly one impl
per (trait, type)” — is the central tension of trait design. It’s covered in depth in its own note:
Blanket impls & coherence.
8. Conditional trait impls — what #[derive] really does
A wrapper should gain a capability only when its contents have it. That’s a conditional trait
impl, and it’s literally what #[derive(PartialEq)] and #[derive(Clone)] expand to:
struct MyBox<T>(T); // no derives — hand-written below
impl<T: PartialEq> PartialEq for MyBox<T> {
fn eq(&self, other: &Self) -> bool { self.0 == other.0 }
}
impl<T: Clone> Clone for MyBox<T> {
fn clone(&self) -> Self { MyBox(self.0.clone()) }
}
The bound lives on the impl block, not on the struct. So MyBox<T> exists for any T; it only
acquires == when T: PartialEq and .clone() when T: Clone. Two consequences the ladder
checks:
MyBox<MyBox<i32>>is comparable, because the requirement recurses:MyBox<i32>: PartialEqholds becausei32: PartialEq, soMyBox<MyBox<i32>>: PartialEqholds in turn.- A
MyBoxof a non-comparable type silently lacks==— no error until you try to use it.
The one place
#[derive]is subtly wrong.#[derive(Clone)]onMyBox<T>mechanically emitsimpl<T: Clone> Clone for MyBox<T>. But if the field were anRc<T>,MyBoxwould be cloneable even whenTitself isn’t (cloning anRcjust bumps a refcount). Hand-writing the impl lets you choose a tighter or looser bound than derive’s reflexiveT: Clone. Crates likederivativeexist precisely to fix this.
Footguns
| Trap | What happens | Fix |
|---|---|---|
Bound on the struct (struct S<T: Debug>) | Every S<NonDebug> fails to construct, even when the capability is never used. | Move the bound to the impl/method that needs it (rung 3). |
Forgetting T is implicitly Sized | Passing a str/[T] gives E0277 “size cannot be known at compile time”. | Add ?Sized and take the value behind a reference (rung 5). |
Trying to bound T::Item / &T inline | Syntax error — inline bounds only attach to a bare T. | Use a where clause (rung 6). |
self-by-value method in a trait without Self: Sized | E0277 — the self parameter needs a known size. | Add where Self: Sized (seen in the capstone). |
Demanding Ord / Eq when PartialOrd / PartialEq suffices | Locks out f64 and other partially-ordered types. | Use the weakest bound the body actually needs (rung 1). |
| Special-casing one type under a blanket impl | E0119 conflicting implementations. | You can’t, on stable — design around it (rung 7). |
Real-world patterns
- Capability slicing across impl blocks.
Vec<T>,HashMap<K, V>,Option<T>all keep the type definition unbounded and attach methods to bounded impls. Mimic this in your own containers. - Blanket extension traits.
itertools::Itertoolsandtower::ServiceExtdeclare a trait with default methods plusimpl<T: Bound> Ext for T {}, instantly adding methods to every existing type. The capstone builds a miniature of this. ?Sizedin generic APIs. Functions that should accept&strand&StringtakeT: AsRef<str> + ?Sizedorimpl AsRef<str>;ToString/Borrow/Hashimpls thread?Sizedthrough so DSTs participate.- Conditional impls = how
#[derive]works. Every derivedClone/PartialEq/Debugis aimpl<T: Trait> Trait for Wrapper<T>. Reading derive output demystifies a huge amount of std.
Capstone insight
The IterExt extension trait fuses every earlier rung into the exact pattern real iterator-adapter
crates use:
trait IterExt: Iterator { // supertrait: gives access to Self::Item + iteration
fn min_max(self) -> Option<(Self::Item, Self::Item)>
where
Self: Sized, // self-by-value needs a known size
Self::Item: Ord + Copy, // per-method capability bound
{ /* fold to running (min, max); `min.zip(max)` yields None if empty */ }
fn counts(self) -> HashMap<Self::Item, usize>
where
Self: Sized,
Self::Item: Eq + Hash, // HashMap key requirements
{ /* *map.entry(item).or_insert(0) += 1 */ }
fn join_with(self, sep: &str) -> String
where
Self: Sized,
Self::Item: std::fmt::Display,
{ /* map(to_string).collect::<Vec<_>>().join(sep) */ }
}
impl<I: Iterator> IterExt for I {} // blanket impl: EVERY iterator gets all three
Three ideas snap together:
- Supertrait bound (
: Iterator) — every method body can useSelf::Itemand consumeselfby iterating. - Blanket impl (
impl<I: Iterator> IterExt for I {}) — like rung 7, this hands the methods to every iterator in the program for free. The method bodies live as defaults in the trait; the impl is empty. This is the canonical Itertools shape. - Per-method
where Self::Item:bounds — like rung 6, each adapter is callable only when the element type qualifies."abc".chars().min_max()works (char: Ord + Copy); an iterator of a non-Ordtype silently won’t offermin_max.
The aha: this is precisely how std::iter::Iterator itself is built. .sum() needs
Self::Item: Sum, .max() needs Ord, .collect::<String>() needs the right FromIterator.
Bounds aren’t bureaucracy bolted onto generics — they’re the dials that let one trait expose a
different API to every element type, decided independently at each call.
Explain it back
- Why can’t you call any methods on a bare
Twith no bounds? What can you still do with it? - A bound restricts the caller and empowers the body. Give one concrete example of each direction
from
min_item. - Why does
min_itemusePartialOrdinstead ofOrd? Which caller wouldOrdexclude? - Where should the bound go: on
struct Stack<T>or on animpl? Why is the struct almost always wrong? - What is the hidden default bound on every
<T>? What exactly does?Sizedchange, and why must a?Sizedvalue sit behind a reference? - Name two bounds that can be written only in a
whereclause, and say why inline syntax can’t express them. - What does
#[derive(Clone)]expand to forstruct MyBox<T>(T)? When is that derived bound too strict? - In
IterExt, why does each method need bothwhere Self: Sizedand awhere Self::Item: ...bound? What goes wrong without each?
See also
- Blanket impls & coherence — the E0119/orphan-rule story behind rung 7, in depth.
- Associated types vs generic params — the other axis of generic API design.
- HRTB —
for<'a>— thefor<'a> &'a C: ...bound from rung 6, fully unpacked. - Static vs dynamic dispatch — what bounds enable at monomorphization vs.
dyn Trait. - Lifetimes in depth —
'a: 'boutlives bounds are bounds too.
Blanket impls & coherence
Ladder:
src/bin/blanket_coherence.rs· Run:cargo run --bin blanket_coherence· Phase 2 · 9 rungs
TL;DR
An impl block is a fact you assert to the compiler: “this trait is implemented for this type.”
Coherence is the rule that there is exactly one such fact for any given (trait, type) pair —
never zero-ambiguity, never two conflicting answers. A blanket impl (impl<T> Trait for T) asserts
a fact about infinitely many types in one block. The orphan rule is the guardrail that stops two
different crates from each asserting conflicting facts about types neither of them owns.
Every error in this topic — E0117, E0119, E0210 — is coherence defending that “exactly one” invariant from a different angle.
Why this exists (from first principles)
Method resolution has to be deterministic and global. When you write x.into(), the compiler must
find the impl — not “an” impl, and definitely not two. Now imagine impls were unrestricted:
- Crate A does
impl Display for Vec<i32>to print[1, 2, 3]. - Crate B does
impl Display for Vec<i32>to print1 2 3. - Your program depends on both.
vec![1,2,3].to_string()now has two answers.
There is no sound way to pick. Worse, adding a dependency could silently change which impl wins, breaking code far away. Rust forbids the situation from ever being written, rather than trying to resolve it after the fact. That ban is coherence, and its crate-boundary half is the orphan rule:
To
impl SomeTrait for SomeType, at least one of{the trait, the type}must be local to your crate.
If both are foreign, you can’t write the impl — which means no two crates can both reach in and define conflicting impls for types they don’t own. The guarantee buys you: any (trait, type) pair resolves to the same impl no matter what crates are linked.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | foundations | impl<T> Named for T | One unconditional blanket impl gives every type a method. |
| 2 | foundations | impl<T: Display> Loud for T | A bound narrows the blanket to a subset of types. |
| 3 | mechanics | From → Into | Reconstruct std’s blanket: implement MyFrom, get .my_into() free. |
| 4 | mechanics | extension trait | impl<I: Iterator> IterExt for I — the itertools pattern. |
| 5 | footgun | orphan rule (E0117) | Foreign trait + foreign type is rejected; local on either side is legal. |
| 6 | footgun | overlap (E0119) | A blanket and a concrete impl that both match one type collide. |
| 7 | footgun | uncovered param (E0210) | A bare T in Self position before a local type is illegal. |
| 8 | real-world | newtype workaround | Wrap the foreign type locally, then impl the foreign trait; Deref for ergonomics. |
| 9 | capstone | sealed extension trait | A private Sealed blanket gates a public trait nobody downstream can implement. |
The ideas, built up
1. A blanket impl is one fact about infinitely many types
trait Named {
fn type_label(&self) -> &'static str;
}
impl<T> Named for T {
fn type_label(&self) -> &'static str {
"a value"
}
}
After that single block, 42i32, String::from("hi"), and your own Widget all have
.type_label(). You never wrote a per-type impl. The generic T ranges over every type that exists,
so the impl is a universally-quantified statement: “for all T, T: Named.”
This is also the first hint at why the orphan rule must exist. If a downstream crate also wrote
impl<T> Named for T, then for i32 there would be two impls — exactly the ambiguity coherence forbids.
Owning the trait Named is what lets you (and only you) make this universal claim.
2. Bounds narrow the blanket to a subset
Real blanket impls almost always carry a bound:
impl<T: Display> Loud for T {
fn loud(&self) -> String {
format!("{}!!!", self)
}
}
Now 7i32.loud() and "hi".loud() work (both are Display), but a non-Display type gets a compile
error if you call .loud() on it. The bound is doing real work: it restricts which types the universal
claim applies to. Mentally, impl<T: Display> Loud for T reads as “for all T where T: Display,
T: Loud.”
Key consequence for rung 6:
impl<T> Loud for Tandimpl<T: Display> Loud for Tcould not coexist for the same trait — everyDisplaytype would match both, and the compiler has no tiebreaker. Two different traits (NamedvsLoud) is fine, because each impl is a separate fact about a separate trait.
3. The From → Into trick (std’s most famous blanket impl)
This is the pattern in the standard library:
// std (paraphrased):
impl<T, U> Into<U> for T where U: From<T> {
fn into(self) -> U { U::from(self) }
}
You implement From, and .into() materializes for free, in the correct direction. The ladder rebuilds
it with MyFrom / MyInto so the machinery is visible:
impl<T, U> MyInto<U> for T
where
U: MyFrom<T>,
{
fn my_into(self) -> U {
U::my_from(self)
}
}
impl MyFrom<Celsius> for Fahrenheit {
fn my_from(c: Celsius) -> Fahrenheit {
Fahrenheit(c.0 * 9.0 / 5.0 + 32.0)
}
}
You write zero direct impls of MyInto — the one blanket covers every convertible pair. Note the
shape: the impl is for T (the source), with U a free type parameter pinned down by the
where-clause.
The inference gotcha. In
let f: Fahrenheit = c.my_into();, what suppliesU? Nothing incsaysFahrenheit— the type annotation does. If you’d also writtenimpl MyFrom<Celsius> for Kelvin, thenc.my_into()with no annotation is ambiguous (E0282/E0283). Coherence guarantees at most one impl per(T, U)pair; it does not pickUfor you. That’s why real.into()calls so often needlet x: Target =or a turbofish.
4. The extension trait — adding methods to types you don’t own
You can’t add an inherent method to Iterator (you don’t own it). But you can define your own trait
and blanket-impl it for everything that is an Iterator. This is exactly how itertools bolts
.chunks(), .dedup(), etc. onto every iterator:
trait IterExt: Iterator<Item = u64> { // supertrait: Self IS the iterator
fn sum_of_squares(self) -> u64
where
Self: Sized,
{
self.map(|n| n * n).sum()
}
}
impl<I: Iterator<Item = u64>> IterExt for I {} // empty body: inherits the default
One blanket impl, and vec![...].into_iter(), (1..=3), and (0..10).filter(...) all gain
.sum_of_squares() — because they’re all Iterator<Item = u64>.
Two design shapes, know both:
// Supertrait form (idiomatic, std/itertools use this):
trait IterExt: Iterator<Item = u64> { ... }
impl<I: Iterator<Item = u64>> IterExt for I {}
// Type-parameter form (works, but threads an extra param everywhere):
trait IterExt<I: Iterator<Item = u64>> { ... }
impl<I: Iterator<Item = u64>> IterExt<I> for I { ... }
The supertrait form makes Self be the iterator — no extra parameter to name in bounds. The
type-parameter form parameterizes the trait, so every bound that mentions it (fn f<T: IterExt<?>>) has
to thread the I. Prefer the supertrait.
Why does this need a separate trait? The orphan rule (next): you can’t blanket-impl a foreign trait
over all iterators, and you can’t add methods to Iterator itself. Owning IterExt is what makes the
blanket legal.
Footguns
E0117 — the orphan rule (foreign trait + foreign type)
// WRONG: Display is foreign, Vec<i32> is foreign -> E0117
impl Display for Vec<i32> { ... }
// OK: you own Summary (local trait), so a foreign type is fine
impl Summary for Vec<i32> { ... }
// OK: you own Temperature (local type), so a foreign trait is fine
impl Display for Temperature { ... }
The rule in one line: at least one of {trait, type} must be yours. The first breaks it from both
sides; the other two each satisfy it from one side. This is also why a blanket impl of a foreign trait
like impl<T> Display for T is doubly forbidden — it’s a foreign trait and it would monopolize
Display, locking every other crate out of implementing it for their own types.
E0119 — overlapping impls (no specialization on stable)
trait Kind { fn kind(&self) -> &'static str; }
// Both legal individually, both in your crate...
impl<T> Kind for T { fn kind(&self) -> &'static str { "generic" } } // (D)
impl Kind for i32 { fn kind(&self) -> &'static str { "integer" } } // (C)
// ...but i32 matches BOTH -> E0119 conflicting implementations
The instinct is “the compiler should just prefer the more specific i32 impl.” That preference is
specialization — and it is nightly-only. On stable Rust there is no tiebreaker, so two impls that
can both match one type is simply ambiguous and rejected. The fix without specialization is to not
overlap: drop the blanket and write concrete impls per type, so exactly one matches each.
Contrast with rung 3:
impl<T, U> MyInto<U> for Tnever conflicted because it was the only impl ofMyInto. Overlap requires two impls of the same trait both covering one type.
E0210 — the uncovered type parameter (the subtle one)
The orphan rule is not just “some type must be local” — it’s about order and coverage. Scanning
Self, then the trait’s type arguments left-to-right, a local type must appear before any bare
(uncovered) type parameter.
- A bare
Tis uncovered. - A
Twrapped in your local type, likeWrapper<T>, is covered.
use std::ops::Add;
struct Meters(f64);
// WRONG: Add is foreign; Self is a BARE T (uncovered), and the only local
// type `Meters` appears AFTER it as Rhs -> E0210
impl<T> Add<Meters> for T { ... }
// OK: local type is in the Self position, first
impl Add for Meters { type Output = Meters; ... }
// OK: From is foreign, but T is COVERED by your local Wrapped<T>
impl<T> From<T> for Wrapped<T> { ... }
Why the asymmetry? impl<T> Add<Meters> for T claims Add<Meters> for types you don’t own — so the
crate that owns some Foo could legitimately add impl Add<Meters> for Foo, and now Foo has two impls
the compiler can’t see across crates. impl<T> From<T> for Wrapped<T> only ever claims From for your
Wrapped, and the orphan rule stops anyone else from impl’ing From<…> for Wrapped<…>. The covered case
is collision-proof; the uncovered one is a future-collision waiting to happen, so it’s banned.
Real-world patterns
The newtype workaround
Rung 5 showed impl Display for Vec<i32> is illegal. The standard escape hatch: wrap the foreign type
in your own local newtype, then impl the foreign trait for the newtype. Now one side is local — legal.
struct Wrapper(Vec<i32>);
impl std::fmt::Display for Wrapper { // legal: Wrapper is local
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
let parts: Vec<String> = self.0.iter().map(|n| n.to_string()).collect();
write!(f, "[{}]", parts.join(", "))
}
}
impl std::ops::Deref for Wrapper { // restore the inner type's methods
type Target = Vec<i32>;
fn deref(&self) -> &Vec<i32> { &self.0 }
}
The cost of a newtype is that you lose the inner type’s methods; Deref buys them back via deref
coercion, so w.len(), w.iter(), w.first() all work.
Derefis fine here, but don’t abuse it. A transparent wrapper that exposes everything is the right use. But if the newtype exists to enforce an invariant (aSortedVec, aNonEmptyVec),Derefleaks the inner type’s mutators (push,clear) and lets callers break the invariant behind your back.Derefshould mean “is-a smart pointer to,” not “has-a field I’m exposing.” For restricting wrappers, expose a curated API instead.
Capstone insight: sealing a trait with a private blanket impl
The capstone ships a tiny stats library: a StatsExt extension trait that adds .mean() and
.variance() to any Iterator<Item = f64> via a blanket impl — and then seals it so downstream code
can use the methods but can never implement the trait.
mod sealed {
pub trait Sealed {}
impl<I: Iterator<Item = f64>> Sealed for I {} // the ONLY impl of Sealed
}
trait StatsExt: Iterator<Item = f64> + sealed::Sealed {
fn mean(self) -> f64;
fn variance(self) -> f64;
}
impl<I: Iterator<Item = f64>> StatsExt for I {
fn mean(self) -> f64 { /* collect to Vec<f64>, average; empty -> 0.0 */ }
fn variance(self) -> f64 { /* mean, then average of (x - mean)^2 */ }
}
The “aha” is how coherence makes the seal unbreakable:
Sealedispubinside a private module — outside code literally cannot name it.- The only impl of
Sealedis your blanket impl. Coherence means no one else can add another. StatsExtrequiresSealedas a supertrait. So to writeimpl StatsExt for MyType, a downstream crate would also needMyType: Sealed— which they can neither name nor satisfy.
The result: a public trait that is fully usable but closed to implementation. This is the production
pattern std uses to keep traits like Error-adjacent helpers (and many crate APIs) extensible internally
while presenting a stable, non-overridable surface. The blanket impl of a private trait is the gate;
coherence is the lock.
Explain it back
Future-you should be able to answer these cold:
- Why does
vec![1,2,3].to_string()having “two answers” have to be a compile error rather than a runtime choice? - State the orphan rule in one sentence. Which of
{trait, type}is local inimpl Display for Wrapper? - Why can’t
impl<T> Kind for Tandimpl Kind for i32coexist on stable Rust? What single nightly feature would make it work, and what would it do? - In
let f: Fahrenheit = c.into(), what supplies the target type? When does omitting the annotation become a hard error? - Why is
impl<T> Add<Meters> for T(E0210) a future-collision risk, butimpl<T> From<T> for Wrapped<T>is not? Define “covered.” - When is
Derefon a newtype the right call, and when does it actively break your type’s guarantees? - In the sealed-trait capstone, name the three things that together make
impl StatsExt for MyTypeimpossible downstream.
See also
- Associated types vs generic params — the other half of “designing a trait”:
type Itemvs<T>, and where E0119 also shows up. - Conversion traits —
From/Into,TryFrom, the orphan rule and reflexivity in the conversion setting. - HRTB — for<’a> — the
DecodeOwned: for<'de> Decode<'de>pattern is another supertrait-based bound, like the sealed-trait supertrait here.
Static vs dynamic dispatch
Ladder:
src/bin/dispatch.rs· Run:cargo run --bin dispatch· Phase 2 · 9 rungs
TL;DR
When you call a trait method, which concrete implementation runs has to be decided somewhere. Rust gives you two places to decide it:
- Static dispatch (
<T: Trait>,impl Trait): the compiler knows the concrete type at the call site. It stamps out a specialized copy of the code per type (monomorphization) and can inline. Fast, zero indirection — but code size grows and the set of types is fixed at compile time. - Dynamic dispatch (
dyn Trait): the concrete type is erased behind a fat pointer(data, vtable). The method is looked up at runtime through the vtable. One copy of the code, and it unlocks runtime flexibility (heterogeneous collections, types chosen by runtime values) — but each call is an indirection that usually can’t be inlined.
Every design choice in this area is picking which side of that trade you want. And
there’s a third option for closed sets — an enum + match — that gets much of
the best of both.
Why this exists (from first principles)
A trait is a promise: “this type has a hello() method.” But hello() for
English and hello() for French are different functions at different machine
addresses. When you write g.hello(), the generated code needs an address to jump
to. The whole topic is: how does the compiler find that address, and when?
Two answers:
-
At compile time. If the compiler can see the concrete type of
gright here, it just bakes in the correct address. To make that true for generic code, it duplicates the function once per concrete type used — monomorphization. Calls become direct, inlinable, free. -
At runtime. If the concrete type isn’t known until the program runs (you chose it from user input, or you stuffed many different types into one list), the compiler can’t bake an address in. Instead it attaches a vtable — a little table of function pointers — to the value, and emits “load the address out of the vtable, then call it.” That indirection is the cost of not knowing the type until runtime.
Neither is “better.” They solve different problems, and a lot of Rust API design is about recognizing which problem you have.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | foundations | stamp_vs_dyn | Same method through <T: Trait> vs &dyn Trait |
| 2 | foundations | impl_trait | impl Trait in arg position (sugar) vs return position (one type) |
| 3 | mechanics | monomorph_proof | type_name::<T>() proves a separate copy is stamped per type |
| 4 | footgun | return_branch | Returning 1 of 2 types: impl Trait fails, Box<dyn> works |
| 5 | footgun | hetero_collection | Vec<T> can’t mix types; Vec<Box<dyn>> can |
| 6 | footgun | returns_self | -> Self: fine under generics, forbidden behind dyn |
| 7 | real-world | closure_pipeline | One closure = generic F: Fn; many = Vec<Box<dyn Fn>> |
| 8 | real-world | enum_dispatch | Closed-set third way: enum + match, inline, no vtable |
| 9 | capstone | pipeline_both_ways | Same pipeline three ways — static, dynamic, enum — one result |
The ideas, built up
1. The same method, two dispatch strategies
Start with one trait and write the identical logic twice — once generic, once dyn:
trait Greet {
fn hello(&self) -> String;
}
fn greet_static<T: Greet>(g: &T) -> String { g.hello() } // monomorphized per T
fn greet_dynamic(g: &dyn Greet) -> String { g.hello() } // one fn, vtable lookup
The bodies are byte-for-byte the same. The difference is invisible in the source and lives entirely in how the call compiles:
greet_staticis generic. The compiler produces a distinct machine-code copy forEnglishand another forFrench. Each call jumps straight to a known address.greet_dynamicis one function.&dyn Greetis a fat pointer(data_ptr, vtable_ptr), andg.hello()reads the method address out of the vtable at runtime.
That second form is what lets you do this — pick the concrete type at runtime and still have a single static type for the variable:
let who: &dyn Greet = if condition { &en } else { &fr };
greet_dynamic(who);
2. impl Trait means two different things by position
impl Trait is one syntax with two opposite meanings depending on where it appears:
fn loudest(g: impl Greet) -> String { g.hello().to_uppercase() } // ARGUMENT
fn default_greeter() -> impl Greet { French } // RETURN
- Argument position is pure sugar for a generic bound:
fn loudest(g: impl Greet)is exactlyfn loudest<T: Greet>(g: T). Static dispatch, monomorphized per call site. The caller picks the type. - Return position means “I return one specific concrete type that I’m not
naming.” Still static — the compiler knows the real type (
French), the caller just can’t name it. The callee picks, and it must be a single type across all return paths.
That “single type” rule is quiet here but becomes a wall in rung 4.
Scaffolding note from the file: a bare
todo!()in a-> impl Greetfunction won’t even compile. The inferred return type would be!, and!: Greetis false. Return-positionimpl Traitdemands a real concrete type at compile time.
3. Seeing monomorphization with your own eyes
“Monomorphization” stays abstract until you prove it. This function reports the name of its own type parameter, with no value argument at all:
fn tag<T>() -> &'static str { std::any::type_name::<T>() }
tag::<English>(); // ".../dispatch::English"
tag::<i32>(); // "i32"
tag::<Vec<String>>(); // "alloc::vec::Vec<alloc::string::String>"
If there were only one compiled tag, it couldn’t possibly know which T it was
called with — it takes no runtime input. It knows because the compiler stamped a
separate copy of tag per T, each with its own type name baked in. That’s
monomorphization made visible. A &dyn parameter erases the type, so a single
function literally cannot recover it this way.
4. The first wall: a type chosen at runtime
Now ask for one of two types based on a runtime flag:
// WRONG — does not compile:
fn broken_pick(french: bool) -> impl Greet {
if french { French } else { English } // `if` and `else` have incompatible types
}
-> impl Greet promised one concrete type, but the type now depends on a runtime
value. There is no single type the compiler can fill in. Static dispatch is out of
road.
The fix is to erase the type behind a trait object, giving both branches the
same type — Box<dyn Greet>:
// OK:
fn pick_greeter(french: bool) -> Box<dyn Greet> {
if french { Box::new(French) } else { Box::new(English) }
}
The cost: a heap allocation plus a vtable lookup per .hello(). The payoff: a type
decided at runtime. The moment the type is a runtime decision, you reach for
dyn.
5. Heterogeneous collections: the headline feature of dyn
A Vec<T> is monomorphic — every element is the exact same T:
// WRONG — different types in one Vec:
let bad = vec![English, French]; // expected `English`, found `French`
To hold a mixed bag of “things that implement Greet,” erase each element:
// OK:
fn build_crowd() -> Vec<Box<dyn Greet>> {
vec![Box::new(English), Box::new(French), Box::new(English)]
}
Now every slot has the same type — a fat pointer — even though the values underneath differ. A list of differently-typed things behind one shared interface is simply impossible with pure static dispatch. This is the single biggest thing dynamic dispatch buys you.
6. The mirror: -> Self is the thing only static can do
Rung 5 showed what dyn can do that generics can’t. Rung 6 is the reverse — a trait
generics handle fine but that cannot become a dyn at all:
trait Doubler {
fn doubled(&self) -> Self; // returns Self -> NOT object-safe
}
fn twice<T: Doubler>(x: T) -> T { x.doubled() } // totally fine
// WRONG — does not compile:
let obj: Box<dyn Doubler> = Box::new(21_i32);
// "the trait `Doubler` cannot be made into an object because method `doubled`
// references the `Self` type"
Why? A dyn Doubler erases the concrete type, but doubled(&self) -> Self returns
a value of that erased type. A vtable can’t describe “returns something whose
size and layout is the type we just threw away.” Under <T: Doubler>, the concrete
type is known at each instantiation, so -> Self is no problem.
This is exactly why there is no
dyn Clone:clone(&self) -> SelfreferencesSelfby value. Generic methods and by-valueSelfare the other common object-safety blockers.
So rungs 5 and 6 bracket the trade:
| Static dispatch can… | Dynamic dispatch can… |
|---|---|
Return Self, take Self by value | Store mixed types in one collection |
| Have generic methods | Choose the concrete type at runtime |
| Inline, monomorphize | Keep code size flat (one copy) |
7. Closures: where everyone meets this decision
Every closure has its own unique, unnameable type — even two closures with identical signatures are different types. So the dispatch choice shows up the moment you handle closures:
fn apply_static<F: Fn(i32) -> i32>(f: F, x: i32) -> i32 { f(x) } // one closure, inlined
// WRONG — two closures, two different types, one Vec:
let steps = vec![|x: i32| x + 1, |x: i32| x * 2];
To store many closures together (a callback registry, an event table, a pipeline), erase them:
fn build_pipeline(add: i32) -> Vec<Box<dyn Fn(i32) -> i32>> {
vec![
Box::new(|x| x + 1),
Box::new(|x| x * 2),
Box::new(move |x| x + add), // captures `add` -> distinct type again
]
}
fn run_pipeline(steps: &[Box<dyn Fn(i32) -> i32>], start: i32) -> i32 {
steps.iter().fold(start, |acc, step| step(acc))
}
The everyday rule: take a closure → generic F: Fn (fast, inlined); store a
collection of closures → Box<dyn Fn> (flexible, one indirection each). It’s
exactly why Iterator::map is generic but a vector of event handlers is boxed.
8. Enum dispatch: the closed-set third way
dyn buys heterogeneity but costs an allocation and a vtable hop per element.
Generics are free but can’t store mixed types. When your set of types is closed
(you know all of them at compile time), an enum + match gets most of both:
enum Shape {
Circle { r: f64 },
Rect { w: f64, h: f64 },
}
impl Shape {
fn area(&self) -> f64 {
match self {
Shape::Circle { r } => std::f64::consts::PI * r * r,
Shape::Rect { w, h } => w * h,
}
}
}
A Vec<Shape> holds circles and rects — heterogeneous like rung 5 — but:
- No
Box, no heap allocation per element. Each value lives inline in the Vec. - Static dispatch.
matchcompiles to a jump on the discriminant; arms can inline. No vtable pointer chase. - The trade: the set is closed. Adding a variant means editing the enum and
every
match(the compiler enforces exhaustiveness — a feature here). And every element is sized to the largest variant.
The size contrast is concrete and worth internalizing:
std::mem::size_of::<Box<dyn Greet>>(); // 16 — a fat pointer (data + vtable)
std::mem::size_of::<Shape>(); // 24 — 16-byte Rect payload + discriminant,
// rounded up to 8-byte alignment
This is why serde_json::Value, AST nodes, and state machines are enums, not
Vec<Box<dyn Node>> — and what the enum_dispatch crate automates.
Footguns
| Trap | What you see | Fix |
|---|---|---|
| Return one of two types by a runtime flag | if and else have incompatible types | Erase to Box<dyn Trait> |
Mixed concrete types in one Vec<T> | expected A, found B | Vec<Box<dyn Trait>> (or an enum) |
Box<dyn Trait> where the trait has -> Self / generic method | “cannot be made into an object” | Keep it generic, or split the method behind where Self: Sized |
Vec of two same-signature closures | different closure types | Box them as dyn Fn, or use one generic F |
Bare todo!() in a -> impl Trait fn | !: Trait is not satisfied | Return a real concrete value |
Real-world patterns
Iterator::map,Option::map, sort keys takeF: FnMut(...)— generic, so the closure inlines and the iterator pipeline fuses to tight code.- Plugin / handler registries are
HashMap<String, Box<dyn Handler>>orVec<Box<dyn Fn(...)>>— the set of handlers isn’t known at compile time, so the type must be erased. Box<dyn Error>is dynamic dispatch for the same reason: a function can fail in many ways and you want one return type.serde_json::Value, syntax trees, VM opcodes, state machines are enums — closed sets where inline storage and exhaustivematchwin.- Returning iterators/futures uses
-> impl Iterator/-> impl Future: static, no allocation, the concrete (often unnameable) type stays hidden.
A useful decision tree:
Is the set of types closed and known at compile time? → enum + match. Is it open, or chosen at runtime, or a heterogeneous collection? →
dyn. Is it a single type flowing through generic code? →<T>/impl Trait.
Capstone insight
The capstone builds the same pipeline — Add(3) → Mul(2) → Neg — three ways and
proves they compute the same result (-16 for input 5):
// (A) STATIC: the whole pipeline is ONE type, fully inlinable, shape fixed forever.
struct Compose<A, B>(A, B);
impl<A: Transform, B: Transform> Transform for Compose<A, B> {
fn apply(&self, x: i32) -> i32 { self.1.apply(self.0.apply(x)) }
}
fn run_static(start: i32) -> i32 {
Compose(Add(3), Compose(Mul(2), Neg)).apply(start) // type: Compose<Add, Compose<Mul, Neg>>
}
// (B) DYNAMIC: pipeline assembled at runtime, any length/order; box + vtable per stage.
fn run_dynamic(start: i32) -> i32 {
let pipe: Vec<Box<dyn Transform>> =
vec![Box::new(Add(3)), Box::new(Mul(2)), Box::new(Neg)];
pipe.iter().fold(start, |acc, t| t.apply(acc))
}
// (C) ENUM: runtime-built like (B), closed set, inline storage, match dispatch.
fn run_enum(start: i32) -> i32 {
let pipe = vec![Op::Add(3), Op::Mul(2), Op::Neg];
pipe.iter().fold(start, |acc, op| op.apply(acc))
}
The “aha” is in the static version’s type: Compose<Add, Compose<Mul, Neg>>. The
entire pipeline — its stages and their order — is encoded in the type itself. That’s
why the compiler can inline it end to end and allocate nothing… and also why its
shape is frozen at compile time. The dynamic and enum versions move that structure
out of the type and into runtime data (a Vec), trading inlinability for the
freedom to build the pipeline on the fly. Same computation, three encodings of
“where does the structure live: in the type, or in the data?”
Explain it back
- What does monomorphization actually duplicate, and how would you prove it happened without looking at assembly?
impl Traitin argument vs return position — who picks the concrete type in each, and what’s the one-type constraint on the return form?- Why does returning one of two types by a runtime flag force
Box<dyn>? - Name two things a trait can have that make it not object-safe, and say why a vtable can’t express them.
- A closure captures a variable. Why does that change its type, and why does it
matter for putting closures in a
Vec? - You have a fixed set of message types to dispatch on. Why might an
enumbeat bothVec<Box<dyn Msg>>and a generic? What do you give up? - In the capstone, where does the pipeline’s “structure” live in each of the three versions?
See also
- Associated types vs generic params — the other axis of
trait design, and where object safety (
dyn Iterator<Item=…>) reappears. - Blanket impls & coherence — how
impl Trait for Tinteracts with the monomorphized world. Box& the heap —Box<dyn Trait>and fat pointers up close.
Closures & Fn / FnMut / FnOnce
Ladder:
src/bin/closures.rs· Run:cargo run --bin closures· Phase 2 · 9 rungs
TL;DR
A closure is an anonymous struct the compiler generates for you. Its fields are the variables it captures from the surrounding scope. How it captures them — by shared reference, by mutable reference, or by value — decides which of three traits it implements:
| Trait | Receiver | Meaning | Callable |
|---|---|---|---|
Fn | &self | only reads captures | many times, shareably |
FnMut | &mut self | mutates captures | many times, exclusively |
FnOnce | self | consumes captures | exactly once |
These nest: Fn ⊂ FnMut ⊂ FnOnce. Anything that’s Fn is automatically FnMut
and FnOnce too. Once you internalize “closure = struct + a call method whose
self-type is the trait”, every confusing closure error becomes decodable.
Why this exists (from first principles)
A plain function can’t remember anything between the place it’s defined and the
place it’s called — it only has its arguments. But constantly we want a “function
plus some context”: multiply by this factor, push into this log, validate
against this min..=max. That context has to live somewhere.
The closure’s answer: bundle the captured context into a hidden struct, and attach a call method to it. So this:
let factor = 3;
let times = |x| x * factor;
is conceptually compiled to:
struct __Times { factor: i32 } // captured env becomes fields
impl __Times {
fn call(&self, x: i32) -> i32 { x * self.factor }
}
let times = __Times { factor: 3 };
Now the only remaining question is what kind of access the call method needs to
its captured fields — and that is exactly what Fn/FnMut/FnOnce encode. The
compiler is enforcing the same borrow rules it always does, just on hidden fields:
- If the body only reads a capture,
&selfsuffices →Fn. - If the body writes a capture, it needs
&mut self→FnMut. - If the body moves a capture out (consumes it), it needs
selfby value →FnOnce.
Everything in this concept is a consequence of that one design choice.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | foundations | capture env by & | a closure reads outer variables without being passed them |
| 2 | foundations | three capture modes | & / &mut / move — the compiler picks the least invasive |
| 3 | mechanics | the trait hierarchy | Fn ⊂ FnMut ⊂ FnOnce; strictest vs loosest bound |
| 4 | mechanics | desugar by hand | build the struct + call/call_mut the compiler generates |
| 5 | footgun | once & mut | FnOnce is callable once (E0382); FnMut needs a mut binding (E0596) |
| 6 | footgun | returning closures | impl Fn (one type) vs Box<dyn Fn> (branchy) |
| 7 | real-world | fn pointers | fn items & non-capturing closures coerce to fn; capturing ones can’t |
| 8 | real-world | stdlib + factories | which Fn trait each adapter wants; closures that build closures |
| 9 | capstone | event dispatcher | Vec<Box<dyn FnMut(&Event)>> registry with subscribe/emit |
The ideas, built up
1. Capture: a closure reaches into its environment
A closure can use a variable that is neither a parameter nor a local — it reads it straight out of the enclosing scope:
fn make_and_use(factor: i32, nums: &[i32]) -> Vec<i32> {
nums.iter().map(|x| x * factor).collect()
// ^^^^^^^^^^^^^^ closure captures `factor`; never passed in
}
factor is captured. x is a parameter. That distinction is the whole point: the
captured value becomes hidden state, the parameter stays an argument. (Side note:
x here is &i32, yet x * factor compiles because std implements Mul for
reference operands and auto-derefs.)
2. The three capture modes
The compiler captures as weakly as the body allows. It tries & first, then
&mut, then by-value — picking the first that makes the body type-check. You can
force by-value with the move keyword.
// (a) read-only -> captured by & -> the closure is `Fn`
fn borrow_capture(data: &Vec<i32>) -> i32 {
data.iter().sum() // only reads `data`; caller can still use it after
}
// (b) mutating -> captured by &mut -> the closure is `FnMut`
fn mut_capture() -> Vec<String> {
let mut log: Vec<String> = Vec::new();
let mut record = |s: String| log.push(s); // note: the *binding* is `mut`
record("event 0".to_string());
record("event 1".to_string());
record("event 2".to_string());
log // borrow of `log` has ended; we can return it
}
// (c) move -> captured by value -> owns the data; can outlive the scope
fn move_capture(owned: String) -> Box<dyn Fn() -> usize> {
Box::new(move || owned.len())
}
Two subtleties this rung surfaces:
- In
(b), the closure holds a&mut log. You cannot readlogwhile that borrow is live — you must finish allrecord(...)calls first, then returnlog. The mutable borrow is released when the closure is last used. - In
(c),moveforcesownedinto the closure by value, which is what lets the returned closure outlivemove_capture’s stack frame. Note its bound isFn, notFnOnce: reading.len()doesn’t consume theString, so it’s re-callable.
movechanges how captures are taken, not which trait results. Amoveclosure that only reads its captures is stillFn.moveanswers “by value?”, the body answers “read, write, or consume?”.
3. The trait hierarchy: strictest vs loosest bound
The three traits form a subtrait chain:
Fn : FnMut : FnOnce
Read it as: every Fn is a FnMut, every FnMut is a FnOnce. Three generic
helpers make the consequence concrete:
fn apply_fn<F: Fn() -> i32>(f: F) -> i32 { f() + f() } // call via &self
fn apply_mut<F: FnMut() -> i32>(mut f: F) -> i32 { f() + f() } // call via &mut self
fn apply_once<F: FnOnce() -> i32>(f: F) -> i32 { f() } // call via self, once
A pure read-only closure satisfies all three bounds:
let read = || 7;
apply_fn(read); // ok
apply_mut(read); // ok — Fn is also FnMut
apply_once(read); // ok — Fn is also FnOnce
But a closure that mutates a capture is only FnMut (and FnOnce), never Fn:
let mut n = 0;
let counter = move || { n += 1; n };
apply_mut(counter); // ok
// apply_fn(counter); // E0525: expected a closure implementing `Fn`, found `FnMut`
And one that consumes a capture is only FnOnce:
let s = String::from("rust");
let consume = move || s.len() as i32; // here just reads, but if it moved `s` out...
apply_once(consume); // ...only `apply_once` would accept it
This is the key API-design lesson:
F: Fnis the strictest bound (fewest closures qualify),F: FnOnceis the loosest (most closures qualify). Demand the least power you actually need: if you call the closure once, takeFnOnce; if you call it repeatedly without mutation, you can afford to demandFn. The looser the bound, the more callers can hand you a closure.
4. Desugar by hand — the “aha” rung
The mental model stops being a metaphor when you build the struct yourself. The
real Fn traits are unstable to implement directly on stable Rust, so the ladder
mirrors them with inherent methods carrying the same self-type:
// `move |x: i32| x + offset` desugars to:
struct AddOffset { offset: i32 } // one captured field
impl AddOffset {
fn call(&self, x: i32) -> i32 { x + self.offset } // &self -> mirrors Fn
}
// `move || { count += step; count }` desugars to:
struct Counter { count: i32, step: i32 } // two captured fields
impl Counter {
fn call_mut(&mut self) -> i32 { // &mut self -> mirrors FnMut
self.count += self.step;
self.count
}
}
The check runs each hand-built struct next to the equivalent real closure and asserts identical output:
let offset = 100;
let hand = AddOffset { offset };
let real = move |x: i32| x + offset;
for x in [-5, 0, 7, 42] {
assert_eq!(hand.call(x), real(x)); // identical
}
The payoff: AddOffset.call(x) and |x| x + offset are the same thing. And the
receiver type on the call method is the trait — &self ⇒ Fn, &mut self ⇒
FnMut, self ⇒ FnOnce. Memorize this and you never have to guess a closure’s
trait again: ask “what does the body do to its captures, and what self would the
hidden call method need?”
Footguns
FnOnce is callable exactly once (E0382)
A closure that moves a captured value out of itself can only run once — the second call would touch a value that’s already been moved away:
fn unwrap_factory(s: String) -> impl FnOnce() -> String {
move || s // hands `s` back by value -> consumes the capture
}
let f = unwrap_factory(String::from("payload"));
assert_eq!(f(), "payload");
// let again = f(); // E0382: use of moved value `f` — the call consumed it
impl FnOnce is the only valid return bound here. Try widening it to impl Fn
and the compiler refuses, because returning s by value can’t be done through
&self.
FnMut needs a mut binding (E0596)
Calling an FnMut goes through &mut self, so the variable (or parameter) holding
the closure must be mutable:
fn run_n_times<F: FnMut() -> i32>(mut f: F, n: usize) -> Vec<i32> {
// ^^^ delete this and you get E0596:
// "cannot borrow `f` as mutable"
let mut results = Vec::new();
for _ in 0..n { results.push(f()); }
results
}
This is the same rule as rung 2’s let mut record = .... A FnMut call mutates
hidden fields, so it needs mutable access to the closure value.
Every closure has a unique, unnameable type
Two closures with identical bodies are still different types. You literally cannot
write the type out — so fn foo() -> { the closure type } is impossible. That forces
the return-position decision in rung 6.
Real-world patterns
Returning closures: impl Fn vs Box<dyn Fn>
// ONE concrete hidden type -> impl Trait. Static dispatch, zero allocation.
fn make_adder(n: i32) -> impl Fn(i32) -> i32 {
move |x| x + n
}
// Different branches build DIFFERENT closure types -> must erase behind a vtable.
fn pick_op(op: char) -> Box<dyn Fn(i32, i32) -> i32> {
match op {
'+' => Box::new(|a, b| a + b),
'-' => Box::new(|a, b| a - b),
'*' => Box::new(|a, b| a * b),
_ => Box::new(|_, _| 0),
}
}
impl Fn means “I return exactly one hidden type, you just don’t get to name it.”
The moment your match arms return distinct closure types, no single impl Trait
type can cover them — you box them into Box<dyn Fn>, a heap-allocated trait object
with dynamic dispatch. Trying -> impl Fn on pick_op yields “if and else have
incompatible types”.
fn pointers vs closures
fn(i32) -> i32 is the function pointer type: one pointer-sized value aimed at
code, with no captured environment. Two things coerce to it:
fn triple(x: i32) -> i32 { x * 3 }
fn transform_all(xs: &[i32], f: fn(i32) -> i32) -> Vec<i32> {
xs.iter().map(|x| f(*x)).collect()
}
transform_all(&[1, 2, 3], triple); // a function ITEM coerces
transform_all(&[1, 2, 3], |x| x + 100); // a NON-capturing closure coerces
// fn pointers are Copy + Sized -> store them inline, no Box, no vtable:
let ops: Vec<fn(i32) -> i32> = vec![triple, |x| x + 1, |x| x * x];
// But a CAPTURING closure does NOT coerce:
let k = 10;
// transform_all(&[1], |x| x + k); // E0308: expected fn pointer, found closure
The dividing line: the instant a closure captures anything, it becomes a distinct
type carrying data and is no longer a bare fn. Note Vec<fn(..)> (rung 7) needs no
Box, unlike Vec<Box<dyn Fn>> (rung 6) — because all fn pointers share one
Copy, Sized type, whereas capturing closures don’t.
The stdlib demands the loosest bound it can
nums.iter().filter(...).map(...) // map / filter / fold take FnMut
rows.sort_by_key(|&(_, n)| n); // sort_by_key takes FnMut
opt.unwrap_or_else(|| default()); // unwrap_or_else takes FnOnce (runs at most once)
Each method asks for exactly the power it uses. And the closure factory is the everyday production pattern — a function that builds and returns a closure capturing its arguments:
fn make_validator(min: i32, max: i32) -> impl Fn(i32) -> bool {
move |x| x >= min && x <= max
}
let valid = make_validator(1, 10);
let kept: Vec<i32> = (0..15).filter(|&n| valid(n)).collect(); // reused many times
make_validator returns an Fn (it only reads min/max), so the resulting
closure is freely re-callable inside filter.
Capstone insight
The event dispatcher fuses every thread of the ladder into one small machine:
struct Dispatcher {
subscribers: Vec<Box<dyn FnMut(&Event)>>,
}
impl Dispatcher {
fn new() -> Self { Self { subscribers: Vec::new() } }
fn subscribe<F: FnMut(&Event) + 'static>(&mut self, f: F) {
self.subscribers.push(Box::new(f)); // generic F -> erased trait object
}
fn emit(&mut self, event: &Event) {
for subscriber in self.subscribers.iter_mut() {
subscriber(event); // &mut access -> FnMut call
}
}
}
Every type choice here is forced by the ladder:
Box<dyn FnMut(&Event)>— subscribers are closures of different types with different captures, so they can’t share a genericF; they must be type-erased behind a trait object (rung 6).FnMut, notFn— a real handler keeps internal mutable state (a counter, a buffer). ChoosingFnMutlets handlers mutate their captures; choosingFnwould forbid the most useful handlers (rung 3’s “demand the least power you need”, here meaning the least that still allows mutation).emit(&mut self)+iter_mut()— calling anFnMutneeds&mut self, which needs&mutaccess to each boxed handler (rung 5’s E0596 in its natural habitat).F: FnMut(&Event) + 'static—subscribeaccepts any matching closure (generic), and'staticguarantees the boxed handler can outlive the call.
The handlers in the test capture an Rc<RefCell<...>> purely so the test can peek
at what they did — the dispatcher itself owns the closures outright. Handler A
mutates a captured seen counter, which is precisely what makes the whole registry
have to be FnMut rather than Fn.
Explain it back
Future-you should be able to answer these cold:
- What hidden data structure is a closure, and what are its fields?
- Given a closure body, how do you predict whether it’s
Fn,FnMut, orFnOnce? (Hint: whatself-type would the generated call method need?) - Why is
F: FnOncethe loosest bound andF: Fnthe strictest? Which should a callback API ask for, and why? - What does
movechange — and what does it not change about a closure’s trait? - When must a returned closure be
Box<dyn Fn>instead ofimpl Fn? - Why does a non-capturing closure coerce to
fn(..)but a capturing one doesn’t? - In the dispatcher, why must
subscribersbeVec<Box<dyn FnMut(_)>>andemittake&mut self?
See also
- Static vs dynamic dispatch —
impl TraitvsBox<dyn Trait>, the same monomorphization-vs-vtable trade-off the closure-return rung hits. Rc<RefCell<T>>patterns — the shared mutable state the capstone’s handlers use for observation.- HRTB — for<’a> — closures over references (
Fn(&T)) and why their bounds need higher-ranked lifetimes. - Iterators end-to-end — where
map/filter/foldconsume the closures built here.
impl Trait & RPIT
Ladder:
src/bin/impl_trait.rs· Run:cargo run --bin impl_trait· Phase 2 · 9 rungs
TL;DR
impl Trait means “some single concrete type that implements this trait, chosen at
this position.” The one question that decides everything is who picks the type?
| Position | Syntax | Who picks | Desugars to |
|---|---|---|---|
| Argument (APIT) | fn f(x: impl Trait) | the caller | an anonymous generic param <T: Trait> |
| Return (RPIT) | fn f() -> impl Trait | the callee | one hidden concrete type the compiler knows but you can’t name |
Everything else — the turbofish footgun, “all branches must be one type,” lifetime
capture, async fn desugaring, RPITIT — falls out of those two facts.
Why this exists (from first principles)
Some types cannot be written down. A closure has an anonymous, compiler-generated
type. An iterator chain like (0..n).filter(...).map(...) has a type like
Map<Filter<Range<u32>, {closure}>, {closure}> where {closure} is unnameable. Before
impl Trait, the only way to return one of these was to erase it behind a
Box<dyn Trait> — heap allocation plus a vtable on every call.
impl Trait in return position fixes this: you promise the caller “this is some
Iterator<Item=u32>,” the compiler fills in the real type behind the scenes, and you
get a by-value, monomorphized return with zero overhead — no box, no vtable.
In argument position it is pure ergonomics: fn f(x: impl Display) reads better than
fn f<T: Display>(x: T), and it is exactly the same thing after desugaring — with one
consequence (you lose the turbofish).
The deepest payoff: async fn is built entirely on RPIT. async fn f() -> T is
sugar for fn f() -> impl Future<Output = T>. Understanding RPIT is understanding how
async functions return their state machines.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | foundations | APIT basics | impl Display arg = sugar for <T: Display>; caller picks |
| 2 | foundations | RPIT basics | return impl Iterator; the real type is unspellable |
| 3 | mechanics | turbofish footgun | APIT == generic, but impl-arg has no name to turbofish |
| 4 | mechanics | the killer app | return a closure & an adapter chain — no Box, no vtable |
| 5 | footgun | one hidden type | if/else two iterators won’t compile (E0308); fix 3 ways |
| 6 | footgun | lifetime capture | edition-2024 auto-capture + + use<> opt-out (E0597) |
| 7 | real-world | async fn IS RPIT | async fn ≡ -> impl Future; the Send question |
| 8 | real-world | RPITIT | impl Trait in trait returns; async-fn-in-trait; not dyn-safe |
| 9 | capstone | combinator toolkit | RPIT/APIT/RPITIT everywhere, Box<dyn> only where forced |
The ideas, built up
1. Argument position: the caller picks (APIT)
fn describe(x: impl Display) -> String {
format!("[{x}]")
}
This is identical, after desugaring, to:
fn describe<T: Display>(x: T) -> String { format!("[{x}]") }
The same function body serves describe(42), describe("hi"), and describe(3.5) —
three different concrete types, each chosen by the caller at the call site. “APIT”
(argument-position impl Trait) is just an anonymous generic parameter.
2. Return position: the callee picks (RPIT)
fn evens_up_to(n: u32) -> impl Iterator<Item = u32> {
(0..n).filter(|x| x % 2 == 0)
}
Now the direction flips. The function body decides the concrete type, and the caller
only knows the interface (Iterator<Item = u32>). The real type is something like
Filter<Range<u32>, {closure}> — you literally cannot write it in the signature,
because the closure type has no name. That impossibility is the entire reason RPIT
exists. The caller has to .collect() (or otherwise consume it) to get back to a type
it can name.
Mental model: RPIT is an existential type — “there exists one type
T: Iteratorand I’m returning it, but I’m hiding which one.” APIT is a universal type — “for allT: Traitthe caller chooses.”
3. The turbofish footgun
APIT and a named generic are the same desugaring — but only the named generic gives you a name to fill with turbofish.
fn count_args(x: impl Display) -> usize { x.to_string().len() } // no name to turbofish
fn default_string<T: Default + Display>() -> String { // named param `T`
T::default().to_string()
}
default_string takes no value argument, so there’s nothing to infer T from — the
only way to call it is default_string::<i32>(). An impl Trait argument literally
cannot express this case, because there is no type parameter in the <...> list to
fill. That is the one real cost of the argument-position sugar.
Note:
count_argsuses.to_string().len(), which counts bytes, not chars. It matches the ASCII test cases, butcount_args("héllo")would be 6, not 5. Use.chars().count()for characters.
4. The killer app: returning closures and chains
fn adder(n: i32) -> impl Fn(i32) -> i32 {
move |x| x + n // `move` captures n by value — without it the closure
} // would borrow n, which is gone when adder returns
fn pipeline<'a>(words: &'a [&'a str]) -> impl Iterator<Item = String> + 'a {
words.iter().filter(|w| w.len() > 3).map(|w| w.to_uppercase())
}
Both return values have types you could never spell by hand. Before impl Trait you
would have written Box<dyn Fn(i32) -> i32> and Box<dyn Iterator<Item = String>> —
heap + dynamic dispatch. RPIT returns them by value, monomorphized.
5. The defining footgun: one hidden type, all branches
RPIT promises exactly one concrete type. So this is rejected:
// WRONG — E0308: `if` and `else` have incompatible types
fn ranged(rev: bool, n: u32) -> impl Iterator<Item = u32> {
if rev { (0..n).rev() } else { 0..n } // Rev<Range<u32>> vs Range<u32>
}
Both arms implement Iterator<Item = u32>, but they are different concrete types, and
a single RPIT can only hide one. Three ways to collapse the branches into one type, each
with a different cost:
// (a) ERASE — both arms coerce to the same trait object. Cost: heap + vtable.
fn ranged_box(rev: bool, n: u32) -> Box<dyn Iterator<Item = u32>> {
if rev { Box::new((0..n).rev()) } else { Box::new(0..n) }
}
// (b) UNIFY — collect each arm into a Vec; both arms become vec::IntoIter<u32>.
// Cost: eager allocation, loses laziness.
fn ranged_vec(rev: bool, n: u32) -> impl Iterator<Item = u32> {
if rev { (0..n).rev().collect::<Vec<_>>().into_iter() }
else { (0..n).collect::<Vec<_>>().into_iter() }
}
// (c) BRANCH-AS-DATA — one enum that is itself an Iterator. No heap, stays lazy.
enum Either<L, R> { Left(L), Right(R) }
impl<L, R> Iterator for Either<L, R>
where L: Iterator, R: Iterator<Item = L::Item> {
type Item = L::Item;
fn next(&mut self) -> Option<Self::Item> {
match self { Either::Left(l) => l.next(), Either::Right(r) => r.next() }
}
}
Option (c) is exactly what itertools::Either is. The cost spectrum — Box (heap+vtable)
→ Vec (eager alloc) → Either (stack + lazy) — is the practical takeaway.
6. Lifetime capture (and edition 2024 changes the rules)
An RPIT’s hidden type may borrow from the function’s inputs, so the question is: which lifetimes/type-params does the hidden type “capture”?
- Edition 2021: RPIT captured nothing unless you spelled it. Borrowing an input
gave
E0700(“hidden type captures lifetime that does not appear in bounds”); you fixed it by adding+ '_/+ 'ato the return. - Edition 2024 (this crate): RPIT auto-captures all in-scope generic params and lifetimes. So a function that borrows its input “just works” with no annotation:
// On 2024 this needs NO `+ 'a`. On 2021 it is E0700 without `+ '_`.
fn lengths<'a>(words: &'a [&'a str]) -> impl Iterator<Item = usize> {
words.iter().map(|w| w.len()) // borrows words internally, yields owned usize
}
The new skill is the opposite problem — opting out of an over-broad capture with
precise-capturing + use<...>:
// WRONG on 2024: auto-captures 'a even though the result owns nothing, so the
// returned iterator is wrongly tied to the borrow — caller can't outlive it (E0597).
fn counter(_data: &[i32]) -> impl Iterator<Item = i32> { 0..3 }
// OK: `use<>` = capture NOTHING. The iterator owns everything and outlives the borrow.
fn counter(_data: &[i32]) -> impl Iterator<Item = i32> + use<> { 0..3 }
Model it as: 2024 captures everything in scope by default; use<...> narrows the
set. use<> captures nothing; use<'a, T> captures exactly those. The compiler even
suggests + use<> in the E0597 message.
7. async fn IS return-position impl Trait
The reveal that ties the ladder together. These two are the same thing:
async fn double_async(x: u32) -> u32 { x * 2 }
fn double_rpit(x: u32) -> impl Future<Output = u32> {
async move { x * 2 }
}
async fn is sugar: the compiler turns the body into an anonymous state-machine type
that implements Future, and hands it back via RPIT. The Output is whatever followed
the original ->. Every RPIT rule still applies:
- Capture: the future borrows whatever the async block borrows.
- The
Sendquestion: the state machine isSendonly if everything held across an.awaitisSend— the same auto-trait reasoning as theSend/Syncladder.double_rpit(5)isSendbecause only au32lives across awaits, whichassert_send(&fut)confirms.
8. RPITIT — impl Trait in trait returns
Since Rust 1.75 you can put impl Trait in a trait method’s return type (“RPITIT”),
and async fn in traits is just RPITIT under the hood:
trait Source {
fn values(&self) -> impl Iterator<Item = u32>; // RPITIT
}
trait Greeter {
async fn greet(&self) -> String; // ≡ fn greet(&self) -> impl Future<Output = String>
}
The catch: a trait with an RPITIT (or async fn) method is not dyn-compatible.
// E0038: `Source` cannot be made into an object.
let _boxed: Box<dyn Source> = Box::new(Squares);
Why? A vtable needs one fixed return type per method to store as a function pointer.
But each impl of values returns a different hidden type (Squares::values → some
Map<...>, another impl → some Filter<...>). There is no single signature to put in
the vtable. So you consume RPITIT traits through generics / static dispatch:
fn sum_source(s: impl Source) -> u32 { s.values().sum() } // OK: monomorphized
This is precisely why async fn in traits historically needed the async-trait crate —
it Boxes the future to erase it back into one nameable type — and why dyn async
traits still need help today.
Footguns
| Trap | Symptom | Fix |
|---|---|---|
Turbofish on an impl Trait arg | “cannot provide explicit generic arguments” | use a named generic param <T> instead |
if/else returns two iterator types | E0308 incompatible types | Box<dyn>, collect-to-Vec, or an Either enum |
| RPIT over-captures a lifetime (2024) | E0597 “does not live long enough” | add + use<> (or + use<'a, T>) to narrow the capture |
| Borrowing input on edition 2021 | E0700 captures lifetime | add + '_ / + 'a to the return type |
dyn Trait on an RPITIT/async-fn trait | E0038 not dyn-compatible | use generics; or Box the return manually / async-trait |
.len() for “char count” | wrong for non-ASCII | .chars().count() |
Real-world patterns
- Returning iterators from library functions without exposing the concrete adapter
type — the single most common RPIT use.
std’s ownVec::iter,HashMap::keys, etc. return named types, but most application code returnsimpl Iterator. itertools::Eitheris the rung-5 enum, productized: the lazy, no-heap way to return one of two iterator types from a branch.async fneverywhere is RPIT in disguise. When you needSendfutures (e.g. to spawn on a multithreaded runtime), you reason about what crosses each.await.async fnin traits (1.75+) for static-dispatch async APIs;#[trait_variant]/async-traitwhen you needdyn.- Precise capturing
use<>for returning owned iterators/futures that must outlive the borrowed data they were built from.
Capstone insight
The capstone builds a small lazy combinator toolkit where every builder hands back
impl Trait — compose (RPIT closure + APIT bounds), naturals() (an infinite
impl Iterator), keep (threads any generic iterator through a filter), a RPITIT
Stage trait — and assembles them into a pipeline that stays lazy until the final
collect():
naturals() -> keep(evens) -> MulStage(10).apply -> compose((x+1)*2) -> take(3)
1,2,3.. 2,4,6 20,40,60 42,82,122 [42,82,122]
The single exception is op_fn, where a runtime match selects one of three closures:
fn op_fn(op: Op) -> Box<dyn Fn(u64) -> u64> {
match op {
Op::Inc => Box::new(|x| x + 1),
Op::Double => Box::new(|x| x * 2),
Op::Square => Box::new(|x| x * x),
}
}
Three different closure types, one per arm — the one-hidden-type rule means RPIT cannot
express it, so you must erase to Box<dyn Fn>. The whole lesson of the ladder in one
function: impl Trait carries you all the way until runtime branching over distinct
types forces type erasure, and there — and only there — you reach for dyn.
Explain it back
- What’s the difference between
fn f(x: impl Trait)andfn f() -> impl Traitin terms of who chooses the type? - Why can’t you turbofish a function whose parameter is written
impl Trait? - Why does
if cond { a } else { b }fail whenaandbare different iterator types behind oneimpl Iteratorreturn — and what are three ways to fix it? - On edition 2024, what does
+ use<>mean, and what error does omitting it cause when an RPIT accidentally captures a lifetime it doesn’t need? async fn f() -> Tdesugars to what signature? When is the resulting futureSend?- Why is a trait with an
async fn/ RPITIT method notdyn-compatible, and how do you consume it instead?
See also
- Static vs dynamic dispatch — the
impl Traitvsdyn Traitvs enum trade-off, and object safety in depth. - Closures & Fn/FnMut/FnOnce — what
impl Fnis actually returning. - Iterators end-to-end — the adapter chains whose types RPIT hides.
Send&Syncdeeply — the auto-trait reasoning behindSendfutures.- HRTB — for<’a> — higher-ranked bounds, the other place lifetimes get subtle.
Marker & auto traits
Ladder:
src/bin/marker_auto_traits.rs· Run:cargo run --bin marker_auto_traits· Phase 2 · 9 rungs
TL;DR
A marker trait is a trait with no methods. It carries no behavior — it is a compile-time tag that means “this type has this property” or “this type is permitted here.” You use it as a bound (T: Marker), and the bound alone is the whole point.
An auto trait is a special marker the compiler implements for you, automatically and recursively, based on a type’s fields. Send and Sync are the famous ones. You reason about them negatively: a type is Send unless it contains something that isn’t. You opt out by adding a !Send field (a raw pointer via PhantomData), and opt back in by promising soundness yourself with unsafe impl.
PhantomData<T> is the glue: a zero-sized field that lets a type carry a type parameter it never stores, controlling auto traits, variance, and drop-checking at zero runtime cost.
Why this exists (from first principles)
Rust needs to express type-level facts that are checked by the compiler but require no methods to call:
- “This duplicates instead of moves” —
Copy. The fact changes assignment semantics, not behavior. - “This has a known size at compile time” —
Sized. Generic code needs it to put values on the stack. - “This is safe to move to another thread” —
Send. “Safe to share&Tacross threads” —Sync.
None of these are behaviors you invoke. They are properties the type system reasons about. A normal trait (with methods) is the wrong tool — there is nothing to implement. So Rust gives you the empty trait as a permission slip and the auto trait as an inferred property. Both are checked at compile time and erased entirely at runtime.
The payoff: you encode invariants — “only authorized types,” “thread-bound handles,” “legal protocol states” — directly into the type system, and the compiler enforces them with zero runtime cost.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | foundations | Marker trait as a permission tag | An empty trait used as a bound gates which types a function accepts. |
| 2 | foundations | Copy is a marker | Copy: Clone flips assignment/argument-passing from move to bitwise copy. |
| 3 | mechanics | Sized and ?Sized | Every generic has a silent T: Sized; relax it to accept DSTs. |
| 4 | mechanics | Auto traits compose structurally | A type is Send/Sync iff all its fields are. |
| 5 | footgun | Rc is !Send | A non-atomic refcount poisons the whole closure; Arc fixes it. |
| 6 | footgun | Negative reasoning & opt-out | PhantomData<*const ()> makes your own type !Send/!Sync. |
| 7 | real-world | PhantomData as a marker | Typed IDs: Id<User> != Id<Post>; the marker shape controls auto traits. |
| 8 | real-world | unsafe impl Send done right | Re-grant auto traits a raw pointer removed — under the right bound, with a SAFETY contract. |
| 9 | capstone | Typestate from markers | Sealed ZST states + PhantomData<S> make illegal operations not compile. |
The ideas, built up
1. A marker trait is a permission slip
An empty trait has no methods, so what could it possibly do? It tags types, and a generic bound consumes the tag.
trait Approved {} // no methods — pure tag
struct Admin;
struct Editor;
struct Guest; // deliberately NOT tagged
impl Approved for Admin {}
impl Approved for Editor {}
fn can_publish<T: Approved>(_user: &T) -> bool { true }
can_publish has no idea what Approved means — it never calls anything on T. The bound T: Approved is the entire mechanism. can_publish(&Admin) compiles; can_publish(&Guest) is a compile error (the trait bound Guest: Approved is not satisfied). You built a type-level access list, enforced at compile time, costing nothing at runtime.
2. Copy is a marker that changes language semantics
Copy is the most famous marker. It has no method of its own — the duplication logic lives on its supertrait Clone (Copy: Clone). What it does is tell the compiler: “duplicate this bit-for-bit on assignment instead of moving it.”
#[derive(Copy, Clone)] // Copy needs Clone — you can't have one without the other
struct Point { x: i32, y: i32 }
fn manhattan(p: Point) -> i32 { p.x.abs() + p.y.abs() }
fn sum_uses_original() -> i32 {
let p = Point { x: 3, y: -4 };
let d = manhattan(p); // p is COPIED in, not moved
d + p.x + p.y // ...so p is STILL valid here
}
Without Copy, manhattan(p) moves p, and the next line is error[E0382]: use of moved value. Adding the marker silently rewrites what manhattan(p) means. Note the structural rule already appearing: a String field blocks Copy entirely, because String isn’t Copy. The property composes from the fields up — exactly how auto traits will behave.
3. Sized and the invisible bound
Sized is auto-implemented for every type whose size is known at compile time. It is not implemented for dynamically sized types (DSTs): str, [T], dyn Trait. You can never hold a bare DST by value — only behind a pointer (&str, Box<str>, &[T]).
The twist: every generic <T> carries a silent T: Sized the compiler inserts for you. So fn f<T>(x: T) secretly means fn f<T: Sized>(x: T). To accept DSTs you opt out with ?Sized — the only place ? ever appears on a bound.
// WRONG: the implicit `T: Sized` rejects str and [u8]
// fn last_byte<T: Bytes>(value: &T) -> Option<u8> { ... }
// OK: relax the implicit bound; keep the value behind a reference
fn last_byte<T: Bytes + ?Sized>(value: &T) -> Option<u8> {
value.view().last().copied()
}
The compiler’s own diagnostic spells out the lesson: “the size for values of type str cannot be known at compilation time … required by an implicit Sized bound … consider relaxing the implicit Sized restriction: + ?Sized.” Once T: ?Sized, you must keep T behind a pointer (&T), because a bare T would have unknown size on the stack. Calling last_byte(s) where s: &str binds T = str (the unsized part), and the reference is yours.
4. Auto traits compose structurally
Now the real auto traits. Send = safe to move to another thread. Sync = safe to share &T across threads. You almost never impl these: the compiler grants them to a type if and only if every field already has them.
The classic way to prove a type carries an auto trait is a zero-cost witness function — all the work is in the bound, the body is empty:
fn assert_send<T: Send>() {} // compiles to nothing; pure type-level check
fn assert_sync<T: Sync>() {}
struct Wrapper { id: u64, name: String, tags: Vec<u8> }
assert_send::<Wrapper>(); // OK — u64, String, Vec<u8> are all Send
assert_sync::<Wrapper>(); // OK
Nowhere did you write impl Send for Wrapper. The compiler walked the fields, found them all Send, and granted it. That is what “auto” means: opt-out, not opt-in.
5. The defining footgun: Rc is !Send
Rc<T> uses a plain, non-atomic reference count. If two threads cloned or dropped the same Rc at once, the count would race and you would get a use-after-free. So the standard library marks Rc as !Send and !Sync. Because Send is an auto trait, that one negative poisons anything containing an Rc.
// WRONG: thread::spawn requires its closure to be Send;
// capturing an Rc makes the closure !Send.
// let data = Rc::new(41);
// thread::spawn(move || *data + 1); // error: `Rc<i32>` cannot be sent between threads safely
// OK: Arc has an ATOMIC refcount, so it is Send + Sync
fn parallel_sum(value: i32, n_threads: usize) -> i32 {
let data = Arc::new(value);
let handles: Vec<_> = (0..n_threads)
.map(|_| {
let data = Arc::clone(&data); // each thread gets its own handle
thread::spawn(move || *data)
})
.collect();
handles.into_iter().map(|h| h.join().unwrap()).sum()
}
Arc and Rc have identical APIs. The only difference is the atomic refcount — and that single invariant is what earns the auto traits back. Collecting handles into a Vec before joining keeps all threads running concurrently.
6. Negative reasoning: opting out on purpose
Auto traits are reasoned about negatively, so to make your own type !Send when all its real fields are Send, you add a zero-sized field whose type is !Send. The canonical “thread-bound” token is PhantomData<*const ()> — a raw pointer is !Send and !Sync, and PhantomData carries that property at zero size.
struct ThreadBound {
id: u32, // perfectly Send on its own
phantom: PhantomData<*const ()>, // ...but this poisons Send + Sync
}
ThreadBound holds nothing but a u32, yet the compiler now refuses to move it across a thread boundary. Why does a raw pointer poison the auto traits? The compiler can’t verify what it points to or who else touches it, so it conservatively refuses. This is exactly how MutexGuard, Rc, and thread-local handles keep themselves on one thread. size_of::<ThreadBound>() is still 4 — pure type-level enforcement.
The ladder verifies
!Send-ness at runtime with an autoref-specialization probe exposed as a macro (is_send!(T)). It must resolve at a concrete type — a genericfn is_send<T>()wrapper erases theSendinfo and always reportsfalse.
7. PhantomData as a marker, and choosing its shape
PhantomData<T> lets a type carry a parameter T it never stores. The classic use is a typed ID: a u64 tagged with which entity it belongs to, so Id<User> and Id<Post> are different types and mixing them is a compile error — at zero runtime cost (it is still just a u64).
The deep part is that the marker shape inside PhantomData controls auto-trait and variance behavior:
PhantomData<…> | Meaning | Auto-trait effect |
|---|---|---|
PhantomData<T> | “I own a T” | inherits T’s Send/Sync; participates in drop check |
PhantomData<fn() -> T> | “I produce T” (pure tag) | always Send + Sync + Copy, covariant, regardless of T |
PhantomData<*const T> | thread-bound token | !Send + !Sync |
A typed ID is a pure tag — holding a user’s id doesn’t mean you own a User. So it should stay Send/Sync/Copy even if User is !Send. The right marker is fn() -> T:
struct Id<T> {
raw: u64,
_tag: PhantomData<fn() -> T>, // pure tag: stays Send + Copy even if T is !Send
}
And the trait impls are hand-written, not derived, so the bound lands where it belongs:
// WRONG: #[derive(Clone)] emits `impl<T: Clone> Clone for Id<T>`
// — needlessly requires the TAG to be Clone.
// OK: hand-write it with no bound on T; the requirement is on the u64.
impl<T> Clone for Id<T> {
fn clone(&self) -> Self { Id::new(self.raw) }
}
impl<T> Copy for Id<T> {} // valid because raw: u64 is Copy
impl<T> PartialEq for Id<T> {
fn eq(&self, other: &Self) -> bool { self.raw == other.raw }
}
Now fetch_user(some_post_id) is a compile error — a whole class of “wrong ID” bugs deleted. This is how sqlx, ECS libraries, and unit-of-measure crates work.
8. unsafe impl Send done right
Sometimes auto-derivation is too conservative. A type built on a raw pointer is automatically !Send/!Sync, but you, the author, may know it is safe — and you take responsibility with unsafe impl. This is the manual opt-IN, the mirror of rung 6’s opt-OUT.
struct MyBox<T> { ptr: *mut T } // *mut T → compiler refuses Send/Sync
impl<T> MyBox<T> {
fn new(value: T) -> Self { Self { ptr: Box::into_raw(Box::new(value)) } }
fn get(&self) -> &T {
// SAFETY: ptr came from Box::into_raw (non-null, aligned, initialized);
// MyBox uniquely owns it until Drop; &self yields only a shared &T.
unsafe { &*self.ptr }
}
}
impl<T> Drop for MyBox<T> {
fn drop(&mut self) {
// SAFETY: unique owner; drop runs at most once, so from_raw reclaims exactly one Box.
unsafe { drop(Box::from_raw(self.ptr)); }
}
}
// Re-grant the auto traits — but only under the SAME bound the safe type needs.
unsafe impl<T: Send> Send for MyBox<T> {} // moving the box moves the T → needs T: Send
unsafe impl<T: Sync> Sync for MyBox<T> {} // get() shares &T → needs T: Sync
The crux is the bound. MyBox owns its T; moving the box to another thread moves the T there, which is sound only if T: Send. Write unsafe impl<T> Send with no bound and you could smuggle a MyBox<Rc<_>> across threads — the exact UB rung 5 prevents. This is character-for-character how std::boxed::Box grants its auto traits. The bound is the safety contract; the // SAFETY: comment is where you write down why it holds.
Footguns
- The invisible
Sizedbound.fn f<T>(x: T)silently meansT: Sizedand rejectsstr/[T]/dyn Trait. Relax with?Sizedand keep the value behind a pointer. - One bad field poisons an auto trait. A single
Rc,Cell, or raw-pointer field makes the whole struct loseSend/Sync. The error often points atthread::spawn’sSendbound, far from the offending field. - Wrong
PhantomDatashape.PhantomData<T>drags T’s thread-safety in;PhantomData<fn() -> T>keeps a pure tagSend/Copy;PhantomData<*const T>opts out. Picking the wrong one silently changes whether your wrapper crosses threads. - Deriving bounds onto phantom tags.
#[derive(Clone)]onId<T>emitsimpl<T: Clone>— a needless bound on a tag you never store. Hand-write the impl so the requirement lands on the real fields. - Forgetting the state-marker field. A generic
Conn<S>that never usesSiserror[E0392]: parameter S is never used. The fix is aPhantomData<S>field. - Over-promising in
unsafe impl.unsafe impl<T> Send(no bound) on an owning wrapper is unsound. Match the bound the safe abstraction would need (T: Send).
Real-world patterns
Box,Vec,Arcall use boundedunsafe impl Send/Syncover their internal raw pointers — exactly the rung-8 pattern.- Thread-bound handles (
MutexGuard,Rc, FFI/GUI context handles) usePhantomData<*const ()>(or*mut) to stay!Send, so the compiler enforces single-thread use. - Typed IDs / units of measure (
sqlx, ECS frameworks,uom) usePhantomData<fn() -> T>to get distinct types with zero runtime cost. - Sealed traits (private supertrait) appear throughout std and crates like
serdeto mark a closed set of types that downstream code cannot extend.
Capstone insight
The capstone builds a Conn<S> whose state is a type parameter, combining every thread of the ladder:
mod sealed { pub trait Sealed {} } // private — only this crate can implement
trait State: sealed::Sealed { const NAME: &'static str; }
struct Disconnected; struct Connected; struct Authenticated; // ZST markers
// impl Sealed + State for each...
struct Conn<S: State> {
peer: String,
log: Vec<String>,
_state: PhantomData<S>, // avoids E0392; zero size
}
impl Conn<Disconnected> { fn connect(self) -> Conn<Connected> { /* ... */ } }
impl Conn<Connected> { fn authenticate(self, t: &str) -> Conn<Authenticated> { /* ... */ } }
impl Conn<Authenticated>{ fn send(&mut self, msg: &str) -> usize { /* ... */ } } // ONLY here
impl<S: State> Conn<S> { fn status(&self) -> &'static str { S::NAME } } // every state
The “aha”: four small features combine into a state machine the compiler checks for free.
- ZST marker structs are the states —
Send, zero-size, behavior-free. - A sealed
Statetrait (privateSealedsupertrait) is the marker that says “this is a legal state,” and being sealed, no downstream crate can invent a new state.impl State for Roguefails with “the traitSealedis not satisfied.” PhantomData<S>tags the state ontoConnat zero cost (and satisfies E0392).- Consuming transitions (
self, not&self) move the old handle away, so a staleConn<Connected>cannot be reused afterauthenticate.
send exists only in impl Conn<Authenticated>, so disconnected_conn.send(..) is no method named send — a compile error, not a runtime check. Your protocol’s rules became type errors. This is the engine behind typed-builder, embedded-HAL peripheral states, and session-typed protocols.
Explain it back
- Why is
Copya marker trait even thoughClonehas theclonemethod? - Why does
T: ?Sizedonly make sense when the value is behind a pointer like&T? - What exactly makes
Rc<T>!Send, and which single property doesArc<T>change to fix it? - Why does a
PhantomData<*const ()>field make a struct!Send, whilePhantomData<()>does not? - When should the marker be
PhantomData<T>vsPhantomData<fn() -> T>? - Why is
unsafe impl<T: Send> Send for MyBox<T>sound, butunsafe impl<T> Sendtoo strong? - How does a sealed supertrait make it impossible for downstream code to add a new typestate?
See also
- Static vs dynamic dispatch —
Sizedand object safety - Blanket impls & coherence — bounds and the orphan rule
- The typestate pattern — the capstone, in depth
Send&Syncdeeply — Phase 4 follow-up on thread safety- Newtype & zero-cost wrappers —
PhantomDatatyped IDs
Error handling architecture
Ladder:
src/bin/error_arch.rs· Run:cargo run --bin error_arch· Phase 3 · 9 rungs
TL;DR
Rust has no exceptions. An error is just a value of type Result<T, E>, and
? is sugar for “if Err, convert via From::from and return early.” So the
entire architecture question collapses to one decision: what is E, and who
chooses its shape?
Two answers, used at different layers:
- Libraries →
thiserror: a typed, exhaustiveenumthe caller canmatchon and recover from. You hand out structure. - Applications →
anyhow: one opaqueanyhow::Errorthat swallows any error, carries.context()breadcrumbs, and bubbles tomain. The caller wants a report, not a branch.
? + From is the weld between the two. And anyhow is not magic — its core is a
blanket From impl plus a .context() that chains the original error as a
source(). The capstone rebuilds it in ~30 lines.
Why this exists (from first principles)
In a language with exceptions, the error path is invisible: any call might throw,
and the type signature doesn’t say so. Rust makes the error path part of the type:
a fallible function returns Result<T, E>, and you cannot get the T out without
acknowledging the E. That’s the whole safety story — no surprise unwinding, no
forgotten failure mode.
But that honesty has an ergonomic cost: every fallible call would need an explicit
match to propagate. ? buys the ergonomics back:
let n = s.parse::<i32>()?; // desugars roughly to:
// let n = match s.parse::<i32>() {
// Ok(v) => v,
// Err(e) => return Err(From::from(e)),
// };
The critical word is From::from. ? will only compile if the function’s error
type implements From<the error at the call site>. Every design choice in this
topic is downstream of that one fact. Pick E = Box<dyn Error> and the blanket
From<E: Error> impl makes everything work. Pick a custom enum and you owe a
From impl per source error (or a #[from] to generate it). Pick anyhow::Error
and its blanket From covers everything.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | foundations | ? + Box<dyn Error> | heterogeneous errors collapse to one trait object |
| 2 | foundations | hand-rolled enum | Display + Error + From is the contract |
| 3 | mechanics | thiserror derive | #[error]/#[from] generate rung 2 verbatim |
| 4 | mechanics | anyhow | opaque error + .context() / bail! / anyhow! |
| 5 | footgun | source chains & downcasting | errors are a linked list; recover the type back |
| 6 | footgun | E0277 + String errors | ? needs From; String: !Error is a trap |
| 7 | real-world | lib/app boundary | typed error survives under the anyhow context |
| 8 | real-world | classification | is_retryable() + #[non_exhaustive] drive a retry loop |
| 9 | capstone | mini-anyhow | blanket From + Context trait + source() chain |
The ideas, built up
1. The quick app error: Box<dyn Error>
When you don’t care about the type of the failure — you just want it to bubble —
return Box<dyn Error>. Different concrete errors unify into one trait object:
fn parse_and_double(s: &str) -> Result<i32, Box<dyn Error>> {
let n = s.parse::<i32>()?; // ParseIntError -> Box<dyn Error>
if n == 13 {
return Err("13 is unlucky".into()); // &str -> Box<dyn Error>
}
Ok(n * 2)
}
Two different errors (ParseIntError and a string) leave through the same return
type with zero ceremony. This works because of two From impls in std:
impl<E: Error + ...> From<E> for Box<dyn Error> (coerces the parse error) and
impl From<&str> for Box<dyn Error> (builds an error from a message). That’s the
seed of the entire “erased error” idea that anyhow industrializes later.
2. The contract for being an error
A library should not force Box<dyn Error> on callers — it should hand out a
type they can match. To be a “real” error in Rust you implement two traits:
Display— the human-readable message.std::error::Error— the marker trait (withDebug + Displayas supertraits) that unlocks?-into-Box<dyn Error>, source chains, and downcasting.
#[derive(Debug)]
enum ConfigError { Missing(String), Parse(ParseIntError) }
impl fmt::Display for ConfigError {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
ConfigError::Missing(k) => write!(f, "missing key: {k}"),
ConfigError::Parse(e) => write!(f, "invalid number: {e}"),
}
}
}
impl Error for ConfigError {
fn source(&self) -> Option<&(dyn Error + 'static)> {
match self {
ConfigError::Parse(e) => Some(e), // expose the underlying cause
_ => None,
}
}
}
impl From<ParseIntError> for ConfigError { // <- this is what makes `?` work
fn from(e: ParseIntError) -> Self { ConfigError::Parse(e) }
}
That From<ParseIntError> impl is the only reason ? can turn a parse failure
into a ConfigError. The source() override is optional now but pays off in rung
5 — it’s the link that lets a caller walk from ConfigError down to the
ParseIntError that caused it.
3. thiserror: the boilerplate, derived
Everything in rung 2 — Display, Error, From, source — is mechanical. The
thiserror derive generates byte-for-byte the same code from attributes:
#[derive(Debug, thiserror::Error)]
enum LoadError {
#[error("io error: {0}")]
Io(#[from] std::io::Error),
#[error("bad number: {0}")]
BadNumber(#[from] ParseIntError),
#[error("input was empty")]
Empty,
}
#[error("...")]generates theDisplayimpl.{0}interpolates the tuple field.#[from]on a field generatesFrom<that type>and wires that field up as thesource(). One attribute, both jobs.
It is a zero-runtime-cost macro — no boxing, no dynamic dispatch. That’s why it’s
the library-layer choice: the caller still gets a fully typed enum to match.
4. anyhow: the application’s opaque error
An application’s top layer rarely wants to match on variants. It wants: “did it
work? if not, give me a good report and bubble to main.” anyhow::Error is one
opaque type that any E: Error + Send + Sync + 'static converts into via ?,
and its superpower is context:
use anyhow::{Context, anyhow};
fn load_user(dir: &str, id: &str) -> anyhow::Result<u64> {
if dir == "missing" {
return Err(anyhow!("no such dir: {dir}")); // ad-hoc error
}
let id = id.parse::<u32>()
.with_context(|| format!("parsing user id {id:?}"))?; // add a breadcrumb
Ok(id as u64 * 2)
}
The key behavior: .with_context(...) makes the context message the outer
Display, while the original ParseIntError is preserved underneath as the
source(). anyhow never throws the real error away — it stacks a readable layer on
top. So e.to_string() is parsing user id "xyz" but e.source() is still
Some(ParseIntError).
.context("literal")— eager..with_context(|| ...)— lazy; the closure only runs on the error path. Use it when building the message costs something.anyhow!(...)builds an ad-hoc error;bail!(...)isreturn Err(anyhow!(...)).
5. Errors are a linked list: walk it, then downcast back
Every error optionally points at the cause it wrapped via .source(). That makes
an error a singly-linked list, and .context() grows it. Walking it gives a full
report:
fn error_chain(err: &dyn Error) -> Vec<String> {
let mut chain = Vec::new();
let mut current = err;
chain.push(current.to_string());
while let Some(source) = current.source() {
chain.push(source.to_string());
current = source;
}
chain
}
// load_user("data","xyz") -> ["parsing user id \"xyz\"", "invalid digit found in string"]
The reverse trick is downcasting: recover a concrete type after it has been
erased into anyhow::Error. This is the escape hatch for “opaque by default, but
typed when the app does need to branch”:
fn classify(err: &anyhow::Error) -> &'static str {
if let Some(load_error) = err.downcast_ref::<LoadError>() {
match load_error {
LoadError::Empty => "empty",
_ => "load",
}
} else {
"other"
}
}
downcast_ref::<T>() walks the chain for you — even when the LoadError was
wrapped in a .context(...), anyhow can still reach down and hand back the concrete
&LoadError.
6. The two footguns ? sets
E0277 — “the trait bound MyError: From<X> is not satisfied”. This is the most
common real-world error message in the whole topic, and it is not mysterious: it
is ? telling you the From impl it needs doesn’t exist. The fix is to add it (or
#[from]):
#[derive(Debug, thiserror::Error)]
enum PipelineError {
#[error("stage a failed: {0}")] StageA(#[from] ParseIntError),
#[error("stage b failed: {0}")] StageB(#[from] TryFromIntError),
#[error("legacy: {0}")] Legacy(String),
}
Result<T, String> is an anti-pattern. String does not implement
std::error::Error, so a stringly-typed error has no source() chain, can’t be
downcast, and can’t be matched — you discarded all structure and kept a sentence.
Wrap it back into a real type at the boundary with .map_err:
fn adapt_legacy(ok: bool) -> Result<i32, PipelineError> {
legacy_op(ok).map_err(PipelineError::Legacy) // tuple variant as a fn value
}
Note the deliberate choice not to put #[from] on Legacy(String): a blanket
From<String> would let ? silently coerce any stray String into your error
type. Forcing an explicit .map_err keeps the wrapping intentional.
7. The boundary: thiserror library, anyhow application
This is the whole architecture in one screen. The library exposes a typed error;
the app wraps it in context and returns opaque anyhow::Error — but the typed error
survives underneath and is recoverable:
mod store { // LIBRARY
#[derive(Debug, thiserror::Error)]
pub enum StoreError {
#[error("key not found: {key}")] NotFound { key: String },
#[error("not a number: {0}")] Parse(#[from] ParseIntError),
}
pub fn get_number(key: &str) -> Result<i64, StoreError> { /* typed */ }
}
fn load_setting(key: &str) -> anyhow::Result<i64> { // APPLICATION
store::get_number(key).with_context(|| format!("loading setting {key:?}"))
}
The payoff, proven by the test:
let e = load_setting("missing").unwrap_err();
assert_eq!(e.to_string(), r#"loading setting "missing""#); // anyhow context outside
let typed = e.downcast_ref::<store::StoreError>(); // ...typed error still inside
assert!(matches!(typed, Some(store::StoreError::NotFound { .. })));
“thiserror for libs, anyhow for apps” isn’t a compromise — downcast_ref means you
get both: opaque convenience and typed recovery.
8. Classify, don’t just propagate: is_retryable + #[non_exhaustive]
A mature error type lets callers decide how to react, not just what failed. Put the policy on the error type as a method, and a fully generic consumer can branch without knowing any variant:
impl ApiError {
fn is_retryable(&self) -> bool {
match self { // exhaustive: a NEW variant forces a compile error here
ApiError::RateLimited { .. } | ApiError::Timeout
| ApiError::ServiceUnavailable => true,
ApiError::NotFound { .. } | ApiError::Unauthorized => false,
}
}
}
fn run_with_retry<T, F>(max_attempts: u32, mut op: F) -> Result<T, ApiError>
where F: FnMut() -> Result<T, ApiError> {
for attempt in 0..max_attempts {
match op() {
Ok(v) => return Ok(v),
Err(e) => {
if !e.is_retryable() || attempt + 1 == max_attempts {
return Err(e); // fatal, or out of attempts
}
// retryable: loop again (real code would back off here)
}
}
}
unreachable!()
}
run_with_retry knows nothing about specific variants — only is_retryable().
Add a variant later and every retry site behaves correctly for free.
#[non_exhaustive] on the enum is the companion: it forces downstream crates to
include a _ arm in their match, so you can add variants later without a breaking
change. Note the split — inside the defining crate the match stays exhaustive
(a forgotten new variant won’t compile, which is a safety net); only foreign crates
are forced to the wildcard.
Footguns
| Trap | What bites | Fix |
|---|---|---|
? won’t compile (E0277) | no From<source> for your error type | add From / #[from], or return Box<dyn Error> / anyhow::Error |
Result<T, String> | String: !Error — no source, no downcast, no match | wrap at the boundary with .map_err(MyError::Variant) |
source() not overridden | the cause chain stops short; reports lose the root | override source() (or use #[from]/#[source]) |
#[from] on an ad-hoc String variant | ? silently coerces any String into your error | drop #[from], force explicit .map_err |
implementing Error for an anyhow-like wrapper | collides with std’s reflexive From<T> for T | don’t impl Error on the wrapper (see capstone) |
matching a #[non_exhaustive] foreign enum without _ | won’t compile downstream | always add a _ => arm for others’ error enums |
Real-world patterns
- Library crates define one
#[derive(thiserror::Error)] #[non_exhaustive]enum per module/crate;#[from]for wrapped sources; classification methods likeis_retryable()/kind()for callers. - Binaries use
fn main() -> anyhow::Result<()>, sprinkle.context(...)at each layer, and let the error bubble; anyhow prints the wholesource()chain. Box<dyn Error>is the std-only middle ground (no dependency) when you want erasure without anyhow’s context/backtrace features.- The std
?+Frommechanism is what makes all three interoperate: a library’s typedStoreErrorflows into an app’sanyhow::Errorwith no glue code.
Capstone insight: anyhow is ~30 lines
The build-it rung strips the magic. anyhow::Error is essentially:
pub struct MyError(Box<dyn Error + Send + Sync + 'static>);
// (1) the single most important impl: this is what makes `?` erase any error.
impl<E: Error + Send + Sync + 'static> From<E> for MyError {
fn from(e: E) -> Self { MyError(Box::new(e)) }
}
Two non-obvious truths fall out of this:
-
MyErrormust NOT implementError. If it did, the blanketFrom<E: Error>above would overlap with std’s reflexiveFrom<MyError> for MyError— a coherence conflict. The realanyhow::Errormakes the exact same choice (it implementsDisplay + Debugbut notError). The thing you reach for to erase errors deliberately isn’t one itself. -
.context()is just another error whosesource()is the old one. Stacking context is growing the linked list by one node:
struct ContextError { msg: String, source: Box<dyn Error + Send + Sync + 'static> }
impl Error for ContextError {
fn source(&self) -> Option<&(dyn Error + 'static)> { Some(&*self.source) }
}
trait WrapErr<T> { fn context<C: Display>(self, ctx: C) -> Result<T, MyError>; }
impl<T, E: Error + Send + Sync + 'static> WrapErr<T> for Result<T, E> {
fn context<C: Display>(self, ctx: C) -> Result<T, MyError> {
self.map_err(|e| MyError(Box::new(ContextError {
msg: ctx.to_string(),
source: Box::new(e),
})))
}
}
Walk .source() on the result and you see the context message on top of the
original error — exactly anyhow’s {:#} output. (Some(&*self.source) is the
deref-then-reborrow that turns the Box back into a &dyn Error.) Once you’ve
written this, anyhow stops being a black box: it’s a blanket From, a boxed trait
object, and a context node that chains via source().
Explain it back
- Why does
?require aFromimpl, and what exactly does it call? - When do you reach for
thiserrorvsanyhowvsBox<dyn Error>? Why not anyhow in a library? - What does
#[from]generate, and why does it also setsource()? - An error has been erased into
anyhow::Error. How do you (a) print the full cause chain and (b) recover a specific typed variant to branch on? - Why is
Result<T, String>an anti-pattern? What capability do you lose? - Why can’t an anyhow-style erased error type implement
std::error::Erroritself? - How does
.context()preserve the original error? What doessource()return for a context node? - What’s the difference between an exhaustive
matchinside the defining crate and the_arm#[non_exhaustive]forces on downstream crates?
See also
- Conversion traits —
From/Intoand how?rides on them. - Box & the Heap —
Box<dyn Trait>erasure, the basis ofBox<dyn Error>. - Associated types vs generic params — trait design choices that show up in error-type APIs.
Custom error types
Ladder:
src/bin/custom_errors.rs· Run:cargo run --bin custom_errors· Phase 3 · 9 rungs
TL;DR
A “custom error type” is just a normal type that satisfies a two-method contract:
impl Display gives it a human message, and impl std::error::Error marks it as
an error and optionally points at the lower-level error underneath it via
source(). Everything else in the error ecosystem — ?, Box<dyn Error>,
downcasting, multi-line Caused by: reports, anyhow, thiserror — is built on
top of those two impls plus From. The one idea that unlocks the whole topic is
the source chain: a linked list of errors you walk from “what failed” down to
“why”, with each link reachable through source().
This ladder builds all of it by hand, no derive macros, so you can see exactly
what thiserror generates and what anyhow does at runtime.
Sibling page: Error handling architecture covers the architecture choice (
thiserrorfor libs vsanyhowfor apps). This page is the machinery underneath that choice.
Why it exists (from first principles)
In Rust, errors are values: a function that can fail returns Result<T, E>
and you choose E. The cheapest E is a String — but a string is a dead end.
The caller can println! it and nothing else: they can’t match on which failure
happened, can’t programmatically recover from one case but not another, and can’t
inspect what caused it. A string has thrown away all the structure.
So the standard library defines a contract for “a real error”:
pub trait Error: Debug + Display {
fn source(&self) -> Option<&(dyn Error + 'static)> { None }
// ... a few unstable methods (backtrace, provide)
}
Two things to notice immediately:
Debug + Displayare supertraits. You literally cannotimpl Errorfor a type that doesn’t already implement both.Displayis the human message;Debugis the developer/{:?}view. This is why every error in this file starts with#[derive(Debug)]and a hand-writtenDisplay.source()has a default ofNone. A “leaf” error (one that originates a failure) inherits that default. An error that wraps another overrides it to hand back the cause. That single optional method is the entire source-chain mechanism.
Once a type implements Error, it gains superpowers it can’t have otherwise: it
coerces into the universal Box<dyn Error>, it slots into ?, and the dyn Error
trait object gives you is::<T>() / downcast_ref::<T>() to recover the concrete
type later. The trait is the membership card.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | foundations | TooLong struct | Display + empty impl Error = a real error; unlocks Box<dyn Error> |
| 2 | foundations | ValidationError enum | one enum, many failure modes the caller can match on |
| 3 | mechanics | ConfigError + source() | wrap a cause; keep the cause OUT of Display |
| 4 | mechanics | LoadError + From | ? calls From::from — this is what #[from] generates |
| 5 | footgun | Box<dyn Error + Send + Sync> | the bounds propagate into fields; Rc → Arc to cross threads |
| 6 | footgun | describe_root / downcast | walk source() to the root, is::<T>() to decide |
| 7 | real-world | TracedError + Backtrace | capture where it failed; capture() vs force_capture() |
| 8 | real-world | AppError 3-level chain | layered library error + anyhow {:#} one-line printer |
| 9 | capstone | Chain + Report | rebuild anyhow’s iterator + Caused by: reporter from scratch |
The ideas, built up
1. The contract: Display + Error
The minimum viable error is a Debug struct, a Display impl, and an empty
Error impl:
#[derive(Debug)]
struct TooLong { len: usize, max: usize }
impl fmt::Display for TooLong {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "username too long: {} chars (max {})", self.len, self.max)
}
}
impl std::error::Error for TooLong {} // empty body — but it does real work
The empty impl Error for TooLong {} looks like it does nothing. It’s the whole
point. Without it, this line fails to compile:
let boxed: Box<dyn Error> = Box::new(err); // needs TooLong: Error
The coercion from Box<TooLong> to Box<dyn Error> is only allowed once the
compiler can prove TooLong: Error. The marker impl is what makes the type a
member of dyn Error. (If you forget it, the error reads:
the trait bound TooLong: std::error::Error is not satisfied ... required for the cast from Box<TooLong> to Box<dyn Error>.)
2. One enum, many failure modes
A struct models one failure. Real code fails several ways, and the idiomatic shape is a single enum with a variant per mode, each carrying exactly the data it needs:
#[derive(Debug)]
enum ValidationError {
TooShort { len: usize, min: usize },
TooLong { len: usize, max: usize },
BadChar { ch: char },
}
Display becomes a match self with one arm per variant. The payoff is on the
caller’s side — they get one type they can match exhaustively:
match validate("ab", 3, 16) {
Err(ValidationError::TooShort { len, min }) => /* tell the user the minimum */,
Err(ValidationError::BadChar { ch }) => /* highlight the bad char */,
// ... the compiler forces you to handle every case
}
That exhaustiveness is precisely what Box<dyn Error> (or a String) throws
away. Typed enum = the caller can branch; erased error = the caller can only print.
3. source(): the cause underneath
Most errors don’t originate a failure — they wrap a lower-level one. “Failed to
load config” because “failed to parse integer”. The Error trait models this with
the one optional method:
#[derive(Debug)]
enum ConfigError {
Malformed { line: String }, // leaf: no underlying cause
BadPort { source: std::num::ParseIntError }, // wraps the real cause
}
impl std::error::Error for ConfigError {
fn source(&self) -> Option<&(dyn Error + 'static)> {
match self {
Self::BadPort { source } => Some(source), // &ParseIntError -> &dyn Error
Self::Malformed { .. } => None, // == the default
}
}
}
Two things make this click:
Some(source)works because&ParseIntErrorcoerces to&dyn Error. Same unsizing coercion as rung 1, just behind a reference.- Display must NOT restate the source.
BadPort’sDisplaysays only"invalid port number"— it does not paste in theParseIntError’s text.
The separation rule.
Displayanswers what failed at this layer.source()answers why. Keep them disjoint. If you bake the cause’s message intoDisplay, every chain printer (rung 8, rung 9,anyhow) prints it twice. This is the single most important habit on this page.
4. From + ?: how #[from] actually works
In rung 3 you wrapped the cause manually with .map_err(|source| BadPort { source }).
The ? operator can do that conversion for you — but only if you teach it how.
The desugaring of expr? is roughly:
match expr {
Ok(v) => v,
Err(e) => return Err(From::from(e)), // <- the magic line
}
? calls From::from on the error before returning it. So if you implement
From<TheLowLevelError> for YourError, ? will silently convert and propagate:
impl From<std::io::Error> for LoadError { fn from(e: std::io::Error) -> Self { Self::Io(e) } }
impl From<std::num::ParseIntError> for LoadError { fn from(e: std::num::ParseIntError) -> Self { Self::Parse(e) } }
fn load_count(raw: &str) -> Result<u64, LoadError> {
if raw.is_empty() {
return Err(std::io::Error::new(std::io::ErrorKind::Other, "empty input"))?;
}
let n = raw.parse::<u64>()?; // ParseIntError -> LoadError, no .map_err
Ok(n)
}
load_count returns Result<_, LoadError> but ?-es values whose error types are
io::Error and ParseIntError. It compiles only because the two From impls
exist. Delete one and that ? stops compiling — that exact coupling is what
thiserror’s #[from] attribute generates for you.
5. The bounds hiding inside Box<dyn Error>
Box<dyn Error> is the lazy form. The type the wider ecosystem actually wants —
and what fn main() -> Result<(), Box<dyn Error>> and anyhow::Error use — is:
type BoxedSendSync = Box<dyn Error + Send + Sync + 'static>;
Send means the error can move to another thread; Sync means &error can be
shared across threads. An error that can’t cross threads is useless to a threaded
server or an async runtime. The footgun: those bounds propagate into every
field. This struct can’t become a BoxedSendSync:
#[derive(Debug)]
struct NotThreadSafe { detail: Rc<str> } // Rc is !Send + !Sync
error[E0277]: `Rc<str>` cannot be sent between threads safely
= note: required for the cast from `Box<NotThreadSafe>`
to `Box<dyn Error + Send + Sync + 'static>`
The fix isn’t to the signature — it’s to the payload. Swap Rc<str> for
Arc<str> (atomically reference-counted, and Send + Sync) and both the plain
and the send-sync boxing compile, and the boxed error survives thread::spawn.
+ Send + Syncis not ceremony. It’s a thread-mobility promise that the auto-traits force every field of your error to keep.
6. Downcasting: get the concrete type back
Box<dyn Error> erases the type. Sometimes you need it back — “if the root cause
was specifically a ParseIntError, retry; otherwise give up.” dyn Error has two
inherent methods (built on Any) for this:
err.is::<T>() -> bool // is the concrete type T?
err.downcast_ref::<T>() -> Option<&T> // borrow it as T if so
These work because Error: 'static, so every error carries a TypeId. Combine
downcasting with the source-chain walk to find and classify the root cause:
fn describe_root(top: &(dyn Error + 'static)) -> String {
let mut cur = top;
while let Some(next) = cur.source() { cur = next; } // walk to the bottom
cur.to_string()
}
fn root_is_parse_error(top: &(dyn Error + 'static)) -> bool {
let mut cur = top;
while let Some(next) = cur.source() { cur = next; }
cur.is::<std::num::ParseIntError>() // decide on the concrete type
}
source() gives you the next link; loop it to reach the root; is/downcast_ref
recover the concrete type so you can branch. This is exactly how
anyhow::Error::downcast_ref and retry-on-specific-error logic work.
7. Backtraces: capture where it failed
A source chain is the logical why (X because Y because Z). A backtrace is the
physical where — the call stack at the instant the error was created. You attach
one with std::backtrace::Backtrace:
#[derive(Debug)]
struct TracedError { msg: String, backtrace: Backtrace }
impl TracedError {
fn new(msg: impl Into<String>) -> Self {
Self { msg: msg.into(), backtrace: Backtrace::force_capture() }
}
fn backtrace(&self) -> &Backtrace { &self.backtrace } // inherent getter
}
Two APIs, and the difference matters:
| API | Behavior | When |
|---|---|---|
Backtrace::capture() | Respects RUST_BACKTRACE / RUST_LIB_BACKTRACE; if unset, returns a cheap disabled backtrace (status() == Disabled) | Real libraries — zero cost unless the user opts in |
Backtrace::force_capture() | Always walks the stack, ignoring env vars (expensive) | When you truly always want it (and for deterministic tests) |
Note the getter is an inherent method, not a trait override. Error::backtrace
exists but is still unstable on stable Rust, so real crates (and this rung)
expose their own fn backtrace(&self) -> &Backtrace instead. And Display writes
only the message — a backtrace is diagnostic data you render separately via
format!("{}", e.backtrace()), never baked into the human message.
8. The layered library error + a chain printer
This is the shape a real library ships: one public enum whose variants each wrap a
different lower-level error, a correct source() exposing every cause, and a way
to render the whole chain. The domain here is a three-level chain:
AppError::Config -> ConfigError::BadPort -> ParseIntError
(your enum) (your enum) (std)
#[derive(Debug)]
enum AppError {
Read { path: String, source: std::io::Error },
Config { source: ConfigError },
}
impl std::error::Error for AppError {
fn source(&self) -> Option<&(dyn Error + 'static)> {
match self {
AppError::Read { source, .. } => Some(source),
AppError::Config { source } => Some(source),
}
}
}
The anyhow-style {:#} printer flattens the chain into one line by walking
source() and joining each level’s Display with ": ":
fn format_chain(err: &dyn Error) -> String {
let mut chain = err.to_string();
let mut cur = err.source();
while let Some(next) = cur {
chain.push_str(&format!(": {next}"));
cur = next.source();
}
chain
}
// "invalid configuration: invalid port number: number too large to fit in target type"
The payoff lands here: because every layer kept its Display high-level and
pushed detail down into source(), the printer renders the full three-level story
with zero duplication. The separation rule you adopted in rung 3 is what makes
this clean.
Footguns
- Forgetting
impl Error.Displayalone is not an error. The emptyimpl Error for T {}is the marker that unlocksBox<dyn Error>,?, and downcasting. The compile error points at theBox::newcoercion, not the impl. ErrorneedsDebug.trait Error: Debug + Display— both supertraits are mandatory. Missing#[derive(Debug)]makesimpl Erroritself fail to compile.- Duplicating the cause in
Display. IfBadPort’sDisplaysays"invalid port: {source}", every chain printer prints the parse error twice. KeepDisplayto this layer; letsource()carry the rest. (Rung 4’sLoadErrordeliberately violates this withwrite!(f, "io error: {e}")— fine in isolation, but it would double-print under aformat_chain-style walk.) Rcin aSend + Syncerror. The auto-trait bounds propagate into fields. AnRc<_>(orRefCell<_>,*const _, etc.) anywhere inside makes the whole error!Send/!Syncand uncoercible toBox<dyn Error + Send + Sync>. Reach forArc/ thread-safe payloads.- Lifetime on
dyn Error + 'static. A function returning a borrow of a&(dyn Error + 'static)has two lifetimes in play (the reference and the'staticbound), so elision can’t pick — you must name it:fn chain<'a>(err: &'a (dyn Error + 'static)) -> Chain<'a>. - Reaching for unstable
Error::backtrace. It doesn’t exist on stable. Expose an inherent getter instead.
Real-world patterns
thiserror= this whole file, generated.#[derive(Error)]writes theDisplay(#[error("...")]), thesource()(#[source]/#[from]fields), and theFromimpls (#[from]). Doing it by hand once means you know exactly what the macro emits and can debug it when it surprises you.anyhow/eyre= rungs 6, 8, 9 packaged.anyhow::Erroris essentially aBox<dyn Error + Send + Sync>plus a captured backtrace, with.context()to push new layers,.downcast_ref::<T>()for recovery,{:#}for the one-line chain, and{:?}for the multi-lineCaused by:report.std::error::Error::sources()(still unstable) is exactly theChainiterator you build in the capstone.- Library/app split: libraries expose a typed enum (callers can match);
applications collapse everything into
anyhow::Error(callers only report). The typed error survives inside the erased one and can be recovered by downcast.
Capstone insight
anyhow’s rich error report — the thing that prints
invalid configuration
Caused by:
0: invalid port number
1: number too large to fit in target type
— is built from only the two trait methods you implemented in rungs 1 and 3. The capstone proves it by rebuilding the two reusable pieces:
// A: a lazy iterator over the source chain (std's unstable Error::sources()).
struct Chain<'a> { next: Option<&'a (dyn Error + 'static)> }
impl<'a> Iterator for Chain<'a> {
type Item = &'a (dyn Error + 'static);
fn next(&mut self) -> Option<Self::Item> {
let current = self.next.take()?; // ? ends iteration at the root
self.next = current.source(); // advance via the ONE trait method
Some(current)
}
}
// B: a Display wrapper that renders the multi-line report, built ON the iterator.
impl fmt::Display for Report<'_> {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{}", self.0)?;
if self.0.source().is_some() {
write!(f, "\n\nCaused by:")?;
for (i, src) in chain(self.0).skip(1).enumerate() {
write!(f, "\n {i}: {src}")?; // index 0 = FIRST cause, not the top
}
}
Ok(())
}
}
The take()? in next() is the elegant core: it yields the current error and
ends the iteration the moment source() returns None — the chain walk you wrote
three times by hand, now a normal Iterator you can .skip(1), .enumerate(),
.count(), or .collect(). Every “rich error” experience in the ecosystem reduces
to Display + source() plus an iterator over them. That’s the entire concept,
owned end to end.
Explain it back
- Why does an empty
impl Error for T {}matter — what stops compiling without it? - What two supertraits must every
Erroralready satisfy, and why does that force#[derive(Debug)]? - What does
?actually call on the error value before returning it, and what must you implement so a foreign error type propagates into your enum? - Why must
Displaynot include the text ofsource()? What breaks if it does? - Why can’t an error containing an
Rc<str>become aBox<dyn Error + Send + Sync>, and what’s the one-word payload fix? - How do you recover the concrete type from a
&dyn Error, and why isError: 'staticwhat makes that possible? capture()vsforce_capture()— which respectsRUST_BACKTRACE, and which one do libraries use by default?- Sketch the
Chainiterator’snext(). Why doesself.next.take()?correctly end the iteration at the root?
See also
- Error handling architecture —
thiserrorvsanyhow, the architecture layer built on top of this machinery. - Conversion traits —
From/Intoand how?leans on them. - Box & the Heap —
Box<dyn Trait>and unsizing coercions. - Static vs dynamic dispatch — what
dyn Erroris and how the vtable works.
Builder pattern
Ladder:
src/bin/builder.rs· Run:cargo run --bin builder· Phase 3 · 8 rungs
TL;DR
A builder is a half-built value that accumulates configuration through a
fluent method chain, then build() turns it into the real, validated thing.
It exists because a single new(a, b, c, d, e) constructor stops scaling the
moment a type has many fields — some optional, some defaulted, some mutually
constrained. The builder splits construction into named steps so callers set
only what they care about, and gives you exactly one place (build()) to
apply defaults and reject invalid combinations.
The two axes that define every builder:
- Ownership of the setter receiver —
selfby value (consuming) vs&mut self(mutating). This decides whether the builder is reusable and how it chains. - When validation happens — never, at runtime (
build() -> Result), or at compile time (typestate, wherebuild()doesn’t even exist until required fields are set).
Why this exists (from first principles)
Start with the problem. A telescoping constructor:
// What we're trying to avoid:
HttpRequest::new("POST", "https://x.com", "body", true, 30, vec![], None)
// ^ which arg is which? what's `true`? what's `30`?
Three things break here as the type grows:
- Unreadable call sites. Positional args give no clue what each value means,
and
true, true, falseis a bug waiting to happen. - No optionality. Every caller must pass every argument, even the ones they don’t care about. Adding a field is a breaking change to every call site.
- Scattered validation. If “port must be non-zero” matters, every constructor and setter has to re-check it, or callers can build nonsense.
The builder fixes all three: named setters document intent, unset fields fall
back to defaults, and build() is the single funnel where validity is decided.
The cost is a second type (the builder) and a small amount of boilerplate — which
is exactly what #[derive(Builder)] macros (the derive_builder crate) exist to
erase.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | Foundations | Consuming builder | self by value, build(self) -> T; chains because it returns Self |
| 2 | Foundations | &mut self builder | borrow-and-return; reusable, but build(&self) must clone |
| 3 | Mechanics | Optionals & defaults | Option<T> fields collapse to defaults in build() |
| 4 | Mechanics | Fallible build | required/invalid fields ⇒ build() -> Result<T, E> |
| 5 | Footgun | Temporary-drop trap | E0716 on a captured &mut chain; the owning-binding fix |
| 6 | Real-world | Repeatable setters + Into | accumulate into Vec/HashMap; impl Into<String> args |
| 7 | Real-world | Typestate builder | markers in the type; build() only on <Yes, Yes> |
| 8 | Capstone | Real config builder | consume + Into + optionals + repeatable + validated build |
The ideas, built up
1. The consuming builder: chaining falls out of the signature
The whole pattern hinges on one signature shape: a setter takes self by
value and returns Self.
fn method(self, m: &str) -> Self {
HttpRequestBuilder { method: m.to_string(), ..self }
}
fn build(self) -> HttpRequest {
HttpRequest { method: self.method, url: self.url, body: self.body }
}
Because each setter consumes and returns the builder, the only thing you can
do with the result is call the next method on it — that is what makes
Builder::new().method(..).url(..).build() read as one fluent chain.
Two details worth internalizing:
..self(functional update syntax) moves the remaining fields out of the old builder into the new one. That’s sound precisely because you ownselfby value and are discarding the old builder — “I own it, so I can dismantle it.” A&self/&mut selfsetter could not do this.buildmoves fields out for free.self.methodis a move, not a clone, becausebuildconsumedself. Hold onto this — rung 2 pays a clone tax for giving it up.
2. The &mut self builder: reusable, at the cost of a clone
The mirror-image choice: borrow &mut self, mutate one field, return
&mut Self. build then takes &self.
fn method(&mut self, m: &str) -> &mut Self {
self.method = m.to_string();
self // auto-reborrows as &mut Self
}
fn build(&self) -> HttpRequest {
HttpRequest {
method: self.method.clone(), // must clone — only a shared borrow
url: self.url.clone(),
body: self.body.clone(),
}
}
What this buys you, that rung 1 cannot:
let mut b = ReqBuilder::new();
b.url("https://reuse.test").method("GET");
let r1 = b.build(); // builder still alive
b.method("DELETE");
let r2 = b.build(); // build again, tweaked
The builder survives build(), so you can build twice, or conditionally set
fields across statements. The price: build(&self) only has a shared borrow, so
it cannot move the fields out — it must clone them. That is the fundamental
trade between the two styles.
Consuming (self) | Mutating (&mut self) | |
|---|---|---|
build cost | moves fields (free) | clones fields |
| Reusable after build | no (consumed) | yes |
| Chains as one expression | yes | yes |
Capture partial chain in a let | yes (owns Self) | no — see rung 5 |
3. Optionals & defaults: model “unset” honestly
A real builder lets callers set only what they care about. Model that directly:
make every builder field an Option, starting None; a setter stores Some;
build() resolves each None to a default.
#[derive(Default)] // gives you new() = Self::default() for free
struct ServerOptsBuilder {
host: Option<String>,
port: Option<u16>,
// ...
}
fn build(&self) -> ServerOpts {
ServerOpts {
host: self.host.clone().unwrap_or_else(|| "127.0.0.1".to_string()),
port: self.port.unwrap_or(8080),
// ...
}
}
The key split: Option-ness lives only in the builder. The finished
ServerOpts has plain String/u16 fields — build() is where “the caller
never set this” collapses into a concrete value.
unwrap_orvsunwrap_or_else.unwrap_or(x)evaluatesxeagerly, every time, even when theOptionisSome. For the cheapCopydefault8080that’s fine. Forhost,unwrap_or("127.0.0.1".to_string())would allocate that string on every build even when the host was set. Useunwrap_or_else(|| ...)so the default is computed only on theNonepath.
4. Fallible build: one validation checkpoint
Defaults cover fields that have a sensible default. But some fields are
genuinely required (a Connection with no name), and some values are invalid
(port == 0). The builder can’t stop a caller from leaving name unset — so
build() becomes the single checkpoint that returns Result.
fn build(&self) -> Result<Connection, BuildError> {
let port = self.port.unwrap_or(8080);
if port == 0 {
return Err(BuildError::InvalidPort);
}
Ok(Connection {
name: self.name.clone().ok_or(BuildError::MissingName)?,
port,
retries: self.retries.unwrap_or(3),
})
}
ok_or(err)turnsOption<T>intoResult<T, E>; the?then early-returns the error and unwraps theStringon the happy path.- Resolve a default then validate (
unwrap_or(8080)before the== 0check).
The lesson: no matter how the caller chained the builder, every path funnels through this one function. Invalid states are caught in exactly one place.
5. The temporary-drop footgun (and its fix)
The &mut self builder chains fine in one expression, because the temporary
builder lives until the end of the statement. The trap appears when you try to
capture a partially-built &mut builder in a let:
// WRONG — E0716 "temporary value dropped while borrowed"
let builder = ConnectionBuilder::new().name("db").port(5432);
let conn = builder.build().unwrap();
Why it fails: new() produces a temporary ConnectionBuilder. .name().port()
return &mut references into that temporary. At the ;, the temporary is
dropped — so builder would be a reference to freed memory. The borrow checker
refuses.
The fix: give the builder an owning binding first, then call setters on it.
Now the references the setters return are created and dropped within each
statement, while the owner b stays alive.
// OK
let mut b = ConnectionBuilder::new();
b.name("svc");
b.port(5432);
if many_retries { b.retries(10); } // the across-statements case &mut excels at
b.build()
This is the defining difference between the two styles: the consuming builder
(which returns owned Self) can be freely split across let bindings; the
&mut builder cannot, because its intermediate values are borrows, not values.
6. Repeatable setters + Into bounds: real-world ergonomics
Two tricks every production builder uses.
Repeatable setters accumulate instead of overwrite. The field is a
collection; the setter pushes/inserts. Calling .to(..) three times appends
three entries (this is exactly how reqwest::RequestBuilder::header works).
fn to(&mut self, addr: impl Into<String>) -> &mut Self {
self.to.push(addr.into()); // append, don't replace
self
}
fn header(&mut self, k: impl Into<String>, v: impl Into<String>) -> &mut Self {
self.headers.insert(k.into(), v.into());
self
}
impl Into<String> arguments let callers pass &str or String (or
anything convertible) without sprinkling .to_string() at every call site. You
call .into() once inside the setter to normalize to the owned type.
b.to("a@x.com") // &str
.to(owned_string) // String — Into<String> covers both
.header("X-Env", "prod");
7. Typestate: make a missing field a compile error
Rung 4 caught a missing required field at runtime (Err(MissingName)). Typestate
moves that check into the type system: build() simply does not exist until
every required field is set.
Encode “is this field set?” in a generic type parameter, using zero-sized marker
types. Each required setter returns a different type with its marker flipped
from No to Yes.
struct Yes;
struct No;
struct ApiCallBuilder<E, T> { // E = endpoint-set?, T = token-set?
endpoint: Option<String>,
token: Option<String>,
timeout_ms: Option<u64>,
_state: PhantomData<(E, T)>,
}
impl ApiCallBuilder<No, No> {
fn new() -> Self { /* both markers start at No */ }
}
impl<E, T> ApiCallBuilder<E, T> {
// flips E -> Yes, THREADS T through unchanged
fn endpoint(self, e: &str) -> ApiCallBuilder<Yes, T> {
ApiCallBuilder { endpoint: Some(e.to_string()), token: self.token,
timeout_ms: self.timeout_ms, _state: PhantomData }
}
fn token(self, t: &str) -> ApiCallBuilder<E, Yes> { /* flips T, threads E */ }
}
// build() EXISTS ONLY for the fully-set type:
impl ApiCallBuilder<Yes, Yes> {
fn build(self) -> ApiCall {
ApiCall {
endpoint: self.endpoint.unwrap(), // .unwrap() is HONEST here —
token: self.token.unwrap(), // the <Yes,Yes> bound proves Some
timeout_ms: self.timeout_ms.unwrap_or(30_000),
}
}
}
The result:
let bad = ApiCallBuilder::new().endpoint("x").build();
// error[E0599]: no method named `build` found for ApiCallBuilder<Yes, No>
Three things make this work:
PhantomData<(E, T)>lets you carry the marker type-params without storing any value of them — they’re compile-time-only state.- Setters are generic over the other marker (
endpointisimpl<E, T>, returns<Yes, T>). ThreadingTthrough unchanged is what remembers “token was already set” across the call. - Because the return type differs from
Self, you can’t use..self— the source and target are different types, so you move each field across by hand.
The payoff: .unwrap() in build() is provably correct. The type
<Yes, Yes> is a proof that both fields are Some, so there is no runtime check
left to do — the compiler already did it.
Footguns
| Footgun | What bites | Fix |
|---|---|---|
Capturing a &mut chain in a let | E0716, temporary dropped while borrowed (rung 5) | bind the builder to an owner first, then call setters |
unwrap_or for an allocating default | allocates on every build, even when the field was set (rung 3) | unwrap_or_else(|| ...) — lazy |
Forgetting build consumes self in typestate | can’t reuse the builder after build() | intended — typestate transitions are one-shot |
| Repeatable setter that assigns instead of pushes | silently overwrites previous values | .push / .insert, never = |
&mut self build(&self) | must clone every field out of a shared borrow | use the consuming style if you want moves |
Real-world patterns
Foo::builder()entry point. Rather than a freeFooBuilder::new(), expose aFoo::builder()associated function — it’s discoverable from the type you actually want and reads asFoo::builder()...build()(std/tokio/reqwestall do this).- Consuming style for one-shot config,
&mutfor reuse.std::process::Commanduses&mut self(so you can conditionally.arg(..)in a loop);reqwest::ClientBuilderconsumes. Pick by whether callers need to reuse the builder. - Repeatable setters +
Intoeverywhere is the house style of HTTP/builder crates:.header(k, v)accumulates, all string args areimpl Into<String>. derive_builder/bongenerate all of this from the struct definition. Knowing the hand-rolled shape is what lets you read and debug the macro output.- Typestate (
bon’s required fields, embedded HALs) for APIs where a missing step should be a compile error, not a runtime panic.
Capstone insight
The capstone (ServerConfig::builder()) fuses every rung into one idiomatic API:
a consuming fluent chain, impl Into<String> args, Option fields with defaults,
repeatable .route(..)/.env(k, v) setters, and a single fallible build() that
validates everything and ends the chain with .build()?.
The structural “aha”: because build consumes self, it can move
routes and env straight into the finished ServerConfig — no clone, unlike
the &mut-style build(&self) of rungs 2 and 6.
fn build(self) -> Result<ServerConfig, ConfigError> {
let bind_addr = self.bind_addr.ok_or(ConfigError::MissingBindAddr)?;
let port = self.port.unwrap_or(8080);
let workers = self.workers.unwrap_or(4);
if port == 0 { return Err(ConfigError::ZeroPort); }
if workers == 0 { return Err(ConfigError::ZeroWorkers); }
if self.routes.is_empty() { return Err(ConfigError::NoRoutes); }
Ok(ServerConfig {
bind_addr, port, workers,
routes: self.routes, // MOVED, not cloned — we own self
env: self.env, // MOVED
})
}
That ok_or(...)? for the required field plus the move-not-clone of the
collections is the senior-Rustacean shape of a builder. Everything else —
defaults, validation, repeatable setters — hangs off those two decisions:
who owns the receiver, and where validity is decided.
Explain it back
- Why does a setter return
Self/&mut Selfat all? What would break if it returned()? - What exactly does
..selfdo, and why is it sound only in the consuming style? - Why does
build(&self)in the&mutbuilder have to clone, whilebuild(self)in the consuming builder can move? - Reproduce the E0716 temporary-drop error from memory. Why does an owning
letbinding fix it? - When should a setter accumulate (
.push) vs overwrite (=)? Give an example of each. - In the typestate builder, why is
endpoint’s impl blockimpl<E, T>and notimpl ApiCallBuilder<No, No>? What does threadingTthrough accomplish? - Why is
.unwrap()in the typestatebuild()not a code smell? unwrap_orvsunwrap_or_else— when does the difference actually matter?
See also
- Custom error types — the
BuildError/ConfigErrorenums the fallible builds return. - Error handling architecture —
Result,?, and where validation errors belong. - Conversion traits —
Into/From, the engine behindimpl Into<String>setters.
The typestate pattern
Ladder:
src/bin/typestate.rs· Run:cargo run --bin typestate· Phase 3 · 9 rungs
TL;DR
Typestate moves a value’s state out of its runtime fields and into its
type. Instead of one Door { is_open: bool } you check at runtime, you have
two distinct types — Door<Open> and Door<Closed> — and you write the methods
that only make sense in one state inside that state’s own impl block. Calling
.close() on a Door<Closed> is then not a runtime error or a panic: it is a
compile error, because the method literally does not exist on that type.
Three mechanical pillars hold it up:
- State as a type parameter, carried by a zero-sized
PhantomData<State>field — so the whole scheme costs zero bytes at runtime. - Transitions consume
selfby value and return the new state type, so the old handle is moved away and a stale state is unusable. impl Type<ThisState>gates each method to the state where it’s valid.
The payoff: an entire class of “wrong order” and “wrong state” bugs becomes
unrepresentable. The cost: states must be known at compile time, so at runtime
boundaries you bridge through an enum.
Why this exists (from first principles)
Start with the bug we want to delete. A connection with a runtime flag:
struct Conn { state: State, /* ... */ }
enum State { Idle, Established, Closed }
impl Conn {
fn send(&mut self, data: &[u8]) {
// Is this even legal right now?
if self.state != State::Established {
panic!("send() called on a {:?} connection", self.state); // runtime!
}
// ...
}
}
Everything about correctness here is deferred to runtime:
send()on a closed connection compiles fine. It only blows up when that line actually executes — maybe in production, maybe in a rare branch your tests miss.- Every method has to re-check the flag, and every check is a place to forget.
- The type
Connclaims to supportsend,connect, andcloseall the time, which is a lie — each is valid only in some states.
Typestate’s move is to make the compiler the enforcer. If send only exists on
Conn<Established>, then code holding a Conn<Closed> cannot name send —
there’s nothing to call, nothing to check at runtime, nothing to test. The
illegal program doesn’t compile, which is the strongest guarantee Rust offers.
The mental shift: stop storing the state as data; start encoding it as a type. A
boolhas two values you check; two types have two vocabularies of methods the compiler enforces.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | Foundations | door_basics | Door<State> with ZST markers + PhantomData; building a Door<Closed> |
| 2 | Foundations | state_methods | open() on Closed only, close() on Open only; the wrong call won’t compile |
| 3 | Mechanics | consuming_transitions | self-by-value transitions thread a data payload through; the stale handle is moved away |
| 4 | Mechanics | zst_and_sealed | size_of proves zero cost; a sealed State trait closes the state set |
| 5 | Footgun | phantom_required | omit PhantomData ⇒ E0392 “parameter never used”; why Rust insists |
| 6 | Footgun | runtime_boundary | typestate is compile-time only; erase to an enum and re-enter via match |
| 7 | Real-world | typestate_builder | required fields tracked in the type; build() exists only when complete |
| 8 | Real-world | generic_over_state | impl<S: State> + associated const for behavior shared by every state |
| 9 | Capstone | protocol_capstone | a TCP-like state machine: sealed states, typed transitions, runtime event loop |
The ideas, built up
1. The state lives in the type, not a field
A state marker is just a zero-sized struct. The stateful type carries it only as a phantom:
struct Open; // marker — zero fields, zero bytes
struct Closed;
struct Door<State> {
_state: PhantomData<State>, // the ONLY "field"
}
impl Door<Closed> {
fn new() -> Door<Closed> {
Self { _state: PhantomData }
}
}
There is no Closed value anywhere — you can’t store one, there’s nothing to
store. The <Closed> in the return type is what fixes State = Closed; the
PhantomData is the placeholder that satisfies the field. The proof it’s free:
assert_eq!(std::mem::size_of::<Door<Closed>>(), 0);
Door<Open> and Door<Closed> are different types that happen to have
identical (empty) layout. That difference is invisible at runtime and total at
compile time.
2. Gate methods by writing them in the state’s impl
The whole pattern is this asymmetry: a method goes in the impl block for the
state where it’s valid.
impl Door<Closed> {
fn open(self) -> Door<Open> { Door { _state: PhantomData } }
}
impl Door<Open> {
fn close(self) -> Door<Closed> { Door { _state: PhantomData } }
}
open exists only on Door<Closed>; close only on Door<Open>. So:
let d = Door::<Closed>::new();
d.close(); // WRONG: error[E0599] no method named `close` found for `Door<Closed>`
That error is the pattern. Not a panic, not an Err — the program that closes
a closed door is rejected before it can run.
3. Transitions consume self — and that’s the safety, not a style choice
Look at the signature: fn open(self, ...) -> Door<Open>. Taking self by
value means the transition moves the old door. After it returns, the old
handle is gone. This is what makes a stale state impossible to use, which
matters the moment a value carries data:
struct File<State> {
path: String,
buffer: Vec<u8>,
_state: PhantomData<State>,
}
impl File<Closed> {
fn open(self) -> File<Open> {
File { path: self.path, buffer: self.buffer, _state: PhantomData } // MOVE the data across
}
}
impl File<Open> {
fn write(&mut self, bytes: &[u8]) { self.buffer.extend_from_slice(bytes); } // &mut: mutate, not transition
fn close(self) -> (File<Closed>, usize) {
let flushed = self.buffer.len();
(File { path: self.path, buffer: Vec::new(), _state: PhantomData }, flushed)
}
}
Two things to internalize:
-
Data is threaded by moving fields, not cloning. You own
self, sopath: self.pathmoves theStringinto the new state for free. A transition is “same data, new type tag.” -
A consumed handle can’t be revived:
let g = File::<Closed>::new("x").open(); let _ = g.close(); // g moved here g.write(b"!"); // WRONG: error[E0382] use of moved value: `g`Use-after-close is the same compile error as use-after-free. The type system’s move semantics are doing state-machine enforcement for free.
Note the receiver choice encodes intent:
selffor a transition (you become a new state),&mut selffor an in-state mutation (writekeeps youOpen).
4. Close the set of legal states with a sealed trait
Bare Door<State> lets anyone write Door<i32> or Door<String>. To say “there
are exactly these states and no others,” bound the parameter with a trait — and
seal that trait so downstream code can’t implement it:
mod door_sealed {
trait Sealed {} // PRIVATE to this module
pub trait State: Sealed {} // public, but requires the private Sealed
pub struct Open2;
pub struct Closed2;
impl Sealed for Open2 {} impl State for Open2 {}
impl Sealed for Closed2 {} impl State for Closed2 {}
pub struct Door2<S: State> { // only real states allowed
_state: PhantomData<S>,
}
}
The mechanism: to implement the public State, a type must also satisfy the
supertrait Sealed — but Sealed is private to door_sealed, so no code
outside this module can ever impl it. Outsiders can name State (e.g. to
write fn f<S: State>()) but can never add a new one. Now:
let _bad: Door2<i32> = /* ... */; // WRONG: error[E0277] the trait bound `i32: State` is not satisfied
This is the sealed trait pattern, and it’s exactly how clap, tokio, and
many stdlib traits keep an “internal only” set extensible by the author but
closed to users. The compiler even warns you the seal is working:
warning: trait Sealed is more private than the item State — that asymmetry is
the whole point.
5. impl<S: State> for what every state shares — plus associated consts
Per-state impls gate state-specific methods. For methods that make sense in
every state, write one generic block, and let an associated const carry
per-state data:
trait ConnState { const NAME: &'static str; }
struct Connecting; impl ConnState for Connecting { const NAME: &str = "connecting"; }
struct Connected; impl ConnState for Connected { const NAME: &str = "connected"; }
struct Disconnected; impl ConnState for Disconnected { const NAME: &str = "disconnected"; }
impl<S: ConnState> Conn<S> {
fn id(&self) -> u32 { self.id }
fn state_name(&self) -> &'static str { S::NAME } // read the type's const
fn reset(self) -> Conn<Disconnected> { // a transition valid from ANY state
Conn { id: self.id, _s: PhantomData }
}
}
state_name returns a string it never stored — it reads S::NAME off the type
parameter. The type is the lookup table. And reset is a single generic
transition usable from every state, instead of one copy per state.
Footguns
PhantomData is not optional — E0392
If you declare a state parameter S but no field mentions it, the compiler flatly
rejects the struct:
struct Lock<S> { held_by: String } // WRONG
// error[E0392]: type parameter `S` is never used
// help: consider removing `S`, referring to it in a field, or using `PhantomData`
Why does Rust care, when S changes nothing about the layout? Because an unused
parameter still affects the type’s identity, variance, drop-check, and
auto-trait (Send/Sync) reasoning — and the compiler refuses to silently
guess which meaning you intended. PhantomData<S> is the explicit answer: “treat
this as if it owns an S,” at zero byte cost.
struct Lock<S> { held_by: String, _state: PhantomData<S> } // OK
// size_of::<Lock<Unlocked>>() == size_of::<String>() — the tag is free
Typestate can’t choose a state at runtime
A value’s type is fixed at compile time. It cannot depend on a runtime if:
// There is no way to write this:
let valve = if config_says_open { Valve::<Open> } else { Valve::<Closed> }; // types differ — won't compile
When the state comes from a config file, a network byte, or user input, you must
leave the type world at that boundary. Erase the state into an enum:
enum AnyValve { Open(Valve<Open>), Closed(Valve<Closed>) }
impl AnyValve {
fn parse(s: &str) -> Result<AnyValve, String> { // ENTER from runtime data
match s {
"open" => Ok(AnyValve::Open(Valve { _state: PhantomData })),
"closed" => Ok(AnyValve::Closed(Valve { _state: PhantomData })),
_ => Err(format!("invalid valve state: {s}")),
}
}
fn state_name(&self) -> &'static str { // RE-ENTER: each arm is a concrete typed value
match self { AnyValve::Open(_) => "open", AnyValve::Closed(_) => "closed" }
}
}
The senior mental model is a sandwich: enums at the I/O edges, strong
typestate in the middle. parse erases runtime input into the enum; match
re-enters the typed core where each arm holds a concrete Valve<Open> /
Valve<Closed> and can call its real typed methods. Typestate doesn’t replace
enums — it complements them.
Real-world patterns
The typestate builder: required fields enforced at compile time
The flagship application. Track “has this required field been set?” in a type
parameter per field, and implement build() only for the all-set combination:
struct Yes; struct No;
struct ReqBuilder<U, M> { // U = url set? M = method set?
url: Option<String>, method: Option<String>, body: Option<String>,
_u: PhantomData<U>, _m: PhantomData<M>,
}
impl<U, M> ReqBuilder<U, M> {
fn url(self, url: impl Into<String>) -> ReqBuilder<Yes, M> { // flip U, KEEP M
ReqBuilder { url: Some(url.into()), method: self.method, body: self.body,
_u: PhantomData, _m: self._m }
}
fn method(self, m: impl Into<String>) -> ReqBuilder<U, Yes> { /* flip M, keep U */ }
}
impl ReqBuilder<Yes, Yes> { // build() EXISTS ONLY here
fn build(self) -> Request {
Request { url: self.url.unwrap(), method: self.method.unwrap(), body: self.body }
}
}
Two insights that make this click:
- Setters are generic over the other parameter.
url()returnsReqBuilder<Yes, M>— it flipsUtoYesbut preserves whateverMyou already had. That’s why the chain works in any order: each setter touches only its own axis. This is the type-level mirror of “thread the data through a transition” from rung 3. unwrap()inbuild()is provably infallible. The<Yes, Yes>type is the proof thaturlandmethodareSome. This is one of the rare, legitimate uses ofunwrap— the typestate discharges the panic.
And the payoff:
ReqBuilder::new().url("/x").build();
// WRONG: error[E0599] no method named `build` found for `ReqBuilder<Yes, No>`
A forgotten required field is a compile error, with no runtime validation and
no Result. This is what the typed-builder crate’s derive macro generates for
you; here you’ve built it by hand.
Capstone insight
The capstone wires every tool into one small TCP-like lifecycle:
Idle --connect--> Handshaking --synack--> Established --close--> Closed
- Sealed
Protocoltrait with a per-stateconst NAME, generated by a tinymacro_rules!— a peek at how real crates erase the four-lineimpl Sealed + impl Traitboilerplate per state. - Typed transitions (
connect,synack,close) that consumeselfand threadpeer/bytes_sentacross, plus asend(&mut self)valid only whileEstablished. - Generic accessors (
state_name,peer,bytes_sent) in oneimpl<S: Protocol>block. - A runtime event loop that erases the state into
AnyConnand drives the machine from strings:
pub fn step(self, event: &str) -> AnyConn {
match self {
AnyConn::Idle(c) => match event.split_once(':') {
Some(("connect", peer)) => AnyConn::Handshaking(c.connect(peer)),
_ => AnyConn::Idle(c), // out-of-state event: ignored
},
AnyConn::Handshaking(c) => match event {
"synack" => AnyConn::Established(c.synack()),
_ => AnyConn::Handshaking(c),
},
AnyConn::Established(mut c) => match event.split_once(':') {
Some(("send", data)) => { c.send(data.as_bytes()); AnyConn::Established(c) }
_ if event == "close" => { let (c, _) = c.close(); AnyConn::Closed(c) }
_ => AnyConn::Established(c),
},
AnyConn::Closed(c) => AnyConn::Closed(c),
}
}
The structural “aha”: match on the state first, the event second. Each state’s
catch-all arm (_ => self unchanged) handles “drop out-of-state packets” without
enumerating every bad combination — a real server silently ignores a SYN on an
established connection, it doesn’t crash. Inside each arm you hold the concrete
typed Conn<...> and call its real typed transition: the enum is just the
runtime carrier, and the moment you match you’re back in the strongly-typed
world. That is the typestate sandwich at full size — a statically-verified core
wrapped in a thin dynamic boundary.
The Established(mut c) binding is the one subtlety: send takes &mut self but
you own c by value, so you bind it mut, mutate in place, and re-wrap it in the
same AnyConn::Established variant.
Explain it back
- Why is
Door<Open>andDoor<Closed>better thanDoor { is_open: bool }? What error does the bad call become, and when? - Why must transitions take
selfby value? What bug does the resulting move prevent? - What is
PhantomData<S>for, and what exact error appears without it? Why does the compiler refuse to just ignore an unused parameter? - How does a sealed trait close the set of states, and why can’t a downstream crate add one? What’s the role of the private supertrait?
- Why can’t typestate pick a state from runtime input, and what’s the standard bridge? Describe the “enum at the boundary, types in the middle” sandwich.
- In the typestate builder, why is
url()generic overM? Why is theunwrap()inbuild()actually safe?
See also
- Builder pattern — rung 7 there is the same typestate-builder idea in its native habitat.
- Blanket impls & coherence — the sealed-trait pattern and why downstream impls are (or aren’t) allowed.
- Generic bounds &
whereclauses — conditionalimpls and bounding the method, not the struct, which gates state-specific methods. Drop& ordering — RAII guards, the other side of “the type enforces a protocol.”
Newtype & zero-cost wrappers
Ladder:
src/bin/newtype.rs· Run:cargo run --bin newtype· Phase 3 · 9 rungs
TL;DR
A newtype is a one-field tuple struct that wraps an existing type:
struct Meters(f64). At runtime it is nothing — same bits, same size, the
wrapper compiles away. But to the type checker it is a brand-new, distinct type.
You spend the compiler’s type system to buy back guarantees the raw type cannot give you:
- Distinct identity —
MetersandSecondsstop being interchangeable. - Your own trait impls — you control
Add,Display,Deref, etc., and you can implement foreign traits on foreign types by wrapping them. - Enforced invariants — a private field plus a smart constructor makes the type itself a proof that the data is valid.
The runtime bill for all of this is zero. The recurring tension to manage: a
newtype hides its inner type by default, and Deref lets you leak the inner
API back for ergonomics — leak too much and the wrapper stops protecting
anything.
Why this exists (from first principles)
Consider a function that computes speed:
fn speed(distance: f64, time: f64) -> f64 {
distance / time
}
Nothing stops a caller writing speed(time, distance). Both arguments are
f64, so the swap type-checks, runs, and silently returns garbage. The type
system has been told these two numbers are the same kind of thing — but they are
not. A meter and a second are different physical quantities.
The fix is to give each its own type:
#[derive(Debug, Clone, Copy)]
struct Meters(f64);
#[derive(Debug, Clone, Copy)]
struct Seconds(f64);
fn speed(distance: Meters, time: Seconds) -> f64 {
distance.0 / time.0
}
Now speed(t, d) is a compile error (expected Meters, found Seconds). The
information “this number is a distance” was lost in the f64 version; the
newtype encodes it back into the type, and the compiler enforces it for free.
distance.0 reaches the inner f64 — the single field of a tuple struct is
named .0.
The newtype’s superpower is the absence of an impl.
Meters + Secondswon’t compile not because anyone forbade it, but because you never wrote that impl. Safety by omission.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | Foundations | Distinct identity | Meters vs Seconds; swapping args is a type error |
| 2 | Foundations | Deriving the basics | A newtype has no behavior until you derive it; derives forward to the inner type |
| 3 | Mechanics | Type-safe arithmetic | impl Add defines the algebra; Meters + Seconds simply doesn’t exist |
| 4 | Mechanics | Deref for ergonomics | Wrap String, deref to str, get its methods for free via coercion |
| 5 | Footgun | The Deref leak | SortedVec must not deref to Vec — that would leak .push and break the invariant |
| 6 | Footgun | Orphan-rule escape hatch | impl Display for a foreign type by wrapping it in a local newtype |
| 7 | Real-world | repr(transparent) | Prove the layout is identical to the inner type; sound slice reinterpret |
| 8 | Real-world | Parse, don’t validate | Email with a private field + smart constructor; the type proves validity |
| 9 | Capstone | Phantom-typed Id<T> | One generic newtype gives Id<User> != Id<Post>, zero-cost, HashMap key |
The ideas, built up
1. A newtype starts with no behavior
A tuple struct inherits nothing from its inner type. UserId(u64) cannot be
printed, compared, copied, or sorted — even though the u64 inside can do all of
those. Every capability must be granted explicitly:
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
struct UserId(u64);
Each derive generates an impl that simply forwards to the inner field. UserId(3) < UserId(9) compares the two u64s; == compares them; .max() works because
Ord + Copy are present:
fn max_id(ids: &[UserId]) -> UserId {
ids.iter().copied().max().unwrap()
}
.copied() is only valid because we derived Copy; .max() only because we
derived Ord. Without those derives, this is a wall of E0277/E0599 errors —
which is the lesson: a newtype is opt-in.
EqneedsPartialEq,OrdneedsPartialOrd. They are supertraits. You derive both halves.
2. You define the algebra (Add)
== and < are derivable; + is not. To add two Meters you implement
std::ops::Add yourself — and that is a feature, because you decide what
arithmetic is meaningful:
use std::ops::Add;
impl Add for Meters {
type Output = Meters;
fn add(self, rhs: Meters) -> Meters {
Meters(self.0 + rhs.0)
}
}
type Output is the associated type that says “adding two Meters yields a
Meters”. Because the only Add impl in scope is Meters + Meters,
Meters + Seconds has no impl and is rejected (E0277). This is exactly how
std::time::Duration works: Duration + Duration is defined, Duration + u64
is not.
fn total(distances: &[Meters]) -> Meters {
distances.iter().copied().fold(Meters(0.0), |acc, d| acc + d)
}
The fold starts from Meters(0.0) and threads your + through the slice.
3. Deref for ergonomics
Sometimes you want the wrapper to behave like the thing it wraps. Implementing
Deref makes &Wrapper coerce to &Target, so the target’s methods and any
&Target-taking function work on the wrapper directly:
use std::ops::Deref;
struct Username(String);
impl Deref for Username {
type Target = str;
fn deref(&self) -> &str {
&self.0 // &String coerces to &str
}
}
Two distinct mechanisms now kick in:
- Method resolution walks the deref chain.
username.len()finds nolenonUsername, derefs tostr, and callsstr::len. - Deref coercion lets
&Usernamebe passed where&stris expected:greet(&u)compiles even thoughgreet(name: &str).
let u = Username(String::from("ferris"));
assert_eq!(u.len(), 6); // Username -> str
assert_eq!(greet(&u), "Hello, ferris!"); // &Username coerces to &str
This is the same machinery that lets you call &str methods on a String, or
&T methods on a Box<T>.
Footguns
The Deref leak
Deref is convenient enough to be dangerous. The temptation is to slap
impl Deref<Target = Vec<i32>> on any wrapper to “inherit” the inner API. But if
the wrapper exists to enforce an invariant, deref leaks the very methods that
break it.
SortedVec keeps its Vec<i32> sorted. If it derefed to Vec, a caller could
reach .push, .swap, or (with DerefMut) mutate the buffer out of order and
silently violate “sorted”. The ladder deliberately does not implement
Deref. Instead it exposes a curated API:
struct SortedVec(Vec<i32>);
impl SortedVec {
fn insert(&mut self, value: i32) {
// partition_point finds the first index where x >= value
self.0.insert(self.0.partition_point(|&x| x < value), value);
}
fn as_slice(&self) -> &[i32] {
&self.0 // read-only window: no .push leaks out
}
}
// sv.push(0); // does NOT compile — push doesn't exist on SortedVec
That non-compilation is the invariant being protected structurally.
Rule of thumb:
Derefis for smart pointers (Box,Rc,Arc), where the wrapper genuinely is a stand-in for the inner value. For an invariant-holding newtype, expose a curated API, notDeref. The Rust API guidelines say the same: don’timpl Derefto emulate inheritance.
The orphan rule (and the escape hatch)
The orphan rule: you may implement a trait for a type only if the trait or
the type is local to your crate. So impl Display for Vec<i32> is illegal
(E0117) — both Display and Vec are foreign.
The newtype is the escape hatch. Wrap the foreign type in a local struct and the type is now yours, so the impl is legal:
use std::fmt;
struct PrettyVec(Vec<i32>);
impl fmt::Display for PrettyVec {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "[")?;
for (i, v) in self.0.iter().enumerate() {
if i > 0 { write!(f, ", ")?; }
write!(f, "{v}")?;
}
write!(f, "]")
}
}
PrettyVec(vec![1, 2, 3]).to_string() is "[1, 2, 3]". This is exactly how
crates add Display, serde::Serialize, and other foreign traits to types they
do not own.
Real-world patterns
#[repr(transparent)] — zero-cost, guaranteed
“Zero-cost” stops being a slogan when you reach for the layout. A newtype over
T has the same size and alignment as T. The optimizer usually exploits
this, but #[repr(transparent)] makes it a guaranteed, ABI-stable fact: the
struct is laid out exactly like its single non-zero-sized field.
use std::mem::{align_of, size_of};
#[derive(Debug, Clone, Copy, PartialEq)]
#[repr(transparent)]
struct Wrapping64(u64);
assert_eq!(size_of::<Wrapping64>(), size_of::<u64>()); // 8 == 8
assert_eq!(align_of::<Wrapping64>(), align_of::<u64>());
The guarantee is what makes it sound to reinterpret a slice of the newtype as a slice of the raw type, with no copy:
fn as_raw_slice(xs: &[Wrapping64]) -> &[u64] {
// SAFETY: Wrapping64 is #[repr(transparent)] over u64, so each element has
// identical layout and every Wrapping64 is a valid u64. The pointer cast and
// length are therefore valid for a &[u64] over the same memory.
unsafe {
std::slice::from_raw_parts(xs.as_ptr() as *const u64, xs.len())
}
}
Direction matters. This cast is sound because every bit pattern of
u64is a validu64. The reverse —&[u64]to&[NonZeroU64]— would be UB for a zero, becauseNonZeroU64has a validity niche.transparentguarantees layout, not that arbitrary bytes are valid.repr(transparent)is also what makes a newtype safe to pass across an FFI boundary where C expects the raw type.
Parse, don’t validate (the validated newtype)
The most powerful newtype move: make the type itself a proof that an invariant holds. Put the data behind a private field, offer no public constructor, and check the invariant exactly once in a fallible smart constructor:
mod email {
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Email(String); // private field: only this module can build one
#[derive(Debug, PartialEq, Eq)]
pub enum EmailError { Empty, MissingAt }
impl Email {
pub fn parse(s: &str) -> Result<Email, EmailError> {
if s.is_empty() { return Err(EmailError::Empty); }
if !s.contains('@') { return Err(EmailError::MissingAt); }
Ok(Email(s.to_string()))
}
pub fn as_str(&self) -> &str { &self.0 }
}
}
The field privacy is the whole trick: code outside mod email literally cannot
write Email(whatever), so the only way to obtain an Email is through
parse. Once you hold one, it is guaranteed to have passed the check. Downstream
code never re-validates:
fn send_to(addr: &email::Email) -> String {
format!("sending to {}", addr.as_str()) // no validation needed
}
You cannot even call send_to with an unvalidated string — there is no way to
construct the argument. This is “parse, don’t validate”: turn unstructured input
into a type that cannot represent the invalid state. It is the pattern behind
std::num::NonZeroU32, url::Url, and most well-designed domain types.
Capstone insight
A database layer hands out numeric ids for every table. Plain u64 ids are a bug
factory — nothing stops you passing a user’s id where a post’s id is expected.
You could write UserId, PostId, OrderId by hand, but that is endless
boilerplate.
Instead, one generic newtype with a phantom type tag:
use std::marker::PhantomData;
struct User; // pure markers — carry no data
struct Post;
struct Id<T> {
raw: u64,
_tag: PhantomData<T>, // "generic over T" without storing a T
}
impl<T> Id<T> {
fn new(raw: u64) -> Id<T> { Id { raw, _tag: PhantomData } }
fn get(&self) -> u64 { self.raw }
}
PhantomData<T> is a zero-sized marker that lets the struct be generic over T
without holding one. Id<User> and Id<Post> are now distinct types that cannot
be mixed, yet each is still just a u64 at runtime. assert_eq!(u1, p1) where
u1: Id<User> and p1: Id<Post> is a compile error.
The subtle part: don’t let the derive bound your tag
You want Id<T> to be Copy + Clone + PartialEq + Eq + Hash + Debug for every
T, so it can be a HashMap key regardless of the tag. The trap:
// SUBTLE: this attaches a `T: Trait` bound you do not want
#[derive(Clone, Copy, PartialEq, Eq, Hash, Debug)]
struct Id<T> { raw: u64, _tag: PhantomData<T> }
The derive macro expands to impl<T: Hash> Hash for Id<T>, impl<T: Copy> Copy for Id<T>, and so on. So Id<T> is only Hash when T is Hash — but the tag
User holds no data and need not implement anything. The derive makes the wrong
thing the bound: it bounds the tag instead of the u64.
The ladder’s working solution made the tags derive everything too, which compiles
— but it is a coincidence. The day a tag does not implement Hash/Copy, the id
silently loses those traits. The robust pattern real crates (slotmap, ECS entity
ids) use is to hand-write the impls so the bound lands on the data, not the
tag:
// OK: no bound on T anywhere — works for ANY tag
impl<T> Clone for Id<T> { fn clone(&self) -> Self { *self } }
impl<T> Copy for Id<T> {}
impl<T> PartialEq for Id<T> { fn eq(&self, o: &Self) -> bool { self.raw == o.raw } }
impl<T> Eq for Id<T> {}
impl<T> std::hash::Hash for Id<T> {
fn hash<H: std::hash::Hasher>(&self, h: &mut H) { self.raw.hash(h); }
}
The structural “aha”: a phantom type appears in the type signature but never in
the data, so trait impls on the wrapper should be bounded by the data, not by the
phantom. That is what makes Id<T> truly zero-cost and tag-agnostic.
Explain it back
Future-you should be able to answer these cold:
- Why does
speed(time, distance)compile withf64args but not withMeters/Secondsargs? What information did the newtype restore? - Why does a fresh
UserId(u64)not support==or{:?}? What does a derive actually generate? Meters + Secondsfails to compile. Which mechanism rejects it — a forbidding rule, or a missing impl?- What two things does
impl Deref for Usernameenable, and how do they differ? - Why does
SortedVecdeliberately not implementDeref<Target = Vec>? What would break? - The orphan rule forbids
impl Display for Vec<i32>. How doesPrettyVecmake the same impl legal? - What does
#[repr(transparent)]guarantee beyond what the optimizer already does? Why is&[Wrapping64] -> &[u64]sound but&[u64] -> &[NonZeroU64]not? - In the
Emailmodule, what single language feature makes the “everyEmailis valid” guarantee airtight? - Why does
#[derive(Hash)]onId<T>produce the wrong bound, and how do the hand-written impls fix it?
See also
- Conversion traits —
From/Into/TryFrom, the in/out of a newtype’s smart constructor. - Blanket impls & coherence — the orphan rule in full, and the newtype workaround from the trait-author side.
- Custom error types —
EmailErroris a tiny error enum; the full treatment lives here. - Static vs dynamic dispatch — monomorphization is why the
phantom
Id<T>andrepr(transparent)wrappers cost nothing at runtime.
API evolution & semver
Ladder:
src/bin/semver.rs· Run:cargo run --bin semver· Phase 3 · 9 rungs
TL;DR
Semantic versioning is a promise: within a major version, an upgrade never breaks a
downstream build. A change is breaking if any valid downstream crate could stop
compiling, stop linking, or change behavior after a cargo update. Rust’s
exhaustiveness checking, type inference, and auto traits make that set of breaking
changes much larger than “I deleted a function” — adding a public field, adding an enum
variant, or even swapping a private field’s type can all break the world.
The defensive toolkit, all proven in this ladder:
- Private fields + constructors — kill the struct literal, so fields can grow freely.
#[non_exhaustive]— keep public fields readable while forbidding literals and exhaustive matches downstream.- Default trait methods and sealed traits — evolve a trait without breaking implementors (or forbid foreign implementors entirely).
- Compile-time auto-trait guards (
const _+assert_send) — catch a silently-droppedSend/Syncin CI instead of in a user’s bug report. - Minimal generic bounds — every bound is a wall some future caller hits.
Why this exists (from first principles)
A version number is a compatibility contract with people you’ll never meet. MAJOR.MINOR.PATCH:
| Bump | Meaning | Promise to downstream |
|---|---|---|
| PATCH | backwards-compatible bug fix | recompiles, behaves the same |
| MINOR | backwards-compatible addition | recompiles, new API available |
| MAJOR | breaking change | may fail to compile/link |
The hard part isn’t the numbering — it’s knowing which bucket a change falls into. In a dynamically-typed language “breaking” mostly means “removed something.” In Rust the compiler lets downstream code lean on your types in ways you never anticipated:
- it can construct your struct with a literal
Foo { a, b }, which silently requires every field; - it can exhaustively match your enum with no
_arm, which silently requires every variant; - it can send your type across threads, which silently requires every field be
Send.
Each of those “silently requires” is a constraint you didn’t write down but are now on the hook for. SemVer in Rust is the discipline of seeing those implicit constraints and deciding, per change, whether you just violated one.
The authoritative rules live in the Cargo SemVer reference; this ladder makes you feel each one with the compiler rather than memorize a table.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | Foundations | required_bump | The canonical, unambiguous cases: bugfix→patch, add fn→minor, remove/rename→major |
| 2 | Foundations | struct-literal landmine | An all-pub struct can’t grow a field; private fields + constructor defuse it |
| 3 | Mechanics | #[non_exhaustive] structs | Keep fields readable, forbid downstream literals, add fields later |
| 4 | Mechanics | enum variants | Adding a variant breaks exhaustive match; #[non_exhaustive] forces a _ arm |
| 5 | Footgun | trait evolution | Required method = break (E0046); defaulted method / non-blanket impl = safe |
| 6 | Footgun | sealed traits | A private Sealed supertrait means no foreign impls — so the trait is free to evolve |
| 7 | Real-world | auto-trait leakage | A private field type change can silently drop Send/Sync; guard it at compile time |
| 8 | Real-world | generic bounds | Loosen = safe, tighten = break; new type params need defaults |
| 9 | Capstone | classify engine + future-proofed mod lib | One classifier for all rules; one library whose v1.1 is a clean minor |
The ideas, built up
1. The baseline rules
Start with the cases nobody argues about. The ladder encodes them as a match:
fn required_bump(change: &Change) -> Bump {
match change {
Change::BugFixInternal => Bump::Patch, // no public surface moved
Change::PerfImprovement => Bump::Patch, // same signature, same result
Change::AddPublicFunction => Bump::Minor, // pure addition
Change::DeprecatePublicFunction => Bump::Minor, // #[deprecated] still compiles
Change::RemovePublicFunction => Bump::Major, // downstream calls vanish
Change::RenamePublicFunction => Bump::Major, // = remove + add
}
}
Two things worth internalizing here. Deprecation is minor, not breaking: #[deprecated]
emits a warning, and warnings don’t fail a build. A rename is a remove plus an add — the
“add” half is harmless, but the “remove” half is what bumps it to major. Renames are the
classic accidental break.
2. The struct-literal landmine
Here’s the first change that looks harmless and isn’t. Ship this:
pub struct RgbColor { pub r: u8, pub g: u8, pub b: u8 }
Downstream is now free to write a struct literal and an exhaustive destructure:
let c = RgbColor { r: 255, g: 0, b: 0 };
let RgbColor { r, g, b } = c;
Add pub a: u8 in v1.1 and both of those break:
- the literal fails with E0063 (missing field
a); - the destructure fails with E0027 (pattern does not mention field
a).
A one-field “feature” just forced a major bump. The root cause: a struct literal implicitly requires all fields, and you can’t add a field without invalidating every existing literal.
The fix is to remove the capability that creates the obligation — make the fields private and hand out a constructor and accessors:
pub struct RgbColor { r: u8, g: u8, b: u8 } // private
impl RgbColor {
pub fn rgb(r: u8, g: u8, b: u8) -> Self { Self { r, g, b, a: 255 } }
pub fn channels(&self) -> (u8, u8, u8) { (self.r, self.g, self.b) }
}
With no public fields, downstream cannot write a literal or an exhaustive destructure, so
adding a later changes nothing for them. (In the ladder, the proof is literal: the file
adds the a field and check_2 keeps compiling untouched.)
3. #[non_exhaustive] — readable fields without the obligation
Private fields cost ergonomics: every read becomes a getter call. #[non_exhaustive] is
the middle path. It lets downstream read public fields directly but forbids the two
fragile operations:
#[non_exhaustive]
pub struct ClientConfig {
pub timeout_ms: u32,
pub retries: u8,
}
impl ClientConfig {
pub fn new() -> Self { ClientConfig { timeout_ms: 30_000, retries: 3 } }
pub fn with_retries(mut self, retries: u8) -> Self { self.retries = retries; self }
}
From another crate:
let cfg = ClientConfig::new().with_retries(5); // OK: constructor is the only door
let t = cfg.timeout_ms; // OK: reading a pub field is fine
let bad = ClientConfig { timeout_ms: 1, retries: 1 }; // ERROR: literal forbidden
Subtlety that the single-file ladder can’t show you directly:
#[non_exhaustive]only restricts foreign crates. Inside the defining crate the attribute is inert — you can still literal-construct and exhaustively match. That’s whyClientConfig::newcan use a struct literal: it lives in the same crate. The restriction (and the safety) is purely a cross-crate property.
4. Enum variants, and the one place the compiler does show you the pain
Adding a variant to a plain public enum breaks every downstream exhaustive match with
E0004 (non-exhaustive patterns). So “add a variant” defaults to major. Mark the
enum #[non_exhaustive] from day one and downstream is forced to include a _ arm — so
later variants are only minor.
The neat trick in this rung: you can feel the real cross-crate error inside a single bin,
because std::io::ErrorKind is itself #[non_exhaustive], and std is a foreign
crate to you. Write a match over it without a _ and the compiler rejects it:
fn describe_io_error(kind: ErrorKind) -> &'static str {
match kind {
ErrorKind::NotFound => "missing",
ErrorKind::PermissionDenied => "denied",
_ => "other", // mandatory — that IS the point
}
}
Drop the _ arm and you get the exact experience your downstream has when you add a
variant. The rule, as code:
fn add_variant_bump(enum_was_non_exhaustive: bool) -> Bump {
if enum_was_non_exhaustive { Bump::Minor } else { Bump::Major }
}
5. Trait evolution: the default body is everything
A public trait is a contract with everyone who impls it. Two opposite moves, opposite costs:
pub trait Plugin {
fn name(&self) -> &str;
// Adding `fn version(&self) -> u32;` (no body) -> BREAKING: every downstream
// `impl Plugin for X` fails with E0046, "not all trait items implemented".
// Adding `fn version(&self) -> u32 { 1 }` (default) -> MINOR: existing impls
// inherit the body and never knew it appeared.
fn version(&self) -> u32 { 1 }
}
That single { 1 } is the entire difference between a quiet minor release and breaking
every implementor in the ecosystem. The impl-side rules round it out:
| Change | Bump | Why |
|---|---|---|
| Add required method (no default) | Major | every foreign impl is now incomplete (E0046) |
| Add defaulted method | Minor | impls inherit the default |
Add non-blanket impl Trait for Concrete | Minor | can perturb inference, but treated as minor |
Add blanket impl<T> Trait for T | Major | can collide with downstream impls (coherence, E0119) |
6. Sealed traits: make “add a method” a minor change
Rung 5 was grim: adding a required method breaks every implementor. But what if no foreign implementor can exist? Then there’s nothing to break, and you can add methods, change defaults, even add supertraits — all as minor changes. That’s a sealed trait.
The pattern is a marker trait in a private module, required as a supertrait:
mod sealed {
pub trait Sealed {} // module is private to the crate
}
pub trait Format: sealed::Sealed { // public, but gated by a private bound
fn extension(&self) -> &str;
}
pub struct Json;
impl sealed::Sealed for Json {} // only this crate can write this line
impl Format for Json { fn extension(&self) -> &str { "json" } }
A foreign crate that tries impl Format for TheirType gets E0277: TheirType: Sealed
is not satisfied — and they can’t fix it, because they can’t reach into your private
sealed module. The visibility dance is the crux: Sealed must be pub (so it can appear
in the public Format bound) yet live in a private mod sealed (so it’s unnameable
outside the crate). This is how serde, bytes, and several std traits stay evolvable.
7. Auto-trait leakage: the break with no signature change
The sneakiest one. Send and Sync are auto traits: the compiler derives them
structurally from a type’s fields. So a type’s thread-safety is a function of its private
internals — and changing a private field can flip it without the public signature moving
one character.
pub struct Job { data: Vec<u8> } // Vec<u8> is Send + Sync -> Job is too
// later, "just an internal refactor":
pub struct Job { data: std::rc::Rc<u8> } // Rc is !Send + !Sync -> Job is now neither
Every downstream thread::spawn(move || ... job ...) now fails with E0277 (“Rc<u8>
cannot be sent between threads safely”). You shipped it as a patch; it was a major break.
(The same hazard hides behind -> impl Trait returns: the opaque type leaks the auto
traits of whatever you built it from.)
The professional defense is a compile-time regression guard — the trick the
static_assertions crate automates:
fn assert_send<T: Send>() {} // the bound IS the test
fn assert_sync<T: Sync>() {}
const _: () = {
let _guard: fn() = || {
assert_send::<Job>();
assert_sync::<Job>();
};
};
If Job ever loses Send or Sync, this block fails to compile — the leak becomes a
build error in your CI instead of a bug report from a user. const _ runs the
type-check at compile time with zero runtime cost.
8. Generic bounds: the loosen/tighten asymmetry
Bounds have a direction:
- Loosening a bound (removing a requirement) is non-breaking. Anyone who satisfied the stricter bound still satisfies the looser one — you only let more callers in.
- Tightening a bound (adding a requirement) is breaking. Callers whose type doesn’t satisfy the new requirement are locked out with E0277.
The practical rule that falls out: ask for the minimum your body actually needs. Every
extra bound is a wall some future caller will hit. The ladder demonstrates with a type that
is Debug but not Clone:
#[derive(Debug)] struct NoClone;
fn process<T: std::fmt::Debug>(items: &[T]) -> usize { // minimal bound
let _ = items.iter().map(|x| format!("{x:?}")).collect::<Vec<_>>();
items.len()
}
process(&[NoClone, NoClone]); // OK
Add + Clone to the bound and that call dies — an over-tight bound is the break. For new
type parameters: adding one without a default changes arity and breaks turbofish/some
calls (major); adding one with a default keeps existing uses working (minor).
Footguns
| Trap | What bites | Fix |
|---|---|---|
All-pub struct | Adding a field breaks every literal (E0063) and exhaustive destructure (E0027) | Private fields + constructor, or #[non_exhaustive] |
| Plain public enum | Adding a variant breaks exhaustive match (E0004) | #[non_exhaustive] from day one |
| Required trait method | Adding one breaks every implementor (E0046) | Give it a default body, or seal the trait |
| Blanket impl | Adding impl<T> Trait for T collides with downstream impls (E0119) | Treat as major; prefer non-blanket impls |
| Auto-trait leakage | A private field type change silently drops Send/Sync (E0277 downstream) | const _ + assert_send/assert_sync guard |
| Over-tight bound | A needless + Clone locks out valid callers | Bound only what the body uses |
| In-crate blind spot | #[non_exhaustive] and sealing don’t restrict the defining crate, so local tests won’t reveal the protection | Reason about it cross-crate; test against a real downstream crate if it matters |
Real-world patterns
#[non_exhaustive]everywhere in std and the ecosystem.std::io::ErrorKind, most error enums inthiserror-based libraries, and config structs use it so they can grow without a major bump.- Sealed traits in
serde(serde::de/serinternals),bytes::Buf/BufMut, andnom— the public trait is callable but not implementable, so the maintainers can add methods freely. static_assertions::assert_impl_all!(Job: Send, Sync)is the packaged version of the rung-7const _guard;tokioandbytesship these to lock auto traits in place.cargo semver-checksautomates much ofclassify— it diffs your public API against the last published version and tells you the required bump. Knowing the rules by hand is how you read its output.
Capstone insight
Two halves, and the point is how they fit.
Part A — the brain. classify(&ApiChange) -> Bump collapses all eight rungs into one
match with guards on boolean fields. The shape of the data model is the lesson: the
breaking-ness of a change is rarely about the change alone — it’s about a condition.
“Add a struct field” isn’t major or minor; it’s major unless the struct was already
sealed from literals. “Change internals” is a patch unless it drops an auto trait.
ApiChange::AddStructField { sealed_from_literals: true } => Bump::Minor,
ApiChange::AddStructField { sealed_from_literals: false } => Bump::Major,
ApiChange::ChangeInternals { keeps_auto_traits: true } => Bump::Patch,
ApiChange::ChangeInternals { keeps_auto_traits: false } => Bump::Major,
Part B — the hands. A mod lib engineered so its v1.1 is a clean minor, by combining
the techniques: a #[non_exhaustive] Settings built through Settings::new(), and a
sealed Codec trait. The downstream consumer is the proof:
fn use_library() -> String {
let settings = lib::Settings::new();
let codec = lib::Gzip;
format!("{} @ level {}", codec.name(), settings.level)
}
The “aha”: because Settings is non-exhaustive (constructed via new) and Codec is
sealed, a v1.1 that adds a field to Settings and adds a defaulted method to
Codec requires zero changes to use_library. Future-proofing isn’t one trick — it’s
choosing, up front, the construction and extension points that keep your future options
open. You decide where downstream is allowed to couple to you, and you make everywhere else
unreachable.
Explain it back
- Why is adding a
pubfield to an existing public struct a major change, and what two distinct downstream operations does it break? - What exactly does
#[non_exhaustive]forbid downstream, and why does it have no effect inside the defining crate? - You add a method to a public trait. When is that a minor change and when is it major?
- How can a sealed trait let you add a required method as a minor release?
- Your “internal refactor” swaps a
Vecfield for anRc. The signature is identical. How can this break a downstreamcargo update, and how would you have caught it in CI? - Loosening vs tightening a generic bound — which direction is safe, and why?
See also
- The typestate pattern — same
Sealed-supertrait trick, used to gate state transitions. - Blanket impls & coherence — why a blanket impl is a major change (E0119) and the sealed-extension-trait pattern.
Send&Syncdeeply — the auto-trait mechanics behind rung 7’s leak.- Newtype & zero-cost wrappers — private-field smart constructors (parse-don’t-validate) as an API-stability tool.
- Generic bounds & where clauses — the bound mechanics rung 8 builds on.
Collections deep-dive
Ladder:
src/bin/collections.rs· Run:cargo run --bin collections· Phase 3 · 9 rungs
TL;DR
Every std collection is the same idea — store many values — bent around a different tradeoff between ordering, lookup cost, and what the key has to prove:
| Collection | Backing | Lookup | Order | Key needs |
|---|---|---|---|---|
HashMap<K,V> | hash table | O(1) avg | none (random) | Hash + Eq |
BTreeMap<K,V> | B-tree | O(log n) | sorted, supports range | Ord |
HashSet<T> | HashMap<T,()> | O(1) avg | none | Hash + Eq |
BTreeSet<T> | BTreeMap<T,()> | O(log n) | sorted | Ord |
VecDeque<T> | ring buffer | O(1) at both ends | insertion | — |
HashMap is the default. Reach for BTreeMap when you need order or range
queries, VecDeque when you push/pop at both ends, and swap the hasher (not
the map) when SipHash’s DoS-resistance isn’t worth its cost. The single most
important technique is the Entry API, which collapses the check-then-act
double lookup into one probe.
Why this exists (from first principles)
You have a pile of values and you want to find one again. The naive answer —
a Vec you scan linearly — is O(n) per lookup. Collections buy you sub-linear
lookup, but nothing is free, so each one makes you pay in a different currency:
- A hash table turns the key into an array index via a hash function. Lookup
is O(1) on average — but the price is that the keys land in hash order, which
is no order at all to a human. And it only works if the key can be hashed
and compared consistently (
Hash + Eq). - A B-tree keeps keys sorted in shallow, cache-friendly nodes. Lookup is
O(log n) — slower than a hash, but now iteration is ordered and you can ask
“give me every key between X and Y,” which a hash table structurally cannot
answer because its keys have no neighbors. The price is the key must be
orderable (
Ord). - A ring buffer (
VecDeque) gives up associative lookup entirely but makes both ends O(1), which a plainVeccan’t (front insert/remove is O(n) because everything shifts).
The compiler enforces the key requirements through trait bounds: you literally
cannot use a type as a HashMap key until it implements Hash + Eq. That’s not
bureaucracy — it’s the table refusing to operate without the one guarantee that
makes it correct.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | foundations | HashMap basics | get returns Option<&V>; absent ⇒ None, not panic |
| 2 | foundations | BTreeMap & ordering | sorted iteration is free; range(lo..=hi) queries |
| 3 | mechanics | the Entry API | one lookup instead of check-then-insert |
| 4 | mechanics | HashSet & set algebra | dedup, union/intersection/difference |
| 5 | mechanics | VecDeque | O(1) both ends; sliding window + BFS frontier |
| 6 | footgun | Borrow lookup + key hazard | get("k") with no alloc; never mutate a key’s hash |
| 7 | footgun | custom Hash/Eq | break k==k' ⇒ hash==hash' and silently lose entries |
| 8 | real-world | choosing one + hashers | decision matrix; swap RandomState for FNV-1a |
| 9 | capstone | MyHashMap from scratch | open addressing: linear probing + tombstones + resize |
The ideas, built up
1. HashMap: get borrows, and absence is a value
fn word_count(text: &str) -> HashMap<&str, usize> {
let mut map = HashMap::new();
for word in text.split_whitespace() {
*map.entry(word).or_insert(0) += 1;
}
map
}
Two things to internalize from the very first rung:
getreturnsOption<&V>, a borrow.wc.get("the")isSome(&3), notSome(3). The map still owns the value; you get a reference into it. And a missing key isNone— absence is an ordinary return value, never a panic. (Indexing withmap[k]does panic on a missing key;getis the safe form.)- The
&strkeys borrow fromtext— noStringis allocated. The lifetime inHashMap<&str, usize>ties the map to the source string.
split_whitespace() (not split(' ')) collapses runs of spaces and handles
tabs/newlines, which is almost always what you want.
2. BTreeMap: order is the feature, range is the payoff
A HashMap iterates in effectively random order. A BTreeMap is always
ascending — you don’t sort anything, the tree is the sort:
fn sorted_word_count(text: &str) -> Vec<(&str, usize)> {
let mut map = BTreeMap::new();
// ... tally ...
map.into_iter().collect() // already ascending by key
}
The capability a hash map cannot match is the range query:
fn score_range<'a>(scores: &BTreeMap<u32, &'a str>, lo: u32, hi: u32) -> Vec<&'a str> {
scores.range(lo..=hi).map(|(_, name)| *name).collect()
}
range(lo..=hi) is two binary-search descents to find the window endpoints, then
an in-order walk — O(log n + k) for k results. A HashMap has no concept of
“the next key,” so it can only answer point lookups.
3. The Entry API: stop looking things up twice
The naive “increment a counter” needs two or three hash lookups:
// WRONG (double lookup): hashes `k` twice
if map.contains_key(k) {
*map.get_mut(k).unwrap() += 1;
} else {
map.insert(k, 1);
}
entry() hashes the key once, returns a handle to that slot (Occupied or
Vacant), and lets you branch on it:
// OK (one lookup)
*map.entry(k).or_insert(0) += 1;
Two idioms the ladder drills:
// Group into Vecs: build the empty Vec ONLY when the key is new.
map.entry(word.len()).or_insert_with(Vec::new).push(word);
// Modify-or-insert: floor on first sight, +1 on every later sight.
map.entry(w).and_modify(|c| *c += 1).or_insert(floor);
or_insert_with(Vec::new)vsor_insert(Vec::new()): the_withclosure runs only on a vacant slot. Plainor_inserteagerly constructs its argument on every call, even when the key already exists — a wasted allocation each time. (or_default()is the same idea forDefaulttypes.)and_modify(...).or_insert(...)reads as “if occupied, run the modify; otherwise insert.” The two arms are mutually exclusive by construction —and_modifyreturns theEntryback soor_insertcan finalize it. That’s why a value seen three times withfloor = 10lands on12(insert 10, then +1, +1), never double-counted.
4. HashSet: a HashMap<T, ()> that speaks membership
// insert returns bool: true if the value was NEW. One probe, doubles as a
// "have I seen this?" test.
for &item in items {
if seen.insert(item) { out.push(item); } // dedup, preserving first-seen order
}
A HashSet destroys order, so “dedup but keep order” combines a HashSet
(the O(1) seen-test) with a Vec (the ordered output). The set algebra returns
lazy iterators of &T, unordered:
let inter: Vec<i32> = a.intersection(&b).copied().collect(); // in both
let only_a: Vec<i32> = a.difference(&b).copied().collect(); // a \ b
let union_size = a.union(&b).count(); // |a ∪ b|, deduped
union(&b).count() already counts each element once, so you never need
a.len() + b.len() - inter.len().
5. VecDeque: a ring buffer, the engine of BFS
A Vec is O(1) at the back but O(n) at the front — remove(0) shifts every
other element. A VecDeque keeps head and tail indices into a circular array, so
push_back/pop_front/push_front/pop_back are all O(1) amortized.
// Bounded sliding window: push, evict the oldest off the FRONT, record the max.
win.push_back(v);
if win.len() > cap { win.pop_front(); }
out.push(*win.iter().max().unwrap());
// BFS: VecDeque frontier + HashSet visited. Mark visited ON ENQUEUE.
visited.insert(0);
queue.push_back(0);
while let Some(node) = queue.pop_front() {
order.push(node);
for &n in &adj[node] {
if visited.insert(n) { queue.push_back(n); } // insert==true ⇒ newly seen
}
}
The subtlety: mark a node visited when you enqueue it, not when you dequeue.
A node reachable from two parents would otherwise get enqueued twice. The
insert-returns-bool trick gates the push_back in a single probe.
Footguns
The Borrow lookup (a feature that looks like magic)
Why does map.get("foo") work on a HashMap<String, V> without building a
String? Because get is generic over anything the key can be borrowed as:
fn get<Q>(&self, k: &Q) -> Option<&V>
where K: Borrow<Q>, Q: Hash + Eq + ?Sized
String: Borrow<str>, and the Borrow contract guarantees "foo" hashes and
compares identically whether it’s a str or a String. So you probe with a
borrowed view and allocate nothing:
// OK: &str query against a HashMap<String, _>, zero allocation
if let Some(v) = map.get(query) { total += v; }
// WRONG: needless allocation per query
if let Some(v) = map.get(&query.to_string()) { total += v; }
Never mutate a key’s hash while it’s in the map
A HashMap files each key into a bucket by hash(key) at insertion time. If
the key’s hash later changes, the entry is stranded in the wrong bucket: lookups
by the new value probe a different bucket and find nothing. The entry is leaked
in place — still consuming memory, permanently unreachable.
Rust normally makes this impossible: keys are owned and never lent out as &mut.
But interior mutability is the escape hatch. The ladder builds a BadKey
wrapping a Cell<u64> that hashes on its inner value:
let mut map = HashMap::new();
map.insert(BadKey::new(1), "value"); // filed under hash(1)
map.keys().next().unwrap().inner.set(999); // mutate the map's OWN key via Cell
map.get(&BadKey::new(999)).is_some() // false — probes hash(999) bucket, empty
keys() hands out a shared &BadKey, and Cell::set mutates through a shared
reference — so you corrupt the real stored key while it sits in its bucket. This
is exactly why Cell/RefCell keys are a latent bug.
Break k == k' ⇒ hash(k) == hash(k') and you silently lose data
This is the one law every map key must obey: equal keys must hash equal.
(The converse isn’t required — unequal keys may collide.) #[derive(Hash, PartialEq, Eq)] can never break it because it threads the same fields through
both. Hand-write them and you can desync — with no error and no panic, just
vanishing entries.
// GoodKey: case-insensitive, law UPHELD (both fold case)
impl Hash for GoodKey {
fn hash<H: Hasher>(&self, s: &mut H) { self.0.to_lowercase().hash(s); }
}
impl PartialEq for GoodKey {
fn eq(&self, o: &Self) -> bool { self.0.to_lowercase() == o.0.to_lowercase() }
}
// BrokenKey: same eq (case-insensitive), but hash reads the RAW bytes
impl Hash for BrokenKey {
fn hash<H: Hasher>(&self, s: &mut H) { self.0.hash(s); } // <-- the bug
}
With BrokenKey, "Foo" == "foo" is true but they hash differently. Insert
"foo", look up "FOO" → the probe lands in the hash("FOO") bucket, which
doesn’t hold the "foo" entry → miss, even though the keys are “equal.”
The discipline: every field
eqlooks at,hashmust look at too. Only hand-write these for a custom notion of equality (case-folding, normalization, a subset of fields); otherwise derive and stay safe.
Real-world patterns
Choosing one (the decision matrix)
push/pop at both ends? -> VecDeque
need sorted iter / range? -> BTreeMap (k→v) or BTreeSet (membership)
key → value lookup? -> HashMap
just membership? -> HashSet
HashMap is the default; everything else is a deliberate upgrade for a property
you actually need. Note the priority order matters — a deque workload wins
regardless of the other flags.
Custom hashers (swap the hasher, not the map)
std’s HashMap defaults to RandomState (SipHash 1-3) seeded with a
per-process random key. That’s deliberate DoS protection: an attacker can’t
precompute keys that all collide into one bucket and turn your O(1) map into an
O(n) linked list. The cost is speed on small keys and nondeterministic
iteration order across runs.
For internal, non-adversarial maps over small keys, crates like fxhash/ahash
swap in a faster non-cryptographic hasher. The ladder builds the minimal one —
FNV-1a:
struct FnvHasher { state: u64 }
impl Default for FnvHasher {
fn default() -> Self { Self { state: 0xcbf2_9ce4_8422_2325 } } // offset basis
}
impl Hasher for FnvHasher {
fn finish(&self) -> u64 { self.state }
fn write(&mut self, bytes: &[u8]) {
for &b in bytes {
self.state ^= b as u64; // xor first ("1a")
self.state = self.state.wrapping_mul(0x100000001b3); // then multiply
}
}
}
type FnvMap<K, V> = HashMap<K, V, BuildHasherDefault<FnvHasher>>;
- Seeding
stateinDefault(rather than lazily insidewrite) is the form real hasher crates ship: it removes a per-writebranch and correctly handles a key that callswritemultiple times (a struct hashing field by field), since the FNV chain flows continuously across calls. BuildHasheris the factory: it makes a freshHasherper key.BuildHasherDefault<H>is the zero-config version that just callsH::default(). The result is a deterministic map — same key, same hash, every run.Hasherhas default impls forwrite_u32,write_u64, etc., all funneling intowrite(&[u8])— so implementingwrite+finishis enough.
Capstone insight
MyHashMap<K, V> is a working hash map built on open addressing — one flat
Vec<Slot<K,V>> where Slot is Empty | Deleted | Occupied(K, V). (std’s table
is this idea plus SIMD probing; this is the readable textbook version.)
Linear probing. To place a key: home = hash(k) % capacity, then walk
forward with wraparound ((i + 1) % cap) until you hit the first Empty (key
absent) or a matching Occupied (overwrite). Lookups probe the same path and
stop at Empty — because if the key were present, insert would have filled that
empty slot before reaching it.
Tombstones. On remove you cannot just set the slot Empty — that would cut
the probe chain and hide keys inserted after it on the same chain. Instead leave
a Deleted marker. Lookups skip past tombstones; inserts may reuse them.
The hardest correctness point: when inserting, remember the first tombstone you pass, but keep probing. Only commit to reusing it once you reach
Empty(proving the key is genuinely absent). Reusing it eagerly would create a duplicate key if that key already lives further down the chain.
Resize / rehash. When (len + tombstones) * 4 >= capacity * 3 (load factor
0.75), allocate a table of double capacity and re-insert every Occupied entry —
which also drops all tombstones for free. The crucial realization: home depends
on self.slots.len(), so after a resize the same key maps to a different
bucket. Resize must rehash, not memcpy. The recursion (resize calls insert,
insert calls resize) is bounded: after doubling, live entries sit well under the
0.75 threshold of the new table, so the re-inserts never re-trigger a resize.
Two Rust mechanics carry the whole thing:
// overwrite: move the old V out from behind &mut, drop the new one in
Some(std::mem::replace(v, value))
// remove: swap the whole slot for a tombstone, extract the old value
let old = std::mem::replace(&mut self.slots[i], Slot::Deleted);
mem::replace is the canonical “move a value out from behind a &mut” tool —
you can’t move out of a Vec element otherwise.
Explain it back
- Why does
HashMap::getreturnOption<&V>instead ofOption<V>, and when doesmap[k]panic wheremap.get(k)wouldn’t? - What can a
BTreeMapdo that aHashMapstructurally cannot, and what does the key pay for it (which trait bound)? - Write the
Entry-API one-liner for “increment a counter,” and explain whyor_insert_with(Vec::new)beatsor_insert(Vec::new()). - Why does
map.get("foo")compile and allocate nothing on aHashMap<String, V>? Name the trait and the bound onget. - State the
Hash/Eqlaw in one line. Describe a concrete way to break it and exactly what symptom the user sees. - Why is mutating a key through a
Cellafter insertion a bug, and why does Rust normally prevent it? - Why does std default to SipHash with a random seed, and when would you swap it out? What do you give up?
- In an open-addressing map, why can’t
removeset a slot toEmpty? What goes wrong, and what’s the fix? - During insert, why must you keep probing past the first tombstone before reusing it?
See also
Borrow/ToOwned— the trait behind no-allocget("k")lookups.Cell&RefCell— the interior mutability that powers the key-mutation footgun.Drop& Ordering —mem::replace/mem::takefor moving values out from behind&mut.- Newtype & zero-cost wrappers — wrapping a key type to give it a custom
Hash/Eq.
Strings & text
Ladder:
src/bin/strings_text.rs· Run:cargo run --bin strings_text· Phase 3 · 9 rungs
TL;DR
Rust’s string zoo is two independent axes multiplied together:
- Owned vs borrowed.
String/OsString/PathBuf/CStringown a heap buffer.&str/&OsStr/&Path/&CStrare views — unsized, always behind a reference. - What the bytes promise.
str= valid UTF-8.OsStr= whatever the OS uses (UTF-8 not guaranteed).CStr= NUL-terminated, no interior NUL.[u8]= no promises at all.
The one sentence that unlocks everything: String is Vec<u8> + the UTF-8
invariant; &str is &[u8] + that same invariant. Every footgun in this topic
is about respecting that invariant, and every conversion method is a gatekeeper for
crossing it.
Why this exists (from first principles)
Why not have one string type? Because “text” means different things at different boundaries, and each boundary enforces a different guarantee:
- Your program logic wants valid Unicode so iteration, comparison, and display
behave. That is
str. - The operating system predates Unicode. A Unix filename is any byte sequence
except
NULand/; a Windows path is UTF-16 that may contain unpaired surrogates. Neither is guaranteed to be valid UTF-8, so forcing it intostrwould either lose data or panic. That isOsStr. - C has no length field — a string is “bytes up to the first
NUL”. To hand a string to C you must guarantee a terminator and no interiorNUL. That isCStr/CString. - Filesystem paths are
OsStrplus structure (separators, components, extensions). That isPath.
If these were all one type, the compiler couldn’t stop you from, say, passing a
non-UTF-8 filename where UTF-8 is required, or building a C string with an interior
NUL that silently truncates. Separate types turn those bugs into compile errors
or explicit Results at the conversion point.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | foundations | str vs String | owned heap buffer vs borrowed view; take &str to accept both |
| 2 | foundations | UTF-8 invariant | len() is bytes; bytes / chars / char_indices |
| 3 | footgun | slicing bites | &s[a..b] panics mid-codepoint; get / is_char_boundary |
| 4 | mechanics | zero-copy parsing | lines / split_once / trim / parse return borrowed slices |
| 5 | real-world | OsStr / OsString | OS text isn’t UTF-8; to_str → Option, to_string_lossy → Cow |
| 6 | real-world | Path / PathBuf | structured paths over string concat |
| 7 | real-world | CStr / CString | NUL-terminated FFI; interior-NUL is an error |
| 8 | real-world | conversions & validation | String ↔ Vec<u8>; from_utf8 Result vs lossy Cow |
| 9 | capstone | hand-rolled UTF-8 decoder | reimplement str::chars() from raw bytes |
The ideas, built up
1. str is a view; String owns the buffer
String owns a growable heap allocation. &str is a borrowed window into UTF-8
bytes — a string literal lives in the binary’s read-only data, and slicing a
String produces a &str pointing into its buffer. &str is unsized, so you
only ever hold it behind a reference.
The idiomatic consequence: take &str as a parameter, not &String. Deref
coercion turns &String into &str automatically, so one signature accepts both an
owned string and a literal, with no clone:
fn shout(s: &str) -> String {
format!("{}!", s.to_uppercase())
}
let owned = String::from("hello");
shout(&owned); // &String coerces to &str
shout("world"); // literal &str
// `owned` is still usable here — we only borrowed it.
Note to_uppercase() returns a new String rather than mutating in place: case
mapping can change the byte length (e.g. ß → SS), so it cannot be done within
the original buffer.
2. The UTF-8 invariant: bytes are not characters
UTF-8 encodes one char (a Unicode scalar value) in 1 to 4 bytes. This single
fact is the source of every surprise:
fn analyze(s: &str) -> StrStats {
StrStats {
byte_len: s.len(), // BYTES, not chars
char_count: s.chars().count(), // decoded scalars
last_char_offset: s.char_indices().last().map(|(i, _)| i),
}
}
s.len()is the number of bytes."café".len()is5, not4(theéis two bytes,0xC3 0xA9)."日本語".len()is9(3 bytes each).s.bytes()yields the rawu8encoding.s.chars()yields decodedchars.s.char_indices()yields(byte_offset, char)— the byte where each char starts. Offsets jump by 1–4, not always by 1.
char_indices().last().map(|(i, _)| i) gives the byte offset of the final char and
returns None for an empty string for free — .last() on an empty iterator is
None.
3. Slicing bites: byte ranges must hit char boundaries
There is no s[i] char indexing in Rust. You slice by a byte range
&s[a..b] — and the endpoints must fall on UTF-8 char boundaries, or it panics
at runtime: &"café"[0..4] splits the é and dies with “byte index 4 is not a
char boundary”.
Two tools make slicing safe:
// OK: non-panicking slice — returns None for out-of-bounds OR mid-codepoint.
fn safe_slice(s: &str, a: usize, b: usize) -> Option<&str> {
s.get(a..b)
}
// OK: drop the first char, multibyte-aware.
fn behead(s: &str) -> &str {
match s.chars().next() {
Some(first) => &s[first.len_utf8()..],
None => "",
}
}
s.get(a..b) is the fallible twin of &s[a..b]: same checks, but None instead of
a panic. s.is_char_boundary(i) answers the boundary question directly.
behead shows the subtle point: &s[1..] would panic on "日本" (first char is 3
bytes), but &s[first.len_utf8()..] is safe even though it’s a raw slice —
len_utf8() is exactly the byte width of that first char, so the start index is
guaranteed to land on the next char’s boundary. You can use the panicking slice when
you can prove the index is a boundary.
4. Zero-copy parsing: split/trim/parse return borrowed slices
The everyday string toolkit — lines, trim, split_once, strip_prefix — all
return &str slices that borrow the original buffer. No allocation, no copy.
Only parse() produces an owned value.
fn parse_config(input: &str) -> Vec<(&str, i64)> {
input
.lines()
.filter(|line| !line.is_empty() && !line.trim().starts_with('#'))
.filter_map(|line| {
line.split_once('=').map(|(key, value)| {
(key.trim(), value.trim().parse::<i64>().unwrap())
})
})
.collect()
}
The returned &str keys point into input — the elided lifetime ties the output
to the input. You can prove it: key.as_ptr() lands inside input’s buffer range.
split_once('=') splits on the first match and returns Option<(&str, &str)>, so a
malformed line (no =) becomes None and filter_map drops it automatically.
5. OsStr / OsString: the OS doesn’t promise UTF-8
std::env::args_os(), Path::file_name(), environment variables — these hand you
OsStr / OsString, not str, because the OS may give you bytes that aren’t
valid UTF-8. Crossing OsStr → str can fail, and the API forces you to choose how
to handle that:
fn describe_os(os: &OsStr) -> String {
match os.to_str() { // -> Option<&str>: None if not UTF-8
Some(s) => format!("utf8: {s}"),
None => format!("lossy: {}", os.to_string_lossy()),
}
}
fn is_cow_borrowed(os: &OsStr) -> bool {
matches!(os.to_string_lossy(), Cow::Borrowed(_))
}
to_str()→Option<&str>:Nonewhen the bytes aren’t valid UTF-8.to_string_lossy()→Cow<str>: never fails — replaces bad bytes with the replacement charU+FFFD(�). It returnsCow::Borrowedwhen the input was already valid UTF-8 (zero-copy) andCow::Ownedonly when it had to allocate to substitute. ThatBorrowed-vs-Owneddistinction is a free “did anything go wrong” signal.
The lesson: there is no infallible
OsStr→str. The type system makes you pick a strategy (fail withto_str, or substitute withto_string_lossy).
The ladder forges an invalid OsStr on Unix via OsStrExt::from_bytes(&[b'a', 0xFF, b'b']) to exercise the failure path — 0xFF can never begin a UTF-8
sequence.
6. Path / PathBuf: structure, not string concatenation
Path / PathBuf wrap an OsStr (so they inherit “maybe not UTF-8”) and add
filesystem semantics. The senior rule: never build paths with
format!("{dir}/{file}") — it breaks on Windows, doubles separators, and mishandles
edge cases. Use the structured API:
fn swap_extension(path: &Path, new_ext: &str) -> PathBuf {
let mut p = path.to_path_buf();
p.set_extension(new_ext); // replaces, or adds if none
p
}
fn is_hidden(path: &Path) -> bool {
path.file_name() // Option<&OsStr> (None for "..")
.and_then(|os| os.to_str()) // Option<&str> (None if not UTF-8)
.map_or(false, |s| s.starts_with('.'))
}
Key methods: Path::new, join (handles separators), file_name, extension,
file_stem, parent, components. The structured accessors return &OsStr, which
compares directly against &str literals (p.extension().unwrap() == "txt").
is_hidden is a clean two-layer Option chain: a path ending in .. has no
file_name(), and a non-UTF-8 name fails to_str() — both short-circuit to
false.
7. CStr / CString: the FFI string
C strings are NUL-terminated with no interior NUL — the string is “everything up
to the first \0”. Rust’s str / String store a length instead and carry no
terminator, so you must convert at the C boundary.
fn to_c(s: &str) -> Result<CString, NulError> {
CString::new(s) // Err if `s` contains an interior NUL
}
fn c_len(s: &str) -> Option<usize> {
let cs = CString::new(s).ok()?;
Some(cs.as_bytes().len()) // EXCLUDES the trailing NUL
}
fn from_c_bytes(buf: &[u8]) -> Option<String> {
CStr::from_bytes_until_nul(buf) // read up to the first NUL
.ok()
.and_then(|cstr| cstr.to_str().ok().map(|s| s.to_owned()))
}
The defining footgun: CString::new must fail on an interior NUL, because
otherwise C would see a truncated string. The Result return type is the type
system enforcing that.
Two byte views to keep straight:
as_bytes()— the content without the terminator (b"hello").as_bytes_with_nul()— content including it (b"hello\0").
Receiving from C: CStr::from_bytes_until_nul(b"hello\0garbage") reads "hello" and
ignores the rest; with no NUL at all it returns an error (None after .ok()).
8. Conversions & validation: validate on the way back
This rung ties the topic together. Going to bytes is free and infallible — the
UTF-8 invariant only loosens. Coming back must validate, which is precisely why
those functions return Result (or substitute via lossy).
// text -> bytes: free
s.as_bytes(); // &str -> &[u8] (borrowed)
s.into_bytes(); // String -> Vec<u8> (just unwraps the Vec)
// bytes -> text: must validate
fn decode_strict(bytes: &[u8]) -> Result<String, std::str::Utf8Error> {
std::str::from_utf8(bytes).map(|s| s.to_owned())
}
fn decode_lossy(bytes: &[u8]) -> (String, bool) {
let cow = String::from_utf8_lossy(bytes);
(cow.to_string(), matches!(cow, Cow::Owned(_))) // Owned == it substituted
}
The map of conversions:
| From | To | Method | Fallible? |
|---|---|---|---|
&str | &[u8] | as_bytes() | no (free, borrowed) |
String | Vec<u8> | into_bytes() | no (free) |
&[u8] | &str | str::from_utf8 | Result<_, Utf8Error> (borrowed) |
Vec<u8> | String | String::from_utf8 | Result<_, FromUtf8Error> (owned) |
&[u8] | Cow<str> | String::from_utf8_lossy | never (substitutes �) |
Use the borrowing str::from_utf8 when the caller should keep ownership of the
bytes; use String::from_utf8 when you already own a Vec<u8> and want to consume
it. decode_lossy reads the Cow variant to report whether substitution happened —
no second scan for � needed.
Footguns
| Trap | What happens | Fix |
|---|---|---|
s.len() as “number of characters” | counts bytes; off for any non-ASCII | s.chars().count() |
&s[0..n] mid-codepoint | runtime panic | s.get(0..n), check is_char_boundary, or slice on a known boundary like len_utf8() |
s[i] char indexing | does not compile | iterate chars() / char_indices() |
Forcing a filename into String | data loss or panic on non-UTF-8 | keep it OsStr; to_str()/to_string_lossy() at the edge |
format!("{dir}/{file}") | breaks cross-platform | Path::join / set_extension |
CString::new with interior NUL | returns Err (would truncate in C) | handle the Result; never .unwrap() on untrusted input |
String::from_utf8 on arbitrary bytes | Err on invalid UTF-8 | from_utf8 (handle Result) or from_utf8_lossy |
Real-world patterns
- Accept
&str, storeString. APIs take&str(orimpl AsRef<str>) for flexibility and own aStringinternally. Cow<str>for “usually borrowed, sometimes owned.”to_string_lossy,from_utf8_lossy, and many parsers returnCowso the common (clean) case is zero-copy and only the exceptional case allocates. (See the dedicatedCownote.)- Stay in
OsStr/Pathas long as possible. Convert tostronly at the boundary where you genuinely need UTF-8 (logging, display, parsing), and decide there how to handle non-UTF-8. CStringlives as long as the C call. Hold theCStringin a binding while C borrows its pointer; if it drops first, the pointer dangles.
Capstone insight
The capstone reimplements str::chars() from raw bytes, which forces you to own
the UTF-8 encoding rather than trust it:
fn decode_utf8(bytes: &[u8]) -> Option<(char, usize)> {
let lead = *bytes.first()?;
let length = match lead {
0x00..=0x7F => 1,
0xC2..=0xDF => 2, // note: starts at C2, not C0
0xE0..=0xEF => 3,
0xF0..=0xF4 => 4, // note: ends at F4, not F7
_ => return None, // continuation byte or invalid lead
};
if bytes.len() < length {
return None; // truncated
}
let mut cp = match length {
1 => u32::from(lead),
2 => u32::from(lead & 0b0001_1111),
3 => u32::from(lead & 0b0000_1111),
4 => u32::from(lead & 0b0000_0111),
_ => unreachable!(),
};
for &b in &bytes[1..length] {
if b & 0b1100_0000 != 0b1000_0000 { // must be 10xxxxxx
return None;
}
cp = (cp << 6) | u32::from(b & 0b0011_1111);
}
char::from_u32(cp).map(|ch| (ch, length))
}
The encoding, made explicit:
| Bytes | Lead pattern | Payload bits | Code point range |
|---|---|---|---|
| 1 | 0xxxxxxx | 7 | U+0000..U+007F |
| 2 | 110xxxxx | 5 + 6 = 11 | U+0080..U+07FF |
| 3 | 1110xxxx | 4 + 12 = 16 | U+0800..U+FFFF |
| 4 | 11110xxx | 3 + 18 = 21 | U+10000..U+10FFFF |
Three layers of validation fall out naturally:
- The lead-byte ranges already exclude the overlong 2-byte encodings (
0xC0/0xC1) and 4-byte leads beyondU+10FFFF(0xF5..). - Each continuation byte is checked against
10xxxxxx. char::from_u32rejects surrogates (U+D800..U+DFFF) and out-of-range values — the final gate that defines a valid Unicode scalar.
Wrapping it in an iterator is then trivial — and the ? gives you “stop at end of
input or the first invalid byte” in one line:
impl<'a> Iterator for Utf8Chars<'a> {
type Item = char;
fn next(&mut self) -> Option<char> {
let (ch, n) = decode_utf8(&self.bytes[self.pos..])?;
self.pos += n;
Some(ch)
}
}
On valid input it yields exactly what str::chars() does — proving the mental model
end to end. (The one gap a fully conformant decoder closes that this one doesn’t:
rejecting overlong 3- and 4-byte encodings, which requires a per-length minimum
code-point check.)
Explain it back
- Why does
"café".len()return5? What returns4? - When does
&s[a..b]panic, and what are the two safe alternatives? - Why can’t a filename always be a
String? What two methods crossOsStr→str, and how do they differ? - Why does
to_string_lossyreturn aCow? What doesCow::Borrowedtell you? - Why does
CString::newreturn aResult? What breaks if it didn’t? - Which direction (text → bytes or bytes → text) is fallible, and why?
- In
decode_utf8, why does the 2-byte lead range start at0xC2instead of0xC0?
See also
Cow(Clone-on-Write) — the return type of the lossy conversions.Borrow/ToOwned— the borrowed/owned pairing (str↔String) generalized.- Collections deep-dive —
HashMapkey lookup viaBorrow<str>, another place the view/owned split shows up.
Iterators end-to-end
Ladder:
src/bin/iterators.rs· Run:cargo run --bin iterators· Phase 3 · 9 rungs
TL;DR
An iterator is a tiny state machine with one required method:
fn next(&mut self) -> Option<Self::Item>;
That’s the whole engine. Everything else — map, filter, zip, sum, collect —
is built on top of next. Two facts unlock the entire topic:
- Adapters are lazy.
map/filter/takedon’t do anything; each one returns a new struct that remembers what to do. No work happens until a consumer starts pulling. - Consumers drive the pull.
for,collect,sum,count,nextare the verbs. They callnext()in a loop, and that cascades the pull back through every adapter to the source — one item at a time.
for x in thing is sugar for IntoIterator::into_iter(thing) followed by a
while let Some(x) = it.next() loop. Master next, laziness, and IntoIterator, and
the rest is vocabulary.
Why this exists (from first principles)
Imagine you didn’t have iterators. To “sum the squares of the even numbers” you’d write:
let mut total = 0;
for &x in &nums {
if x % 2 == 0 {
total += x * x;
}
}
This works, but it fuses three independent ideas — selecting, transforming, accumulating — into one tangled loop with a mutable accumulator. You can’t reuse the “keep evens” step, you can’t swap the accumulation, and the intent is buried in mechanics.
The iterator abstraction separates these concerns into composable pieces:
fn sum_of_even_squares(nums: &[i32]) -> i32 {
nums.iter().filter(|&x| x % 2 == 0).map(|x| x * x).sum()
}
Each verb does one thing. The catch: if every step eagerly built an intermediate Vec,
this would be slower than the hand-written loop and couldn’t handle infinite sequences.
So Rust makes adapters lazy — they compile down to roughly the same machine code as
the hand-written loop (zero-cost), and they compose, and they work on endless
streams. That combination is why the abstraction is worth having.
The compiler is enforcing one core protocol — the
Iteratortrait — and giving you ~70 default methods for free the moment you supplynext.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | foundations | filter/map/sum chain | replace the manual loop with composable verbs |
| 2 | foundations | iter / iter_mut / into_iter | the same data yields &T, &mut T, or T |
| 3 | mechanics | adapter zoo | enumerate, zip, flat_map, filter_map, fold |
| 4 | mechanics | laziness, proven | a closure that runs 0 times; an infinite source tamed by take |
| 5 | footgun | ownership & collect traps | the move trap (E0382), turbofish, Result short-circuit |
| 6 | footgun | impl Iterator for Fib | write next() once, inherit every adapter |
| 7 | real-world | IntoIterator + DoubleEndedIterator | how for works; rev(); size_hint |
| 8 | real-world | custom lazy adapter + extension trait | .pairs() on every iterator (the itertools pattern) |
| 9 | capstone | mini iterator engine from scratch | own trait + lazy adapters + a consumer; prove the pull-chain |
The ideas, built up
1. A chain is three verbs, not one loop
nums.iter().filter(|&x| x % 2 == 0).map(|x| x * x).sum()
The subtlety hides in the filter closure. nums.iter() yields &i32, so filter’s
closure receives &&i32 (filter borrows each item to inspect it without consuming).
The pattern |&x| strips one reference layer, so inside the closure x: &i32, and
x % 2 auto-derefs the rest. This |&x| destructuring-in-the-binding is the idiomatic
way to deal with the double reference — cleaner than writing **x.
sum() is a consumer: it’s the verb that finally calls next() repeatedly and
folds the results. Without it, nothing runs (see rung 4).
2. One collection, three iterators
A Vec<T> gives you three entry points, distinguished by the item type they yield:
| Call | Item type | Effect on the source |
|---|---|---|
.iter() | &T | borrows; source survives |
.iter_mut() | &mut T | borrows mutably; mutate in place |
.into_iter() | T | consumes; source is gone afterward |
fn count_long(words: &[String]) -> usize {
words.iter().filter(|w| w.len() > 3).count() // &T: caller keeps `words`
}
fn double_in_place(nums: &mut Vec<i32>) {
nums.iter_mut().for_each(|n| *n *= 2); // &mut T: write through the ref
}
fn join_owned(words: Vec<String>) -> String {
words.into_iter().collect::<Vec<_>>().join(", ") // T: takes ownership, `words` consumed
}
The choice is forced by what you need to do: read-only (iter), mutate (iter_mut), or
take ownership of the values (into_iter). for_each here is itself a consumer — it’s
the iterator-land equivalent of a for loop body.
3. The adapter zoo
Five workhorses you reach for daily:
// enumerate yields (index, &value); keep the index where the value is even
nums.iter().enumerate()
.filter_map(|(i, &x)| if x % 2 == 0 { Some(i) } else { None })
.collect()
// zip welds two iterators and STOPS at the shorter one
names.iter().zip(scores).map(|(n, s)| format!("{}={}", n, s)).collect()
// flat_map: each item produces an iterator; they're concatenated flat
words.iter().flat_map(|w| w.chars()).collect()
// filter_map: filter + map in one pass; .ok() turns Result -> Option, dropping Errs
strs.iter().filter_map(|s| s.parse().ok()).collect()
// fold: thread an accumulator; the closure must RETURN the (mutated) acc
s.chars().fold(HashMap::new(), |mut acc, c| {
*acc.entry(c).or_insert(0) += 1;
acc
})
Two facts worth burning in:
zipstops at the shorter input.["a","b","c"].zip([9])yields just("a", 9). No panic, no padding — it’s how you safely walk two sequences of unknown relative length.filter_mapis filter + map fused. Whenever you find yourself writing.filter(...).map(...)where the filter and map both inspect the same thing (especiallyOption/Result),filter_mapdoes it in one pass.
4. Laziness, proven
This is the conceptual heart. Build a million-element chain but never consume it:
fn lazy_never_runs(log: &mut Vec<i32>) {
let _lazy = (0..1_000_000).map(|x| log.push(x));
// no consumer called -> the closure body runs ZERO times
}
// afterwards: log.len() == 0
The map closure pushes to log every time it runs — and it runs zero times,
because nobody pulled. The compiler even hints at this: _lazy triggers a must_use /
unused warning, which is literally “you built an iterator and never drove it.”
Laziness is also what makes infinite iterators usable:
fn first_4_triple_squares() -> Vec<u64> {
(0u64..) // endless
.filter(|n| n % 3 == 0 && *n != 0) // note: 0 is divisible by 3 — exclude it
.map(|n| n * n)
.take(4) // stops the pull after 4 items
.collect()
}
// -> [9, 36, 81, 144] (from 3, 6, 9, 12)
If any adapter were eager, (0u64..) would hang your machine forever. take(4) bounds
the pulling. The mental model to lock in:
Adapters are nouns (a recipe). Consumers are verbs (they run it).
5. Where iterators bite
The move trap (E0382). into_iter() takes ownership of the receiver:
// WRONG — does not compile
let total: i32 = v.into_iter().sum();
let n = v.len(); // error[E0382]: borrow of moved value: `v`
The fix isn’t .clone() (the compiler suggests it, but that allocates a whole new Vec).
Either borrow instead of consume, or capture the length first:
// OK — borrow to sum; `v` is fully intact for .len()
fn sum_then_len(v: Vec<i32>) -> (i32, usize) {
let total: i32 = v.iter().sum();
let n = v.len();
(total, n)
}
collect needs a target type. collect is generic over its return type via
FromIterator. With nothing telling it what to build, inference fails. Pin it with a
binding annotation or a turbofish:
let v: Vec<i32> = (0..5).map(|x| x * 2).collect(); // annotate the binding
(0..5).map(|x| x * 2).collect::<Vec<i32>>() // or turbofish
When the function’s return type already pins it, you need neither.
collect into Result short-circuits. The single most-loved collect trick:
fn parse_all_or_fail(strs: &[&str]) -> Result<Vec<i32>, std::num::ParseIntError> {
strs.iter().map(|s| s.parse::<i32>()).collect()
}
collect transposes an iterator of Result<T, E> into a single Result<Vec<T>, E>:
Ok(vec) if every element parsed, or the first Err the moment one fails (and it
stops pulling). That’s validate-all-or-bail in one line. (The same works for
Option: Iterator<Item = Option<T>> collects to Option<Vec<T>>.)
6. Implement Iterator yourself
The payoff rung. The entire trait is one required method; supply it and dozens of adapters
appear for free, because they’re default methods riding on next:
struct Fib { curr: u64, next: u64 }
impl Iterator for Fib {
type Item = u64;
fn next(&mut self) -> Option<Self::Item> {
let curr = std::mem::replace(&mut self.curr, self.next);
self.next = curr + self.next;
Some(curr) // infinite: never None — bounding it is the caller's job
}
}
std::mem::replace(&mut self.curr, self.next) does two jobs atomically: it returns the
old curr (the value to yield) while overwriting self.curr with self.next. That
sidesteps the classic stale-value bug where you overwrite a field before you’ve finished
reading it.
The architectural lesson: three lines of next() bought you take, filter, sum,
nth, collect and the rest:
Fib::new().take(10).collect::<Vec<_>>(); // [0,1,1,2,3,5,8,13,21,34]
Fib::new().take(10).filter(|n| n % 2 == 0).sum::<u64>(); // 44
Fib::new().nth(7); // Some(13)
7. How for actually works: IntoIterator
for is not compiler magic. for x in thing { ... } desugars to:
let mut it = IntoIterator::into_iter(thing);
while let Some(x) = it.next() { ... }
So to make your own type loopable, implement IntoIterator. Real collections implement
it three times so for x in v, for x in &v, and for x in &mut v each pick the
right item type (T, &T, &mut T). The &v impl not consuming v is exactly what
lets you loop over a collection you still need afterward.
The ladder builds the consuming (T) variant by delegating to the standard library’s
vec::IntoIter:
struct MyVec<T> { items: Vec<T> }
struct MyVecIntoIter<T> { inner: std::vec::IntoIter<T> }
impl<T> IntoIterator for MyVec<T> {
type Item = T;
type IntoIter = MyVecIntoIter<T>;
fn into_iter(self) -> Self::IntoIter {
MyVecIntoIter { inner: self.items.into_iter() }
}
}
impl<T> Iterator for MyVecIntoIter<T> {
type Item = T;
fn next(&mut self) -> Option<T> { self.inner.next() }
fn size_hint(&self) -> (usize, Option<usize>) { self.inner.size_hint() }
}
Two extras that matter for real APIs:
size_hintreturns(lower, Option<upper>). Consumers likecollectuse it to pre-allocate exactly the right capacity. Forwarding it (here(4, Some(4))for a 4-element vec) avoids reallocation churn.DoubleEndedIteratoraddsnext_back()— pull from the other end. That single method is allrev()needs:
impl<T> DoubleEndedIterator for MyVecIntoIter<T> {
fn next_back(&mut self) -> Option<T> { self.inner.next_back() }
}
// now: coll.into_iter().rev().collect() works
8. A custom lazy adapter — the itertools pattern
To add a new adapter that works on every iterator, you write two things: a stateful
struct that implements Iterator, and a blanket extension trait that hands out the
method. The ladder builds .pairs(), which turns [1,2,3,4] into overlapping windows
(1,2), (2,3), (3,4):
struct Pairs<I: Iterator> {
inner: I,
prev: Option<I::Item>,
}
impl<I> Iterator for Pairs<I>
where
I: Iterator,
I::Item: Clone, // we keep a copy of prev AND emit it
{
type Item = (I::Item, I::Item);
fn next(&mut self) -> Option<Self::Item> {
if self.prev.is_none() {
self.prev = self.inner.next(); // seed once, on the first call
}
let curr = self.inner.next()?; // ? bails on exhausted/empty source
let prev = self.prev.replace(curr.clone())?; // install new prev, hand back old
Some((prev, curr))
}
}
self.prev.replace(curr.clone()) is the elegant move: it stores curr as the new
remembered value and returns the previous one to emit — slide and extract in a single
call. The critical invariant is that each next() pulls at most one new item from
inner; that’s what keeps .pairs() lazy enough to run on an infinite source.
The extension trait grafts the method onto everything:
trait IterPairsExt: Iterator + Sized {
fn pairs(self) -> Pairs<Self> {
Pairs { inner: self, prev: None }
}
}
impl<I: Iterator> IterPairsExt for I {} // blanket impl: every Iterator now has .pairs()
This composes like any built-in adapter, including on infinite streams:
let diffs: Vec<u64> = (0u64..).map(|x| x * x).pairs().map(|(a, b)| b - a).take(4).collect();
// squares 0,1,4,9,16 -> consecutive diffs [1, 3, 5, 7]
This is precisely how the itertools crate delivers .tuple_windows(), .dedup(),
.chunks(), and friends.
Footguns
| Trap | What bites | Fix |
|---|---|---|
into_iter() move | v is consumed; later v.len() is E0382 | use .iter() to borrow, or read len() first; don’t .clone() |
collect can’t infer | “type annotations needed” | annotate the binding or turbofish collect::<Vec<_>>() |
zip length mismatch | silently stops at the shorter side | intended — but know it won’t error on ragged inputs |
| infinite source, eager step | hangs forever | bound with take/take_while; keep every adapter lazy |
| building but never consuming | a must_use warning; nothing happens | remember adapters are inert until a consumer pulls |
0 in divisibility filters | 0 % n == 0 for all n | guard && *x != 0 when you mean “positive multiples” |
Real-world patterns
collect::<Result<_, _>>()for “parse/validate everything or fail fast” — ubiquitous in config loading, deserialization, and request handling.- Returning
impl Iterator<Item = T>from functions to expose a lazy stream without committing to a concrete type or allocating aVec. - Extension traits with blanket impls (
itertools::Itertools,rayon’sParallelIterator) — the standard way third-party crates bolt new methods onto every iterator. size_hint+DoubleEndedIteratorare whyVec/slice iteration pre-allocates perfectly and supportsrev(),rposition, etc.
Capstone insight
The build-it-from-scratch rung re-implements the core of std::iter with no help from it:
a MyIterator trait (one required next), default methods map/filter/take that
return lazy adapter structs, a collect_vec consumer, and a Counter source.
trait MyIterator: Sized { // Sized so adapters can take `self` by value
type Item;
fn next(&mut self) -> Option<Self::Item>;
fn map<B, F: FnMut(Self::Item) -> B>(self, f: F) -> MyMap<Self, F> {
MyMap { iter: self, f } // just builds a struct — no work yet
}
// filter, take similar...
fn collect_vec(mut self) -> Vec<Self::Item> { // THE consumer: where pulling happens
let mut out = Vec::new();
while let Some(x) = self.next() { out.push(x); }
out
}
}
Each adapter implements MyIterator by pulling from its inner iterator inside its own
next:
impl<I: MyIterator> MyIterator for MyTake<I> {
type Item = I::Item;
fn next(&mut self) -> Option<I::Item> {
if self.remaining == 0 { None } // <- the brake that stops an infinite source
else { self.remaining -= 1; self.iter.next() }
}
}
The aha: when collect_vec calls next() on the outermost MyTake, it triggers a
pull-chain — take asks filter, filter asks map, map asks Counter — one
item at a time, on demand. filter may loop internally and skip arbitrarily many source
items before returning one (so take never “sees” the skips), and take’s counter is the
only thing keeping the infinite Counter from running forever. That cascade is how
every iterator in Rust works. Note also that calling a closure stored in a struct field
needs parens — (self.f)(x) — to disambiguate from a method call.
Explain it back
- What is the only method you must implement for
Iterator, and why does that give youmap/filter/sumfor free? - What does
for x in thingdesugar to, exactly? Which trait does it call? - Why does
(0u64..).filter(...).map(...).take(4).collect()not hang, but(0u64..).filter(...).collect()would? - What’s the difference between an adapter and a consumer? Name three of each.
- Why does
let total = v.into_iter().sum(); v.len()fail to compile, and what are two fixes that don’t clone? - How does
collect::<Result<Vec<_>, _>>()decide betweenOkandErr, and when does it stop pulling? - In the
.pairs()adapter, what guarantees laziness — i.e., why does eachnext()pull at most one new item? - What single method unlocks
rev(), and what doessize_hintbuy a consumer?
See also
- Associated types vs generic params — why
Iterator::Itemis an associated type, not a generic parameter. - Blanket impls & coherence — the mechanism behind the
.pairs()extension trait. - Collections deep-dive — what you’re usually iterating over.
- Static vs dynamic dispatch — monomorphized iterator chains vs
Box<dyn Iterator>.
Threads & scoped threads
Ladder:
src/bin/threads.rs· Run:cargo run --bin threads· Phase 4 · 9 rungs
TL;DR
There are two ways to start an OS thread in std, and the whole topic is the difference between them:
thread::spawnlaunches a thread that can outlive the function that started it. Because nobody promises when it ends, its closure must own everything it touches: the bound isF: 'static. You get aJoinHandle<T>to later collect the thread’s return value (or its panic).thread::scopeopens a region that blocks until every thread spawned inside it has finished, right at the closing brace. Since the threads provably can’t escape that region, the borrow checker relaxes'staticdown to “outlives the scope” — so scoped threads can borrow local variables, even mutably.
'static ownership versus structured borrowing. That is the entire mental model.
Why this exists (from first principles)
A thread is a separate flow of execution that the OS scheduler can run at any time, in any order, possibly still running after the function that spawned it has returned. That last clause is the source of every rule here.
Consider what spawn would have to allow if it let a closure borrow a local:
fn danger() {
let data = vec![1, 2, 3];
thread::spawn(|| println!("{:?}", data)); // borrows `data`
} // <- `data` is dropped HERE, freeing its heap buffer
// ...but the spawned thread may not have run yet.
The thread holds a reference into data’s heap allocation, but danger frees that allocation the instant it returns. If the scheduler runs the thread afterward, it reads freed memory — a use-after-free. Rust has no garbage collector and no runtime to keep data alive, so the only way to make this sound at compile time is to forbid it. That is what the 'static bound on spawn does: any reference the closure captures must be valid for the entire rest of the program (&'static), which a borrow of a local is not.
'static does not mean “no outside data.” Owned data is fine — you can move a String or Vec into the thread, transferring ownership so there is no dangling borrow to worry about. 'static specifically forbids borrows that could dangle.
So spawn gives you safety by demanding ownership. But sometimes you genuinely want to lend a local to a few threads, run them in parallel, and get the borrow back — the classic “split this array across cores” pattern. thread::scope exists to make exactly that sound: it adds the one missing guarantee (all threads join before the borrow ends) so the borrow checker can permit the borrow.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | Foundations | Spawn & join | spawn returns a JoinHandle<T>; join() retrieves the value |
| 2 | Foundations | Many handles | Spawn all, then join all — joining in the loop serializes |
| 3 | Mechanics | move & ownership | The closure must own captured data; move makes it 'static |
| 4 | Mechanics | Panicking threads | A panic is caught and returned as join() -> Err(payload) |
| 5 | Footgun | The 'static wall | Borrowing a local in spawn fails (E0373/E0597) — and why |
| 6 | Footgun | thread::scope rescue | Same borrow, now legal; many shared & reads at once |
| 7 | Real-world | Scoped parallel mutate | split_at_mut → disjoint &mut chunks mutated in parallel |
| 8 | Real-world | Parallel fold / fan-in | Map-reduce: chunk → partial results → combine in main |
| 9 | Capstone | parallel_map | A generic, order-preserving rayon-lite |
The ideas, built up
1. Spawn returns a handle; join collects the result
fn spawn_and_join() -> i32 {
let handle = thread::spawn(|| 2 + 2);
handle.join().unwrap()
}
thread::spawn(closure) returns immediately with a JoinHandle<T>, where T is the closure’s return type. The new thread runs concurrently. handle.join() blocks the calling thread until that thread finishes, then hands back its result.
Why does join() return a Result rather than a bare T? Because the thread might not have finished cleanly — it could have panicked. Ok(value) means it returned normally; Err(payload) means it panicked. The .unwrap() here says “I expect success,” which is fine until rung 4 deliberately breaks it.
2. Spawn all, then join all
fn squares_in_parallel(n: usize) -> Vec<usize> {
let mut handles = Vec::with_capacity(n);
for i in 0..n {
handles.push(thread::spawn(move || i * i)); // spawn ALL first
}
handles.into_iter().map(|h| h.join().unwrap()).collect() // THEN join
}
The trap this rung sets: if you call .join() inside the spawn loop, each thread finishes before the next one starts, and you have written sequential code with extra steps. To get real parallelism you must launch every thread first, then join them in a second pass.
There is a subtle correctness point too: the results come out in spawn order, not completion order. Thread 4 might finish before thread 1, but because you join the handles in the order you stored them, the squares land sorted regardless of who finished when. Order is determined by which handle you join, never by timing.
Note move || i * i: each iteration’s i is captured by value, so every closure owns its own copy. Without move, the closure would try to borrow the loop variable.
3. move and the 'static contract
fn append_in_thread(s: String) -> String {
let handle = thread::spawn(move || s + " world");
handle.join().unwrap()
}
Write this without move and the compiler stops you with E0373: closure may outlive the current function, but it borrows s. That error is the 'static rule speaking. move resolves it by transferring ownership of s into the closure: the thread now owns the String, there is no borrow of a local left to dangle, and the closure satisfies 'static.
(s + " world" consumes the owned String and reuses its buffer — a small idiom worth noting, but the lesson is the move.)
4. A panic in a thread is caught, not propagated
fn catch_panic_message() -> String {
let handle = thread::spawn(|| { panic!("boom"); });
match handle.join() {
Ok(value) => value,
Err(payload) => payload.downcast_ref::<&str>().unwrap().to_string(),
}
}
A panic in a spawned thread does not unwind into the parent. It unwinds that thread, is captured at the thread boundary, and is delivered to whoever calls join() as Err(payload). This is why threads isolate failure: one worker blowing up doesn’t tear down main.
The payload type is Box<dyn Any + Send + 'static> — type-erased, because a panic can carry any value. panic!("boom") with a string literal stores a &'static str, so you recover it with downcast_ref::<&str>(). (A panic!("{}", x) with formatting stores a String instead, so the downcast target depends on how the panic was raised.)
Caveat: this catch-and-return behavior only exists under the default
panic = "unwind". If the crate is built withpanic = "abort", a panicking thread aborts the whole process andjoinnever gets the chance to returnErr.
5. The 'static wall, seen directly
This rung is built to fail to compile — the error is the lesson.
fn sum_with_spawn() -> i32 {
let data = vec![1, 2, 3, 4];
let handle = thread::spawn(|| data.iter().sum::<i32>()); // E0373/E0597
handle.join().unwrap()
}
The compiler rejects the borrow of data because spawn requires 'static and data is a local that drops when the function returns. Adding move makes it compile — but notice that it is now a different program: the thread owns data, so the function can no longer use data afterward, and if data had been borrowed from a caller you don’t own, move wouldn’t even be available.
That gap — “I want to lend a local to a thread and get it back” — is precisely the hole the next rung fills.
6. thread::scope relaxes 'static to “outlives the scope”
fn sum_halves_scoped(data: &[i32]) -> i32 {
let mid = data.len() / 2;
thread::scope(|s| {
let s1 = s.spawn(|| data[..mid].iter().sum::<i32>());
let s2 = s.spawn(|| data[mid..].iter().sum::<i32>());
s1.join().unwrap() + s2.join().unwrap()
})
}
thread::scope(|s| { ... }) gives you a scope handle s; you spawn with s.spawn(...) instead of thread::spawn(...). The defining property: the scope does not return until every thread spawned inside it has finished. That join happens automatically at the closing brace.
Because the runtime now guarantees the threads end before the scope (and therefore before data) does, the borrow only needs to outlive the scope, not the whole program. So the closures can capture &data and mid by reference, with no move and no 'static. Two threads holding shared & borrows of the same slice at once is fine — shared reads never conflict. No Arc, no Mutex, no clone.
7. Disjoint mutable borrows in parallel
Shared reads were easy. The hard, genuinely useful case is mutating different parts of one slice from different threads.
fn double_in_parallel(data: &mut [i32]) {
let (left, right) = data.split_at_mut(data.len() / 2);
thread::scope(|s| {
s.spawn(move || left.iter_mut().for_each(|x| *x *= 2));
s.spawn(move || right.iter_mut().for_each(|x| *x *= 2));
});
}
Two &mut into the same slice is normally a borrow-checker violation. The key that unlocks it is slice::split_at_mut, which returns two provably non-overlapping mutable sub-slices — the standard library guarantees they share no element, so handing each to a different thread races nothing.
Note the closures need move this time, in contrast to rung 6. A &mut is not Copy and cannot be shared, so each thread must take its sub-slice by moving it in. And since the scope joins at }, you can let the threads auto-join without binding their handles. This split → scope → parallel-mutate skeleton is exactly how rayon’s par_iter_mut works underneath.
8. Map-reduce: fan out partials, fold them in
fn parallel_sum_of_squares(data: &[i64], n: usize) -> i64 {
if data.is_empty() || n == 0 { return 0; }
thread::scope(|s| {
let chunk_len = (data.len() + n - 1) / n; // ceil(len / n)
let mut handles = Vec::with_capacity(n);
for chunk in data.chunks(chunk_len) {
handles.push(s.spawn(move || chunk.iter().map(|x| x * x).sum::<i64>()));
}
handles.into_iter().map(|h| h.join().unwrap()).sum()
})
}
The generalized form of rung 7 and the heart of map-reduce: split into chunks, each worker computes a partial result, the main thread combines the partials. data.chunks(chunk_len) yields shared &[i64] sub-slices; chunk_len = ceil(len / n) ensures at most n chunks; the guards handle n == 0 and empty input.
The collect-then-join discipline from rung 2 reappears: push every handle, then join them, so the workers actually run concurrently. (For a sum the order of partials is irrelevant, but the parallelism still depends on not joining inside the loop.)
Footguns
| Trap | What happens | Fix |
|---|---|---|
join() inside the spawn loop | Threads run one at a time — no parallelism | Spawn all handles first, join in a second pass |
Borrowing a local in thread::spawn | E0373 / E0597 — closure may outlive the borrow | move to transfer ownership, or use thread::scope to borrow |
Reaching for move to “fix” rung 5 | Compiles, but the thread now owns the data — a different program | Use thread::scope when you need to borrow and get it back |
Two &mut into one slice | Borrow-checker rejection | split_at_mut / chunks_mut for provably-disjoint sub-slices |
| Forgetting the panic boundary | Under panic = "abort", a thread panic kills the process | Default unwind lets join() return Err; don’t rely on it under abort |
| Assuming result order = finish order | Results follow handle order, not timing | Keep handles ordered; order is deterministic regardless of scheduling |
Real-world patterns
- Structured parallelism (
thread::scope) is the modern default for “fork a few workers over borrowed data and join them here.” Before it was stabilized (Rust 1.63), this requiredArc+'staticgymnastics or unsafe; now it’s a safe, allocation-free block. split_at_mut/chunks_mutis the standard way to express “these regions are disjoint” to the borrow checker — the foundation of every data-parallel mutation.- Collect-then-join is how you keep
JoinHandles parallel; joining eagerly is the most common accidental-serialization bug. rayonis the production answer to everything in rungs 6–9:par_iter(),par_iter_mut(),map().sum(),map().collect(). The capstone is a hand-rolled, fixed-chunk version of exactly that API, which is why writing it cements whatrayondoes for you.
Capstone insight
fn parallel_map<T, R, F>(data: &[T], n: usize, f: F) -> Vec<R>
where
T: Sync, // workers share &[T]; &T crossing threads needs Sync
R: Send, // each result is produced on a worker and moved back
F: Fn(&T) -> R + Sync, // f runs on many threads at once → &F must cross → Sync
{
if data.is_empty() || n == 0 { return Vec::new(); }
thread::scope(|s| {
let f = &f; // borrow the closure ONCE
let chunk_len = (data.len() + n - 1) / n;
let mut handles = Vec::with_capacity(n);
for chunk in data.chunks(chunk_len) {
handles.push(s.spawn(move || chunk.iter().map(f).collect::<Vec<R>>()));
}
handles.into_iter().flat_map(|h| h.join().unwrap()).collect() // concat in order
})
}
Three insights make this work, and each is a payoff from an earlier rung:
-
The bounds are forced by data flow, not decoration.
T: Syncbecause each worker reads&[T]and a&Tonly crosses a thread boundary safely ifT: Sync.R: Sendbecause each result is created on a worker and moved back to main.F: Fn(&T) -> R + Syncbecause the same closure is called from several threads concurrently, so&Fmust cross threads — which is whatSynccertifies. Write it without the bounds and the compiler dictates them one error at a time. -
let f = &f;thenmove ||shares one closure. A baremovewould try to movefinto every worker, but you only have onef. Rebinding to a shared reference and moving that reference (which isCopy) lets all workers borrow the single closure. This is the concrete reason the bound isF: Syncrather thanF: Send + Clone. -
Order falls out of structure. Each worker returns a
Vec<R>for its chunk. Becausedata.chunks()yields chunks left to right and the handles stay in that order,flat_mapover the joined results concatenates per-chunk outputs in global order — no sorting, no indices. Order preservation is a consequence of keeping handles ordered (the rung-2 lesson), not an extra step.
Put together: scope provides the borrowing, chunking provides the parallel decomposition, the Send/Sync bounds provide the safety proof, and ordered handles provide deterministic output. That is rayon::par_iter().map().collect() with the lid off.
Explain it back
- Why does
thread::spawnrequireF: 'static, and why is moving an ownedStringin still allowed? - What does
thread::scopeguarantee that lets the borrow checker accept a borrow of a local? Where exactly does the join happen? - You spawned 5 threads and joined them in order. A later-spawned thread finishes first. What order are the results in, and why?
- Why does
join()return aResult? Under what build setting does that stop being true? - Why is
split_at_mutthe thing that makes parallel mutation type-check, when two&mutto a slice normally don’t? - In
parallel_map, why is the bound onFSyncand notSend? What would change if each worker needed its own copy off?
See also
Send&Sync— the two marker traits the capstone’s bounds rest on (next ladder in Phase 4).- Rc / Arc —
Arc<Mutex<T>>is the alternative when you can’t use a scope (threads that outlive the spawning frame). - Lifetimes in depth — the
'staticbound and “outlives” reasoning that scope relaxes. - Drop & ordering — why “the scope joins at the closing brace” matters for when borrows end.
Send & Sync deeply
Ladder:
src/bin/send_sync.rs· Run:cargo run --bin send_sync· Phase 4 · 9 rungs
TL;DR
Two marker traits decide what is allowed to cross a thread boundary, and they mean exactly two different things:
T: Send— it is safe to move ownership of aTto another thread.T: Sync— it is safe to share&Tbetween threads. Formally:T: Sync ⟺ &T: Send.
Both are auto traits: the compiler implements them for you, structurally, from
your fields. There is no #[derive(Send)]. A struct is Send iff every field is
Send; Sync iff every field is Sync. One non-Send field poisons the whole
type — like a single rotten apple.
The two axes are independent. The four combinations all exist, and the
surprising ones (Cell is Send but not Sync; MutexGuard is Sync but not
Send) fall straight out of the one question: can a reference to this safely
cross threads?
Why this exists (from first principles)
A data race is two threads touching the same memory at the same time with at least
one write and no synchronization. It is undefined behavior in every systems
language. Most languages fight data races at runtime (locks you must remember to
take) or not at all. Rust eliminates a whole class of them at compile time — and
Send/Sync are the mechanism.
The insight: a data race needs shared mutable access across threads. Rust already controls sharing and mutation within a single thread via ownership and borrowing. To extend that guarantee across threads, the compiler needs to know two facts about every type:
- Is it sound to hand this value off to another thread? (
Send) - Is it sound for two threads to hold a shared reference to it at once?
(
Sync)
thread::spawn then simply requires these bounds. If your type can’t prove it,
the code doesn’t compile. The race becomes impossible to write rather than a bug
you find in production.
pub fn spawn<F, T>(f: F) -> JoinHandle<T>
where
F: FnOnce() -> T + Send + 'static,
T: Send + 'static,
The closure is moved onto the new thread, so it must be Send; everything it
captures must therefore be Send too. That single bound is the gate everything
else passes through.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | foundations | sum_on_thread | spawn requires Send; move owned data in, join the result out |
| 2 | foundations | parallel_contains | Sync = shareable &T; many threads read one &haystack via scope |
| 3 | mechanics | assert_send/assert_sync | auto-derivation is structural; build compile-time probes |
| 4 | mechanics | check_4 | predict then verify Send/Sync across the std library |
| 5 | footgun | count_racy vs count_atomic | reproduce the non-atomic refcount race that makes Rc !Send |
| 6 | footgun | the four quadrants | Cell/RefCell = Send+!Sync; MutexGuard = !Send+Sync |
| 7 | real-world | concurrent_sum | Arc<Mutex<T>> (Send+Sync) vs Rc<RefCell<T>> (neither) |
| 8 | real-world | ThreadBound / Buffer | opt out with PhantomData, opt in with unsafe impl Send |
| 9 | capstone | SpinLock<T> | build a lock; unsafe impl<T: Send> Sync and why only Send |
The ideas, built up
1. Send is about moving
The first rung does nothing but move owned data across a boundary:
fn sum_on_thread(data: Vec<i64>) -> i64 {
thread::spawn(move || data.iter().sum::<i64>())
.join()
.unwrap()
}
Vec<i64> is Send — it owns its heap buffer with no shared aliasing, so handing
the whole thing to another thread transfers exclusive access. The move keyword
is load-bearing: it makes the closure own data rather than borrow it. A borrow
of a local can’t satisfy 'static, which previews rung 5’s wall.
2. Sync is about sharing — and it’s defined via Send
Rung 2 has several threads read the same data at once through shared references:
fn parallel_contains(haystack: &[i64], needles: &[i64]) -> Vec<bool> {
thread::scope(|s| {
let mut handles = Vec::with_capacity(needles.len());
for needle in needles {
handles.push(s.spawn(move || haystack.contains(needle)));
}
handles.into_iter().map(|h| h.join().unwrap()).collect()
})
}
Each closure captures &haystack (a &[i64]). For that shared reference to cross
into a thread, [i64] must be Sync. And here is the definition that runs the
whole topic:
T: Syncis defined as&T: Send.
“It’s safe to share &T across threads” is literally “it’s safe to send a &T to
another thread.” Sync isn’t a separate idea bolted on — it’s Send applied to
references. thread::scope is what lets borrows (not just 'static data) cross,
because the scope joins every thread before the borrowed data can die.
3. The traits are inferred from your fields
There is no derive. The compiler walks your type’s layout: a struct is Send iff
every field is Send, Sync iff every field is Sync. To observe a marker bound
you use a generic function whose only content is its bound:
fn assert_send<T: Send>() {}
fn assert_sync<T: Sync>() {}
If assert_send::<Foo>() compiles, then Foo: Send. If it doesn’t, you get a
precise error pointing at the offending type. These two empty functions are the
instrument the rest of the ladder runs on.
struct Telemetry { count: u64, label: String }
// Telemetry is Send + Sync — not by derive, but because u64 and String both are.
assert_send::<Telemetry>();
assert_sync::<Telemetry>();
Swap label to Rc<str> and assert_send::<Telemetry>() stops compiling — and the
error names the struct, not the field. One rotten apple.
4. Probing the standard library
Auto traits prove positives: a probe that compiles is proof. There is no stable
negative bound, so you witness negatives by uncommenting a probe that should
fail and reading the compiler’s prose (“Rc<i32> cannot be sent between threads
safely”). Predict first, then let the compiler grade you:
| type | Send | Sync | why |
|---|---|---|---|
i32, String, Box<i32> | yes | yes | owned, no shared aliasing |
&i32 | yes | yes | &T: Send because i32: Sync; &T: Sync because i32: Sync |
Rc<i32> | no | no | non-atomic refcount (rung 5) |
Arc<i32> | yes | yes | atomic refcount |
Cell<i32> | yes | no | interior mutation, unsynchronized |
RefCell<i32> | yes | no | non-atomic borrow flag |
Mutex<i32> | yes | yes | real lock provides synchronization |
*const i32 | no | no | compiler assumes nothing about a raw pointer |
The rows people get wrong are Cell/RefCell (they are Send) and the raw
pointer (it is neither). Keep reading.
5. Why Rc is !Send — the actual race
Rc::clone is, in essence, self.count += 1 on a plain integer. Arc::clone is
self.count.fetch_add(1, ...) — a single atomic read-modify-write. If two threads
could clone an Rc at once, their non-atomic increments would interleave and lose
updates. A refcount that reads too low frees memory that is still referenced:
use-after-free, then double-free.
You can’t share an Rc across threads (the compiler forbids it), so the ladder
reproduces the mechanism directly on a shared atomic, two ways:
// non-atomic style: load, then a SEPARATE store — mimics `Rc`'s `count += 1`
let v = c.load(Relaxed);
c.store(v + 1, Relaxed);
// atomic: one indivisible operation — mimics `Arc`'s clone
c.fetch_add(1, Relaxed);
Run it with 8 threads × 50,000 iterations and the atomic version is always exactly 400,000, while the racy version loses hundreds of thousands of updates:
atomic=400000 (exact, = Arc), racy=53462 lost 346538 updates
Translate that to Rc: 346,538 clones whose count never registered. That is the
corruption Rc: !Send makes impossible to even write.
The takeaway:
Send/Syncconvert a class of runtime data races into compile errors. The marker is a proof obligation; the auto-derive discharges it structurally.
6. The four quadrants
Send and Sync are independent axes, and every box is occupied:
Sync (can share &T) | !Sync (cannot share &T) | |
|---|---|---|
Send | i32, String, Mutex<T>, Arc<T> | Cell<T>, RefCell<T> |
!Send | MutexGuard<'_, T> | Rc<T>, *const T, *mut T |
The two that bend intuition:
Cell/RefCell:Send+!Sync. Moving the whole cell to one thread (exclusive ownership, one accessor) is fine. Sharing&Cellwould let two threads.set()concurrently with zero synchronization — a data race. Move ≠ share.MutexGuard:!Send+Sync. The canonical Sync-but-not-Send type. Many platforms require the locking thread to unlock, so the guard must not be moved to another thread (!Send). But lending&guard(which derefs to&T) out is fine whenT: Sync. This is also why holding astd::sync::MutexGuardacross an.awaitmakes a future!Send.
A corollary worth internalizing: &T: Send ⟺ T: Sync. So &Cell<i32> is not
Send even though Cell<i32> itself is Send — because Cell is !Sync.
7. The shared-mutable-state workhorse
The famous idiom is just composition of everything above:
Rc<RefCell<T>> (single-threaded) Arc<Mutex<T>> (multi-threaded)
Rc: !Send !Send Arc: Send Sync (atomic count)
RefCell: Send !Sync Mutex: Send Sync (real lock)
=> NEITHER Send nor Sync => Send + Sync
Going from one to the other is literally swapping non-atomic machinery for
atomic/locked machinery — and the marker traits flip as a consequence. The rung
forces std::thread::spawn (not scope), so the 'static bound requires Arc:
let accumulator = Arc::new(Mutex::new(0));
for chunk in values.chunks(chunk_len) {
let accumulator = Arc::clone(&accumulator); // same Mutex, new handle
let chunk = chunk.to_vec(); // own the data ('static)
handles.push(thread::spawn(move || {
let partial = chunk.into_iter().sum::<i64>();
*accumulator.lock().unwrap() += partial; // lock only to combine
}));
}
Note the discipline: each thread sums its chunk without the lock held, then takes the lock only to add its partial. Holding the lock while iterating would serialize the threads and defeat the parallelism.
8. Overriding the auto-derive — both directions
You can steer the inference instead of just accepting it.
Opt OUT (safe). A field whose type isn’t Send/Sync drags the whole type out.
The zero-cost, deliberate way is a PhantomData<*const ()> marker — a raw pointer
is !Send + !Sync, and PhantomData<T> makes your struct behave, for auto-trait
purposes, as if it owned a T, storing nothing:
struct ThreadBound {
id: u32,
_pd: PhantomData<*const ()>, // now !Send and !Sync, at zero runtime cost
}
This is how you build a handle that must never leave its thread (an FFI/thread-local context).
Opt IN (unsafe). A type holding a raw pointer is !Send by default — the
compiler won’t assume anything about it. If you know the access is sound, you
promise it:
struct Buffer { ptr: *mut u8, len: usize }
// SAFETY: Buffer uniquely owns the allocation described by ptr/len.
// Moving it to another thread transfers that ownership; no aliases are exposed,
// and Drop reconstructs and frees the allocation exactly once.
unsafe impl Send for Buffer {}
unsafe here means “compiler, I take responsibility for this invariant.” It is
exactly how Arc, Vec, Box, and channels get their Send/Sync impls.
Buffer deliberately does not impl Sync: moving it is sound (unique
ownership), but sharing &Buffer with an unsynchronized raw read is a different,
unproven claim.
The
// SAFETY:comment is not decoration. Stating the invariant is the work — it’s the audit discipline realunsafedemands. “Owned by this thread” is the wrong justification for aSendtype; the whole point is another thread owns it.
Footguns
- A green test can hide a wrong model. Forcing
unsafe impl Send + Synconto a type that you wanted to be thread-bound makes the code compile while lying to the compiler. The probe must match the intent: thread-bound types belong in the commented negative block, proven by failing to compile. Cell/RefCellareSend. Easy to assume “interior mutability = not thread-safe = neither trait.” Wrong: they’reSend(move is fine), only!Sync.&Cell<T>is!Sendeven thoughCell<T>isSend. The reference’s Send-ness follows the cell’s Sync-ness, not its Send-ness.MutexGuardacross.awaitmakes a future!Send, breakingtokio::spawn. Same root cause asMutexGuard: !Send.- Reading offset 0 of a possibly-empty buffer is UB.
Buffer::new(0)thenfirst()would read out of bounds; the fix is an explicitassert!(len > 0)before theunsaferead. Run unsafe rungs undercargo mirito catch this.
Real-world patterns
Arc<Mutex<T>>/Arc<RwLock<T>>— shared mutable state across threads, the default reach.Arc<T>(no lock) — shared immutable state; needs onlyT: Send + Sync.PhantomData<*const ()>— opt a handle out ofSend/Syncdeliberately.unsafe impl Send/Sync— how every concurrency primitive in std bridges from raw pointers /UnsafeCellback to safe, shareable types.Sendbound onspawnandtokio::spawn— the entire fearless-concurrency guarantee enters through this one bound.
Capstone insight — SpinLock<T>
Building a lock from scratch proves you own the model end to end:
struct SpinLock<T> {
locked: AtomicBool,
value: UnsafeCell<T>,
}
unsafe impl<T: Send> Send for SpinLock<T> {}
unsafe impl<T: Send> Sync for SpinLock<T> {} // <- the whole ladder, in one line
Two pieces matter:
UnsafeCell<T> is the only legal way to get &mut T from &self. Every
interior-mutability type — Cell, RefCell, Mutex, the atomics — is built on it.
A plain field behind &self can never yield &mut. UnsafeCell is also exactly
what makes a type !Sync by default, which is why you must opt back in.
The bound is T: Send, not T: Sync — and that is the entire topic.
The lock guarantees mutual exclusion: only one thread ever touches the
Tat a time. So the value is effectively handed between threads (Send), never simultaneously shared (which would need Sync). Two threads never hold&Tat once, soT: Syncis never required.
This is precisely the signature of std::sync::Mutex<T>: Sync where T: Send. The
lock/unlock use Acquire/Release ordering so that one holder’s writes are
visible to the next:
fn lock(&self) -> SpinGuard<'_, T> {
while self.locked
.compare_exchange(false, true, Acquire, Relaxed)
.is_err()
{
std::hint::spin_loop();
}
SpinGuard { lock: self }
}
impl<T> Drop for SpinGuard<'_, T> {
fn drop(&mut self) {
self.lock.locked.store(false, Release); // publish writes to next holder
}
}
Share one &SpinLock across eight scoped threads, each locking to increment, and
the total is exact — the lock serializes what rung 5 showed racing. (And it’s
Miri-clean.)
Explain it back
- Define
SendandSyncwithout using the other word, then state the one-line relationship between them. - Why is
Rc!SendbutArcSend? Describe the exact race, not just “it’s not thread-safe.” Cell<i32>isSendbut!Sync. Why is moving it fine but sharing&to it not?- Why is
MutexGuardSyncbut!Send? - Is
&Cell<i32>Send? Derive the answer fromT: Sync ⟺ &T: Send. - In
unsafe impl<T: Send> Sync for SpinLock<T>, why is the boundT: Sendand notT: Sync? - What does
UnsafeCellprovide that an ordinary field cannot, and why does a type containing one need an explicitunsafe impl Sync?
See also
- Threads & scoped threads — where
Send/Syncbounds first bite. Rc/Arc— the atomic-vs-non-atomic refcount this ladder dissects.Cell/RefCell— the interior-mutability types in the Send-but-!Sync quadrant.Rc<RefCell<T>>patterns — the single-threaded counterpart toArc<Mutex<T>>.
Mutex / RwLock
Ladder:
src/bin/mutex_rwlock.rs· Run:cargo run --bin mutex_rwlock· Phase 4 · 9 rungs
TL;DR
A Mutex<T> protects data, not code. The only way to reach the T is to
lock(), which hands you a MutexGuard — an RAII token that is &mut T
and unlocks when it drops. The borrow checker then enforces, at compile time,
that you can only touch the data while you hold the lock. RwLock<T> splits that
into many readers XOR one writer.
Everything hard about locks is one of three things:
- Guard lifetime — the lock is held for exactly as long as the guard is alive. Hold it too long and you serialize the program; hold it across a second lock in the wrong order and you deadlock.
- Poisoning — a panic while locked taints the lock so later
lock()calls returnErr, warning you the data may be half-updated. - Lock ordering — two locks taken in opposite orders by two threads form a cycle and hang forever (ABBA deadlock). The fix is a global acquisition order.
Why this exists (from first principles)
Shared mutable state across threads is the original sin of concurrency. If two
threads run counter += 1 at the same time, the operation is really read,
add, write — and the two reads can both see the old value, so one increment is
lost. This is a data race, and in most languages it is silent corruption or
undefined behavior.
Rust makes data races a compile error. The rule: you may have many &T
(shared, read-only) or one &mut T (exclusive), never both. But a counter
shared by 8 threads needs all of them to write. How do you get &mut from a
shared &?
A Mutex<T> is the answer: it provides interior mutability guarded at
runtime. You only ever hold a shared &Mutex<T>, but lock() returns a guard
that derefs to &mut T. The mutex guarantees that at most one guard exists at a
time, so the &mut it hands out is genuinely exclusive — the borrow rule holds,
just enforced by a runtime lock instead of the compiler.
The mental shift: a
Mutexdoesn’t make a region of code atomic. It makes access to a piece of data exclusive. The “critical section” is exactly the span where the guard is alive.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | foundations | Mutex basics | lock().unwrap() → guard → *guard += by; let mut guard for DerefMut |
| 2 | foundations | Arc<Mutex> counter | share one mutex across N threads; hold the lock across read-modify-write |
| 3 | mechanics | Guard lifetime | snapshot + drop(guard) to shrink the critical section |
| 4 | mechanics | RwLock | many readers XOR one writer; read() is &T, write() is &mut T |
| 5 | footgun | Poisoning | panic-while-locked poisons; recover via into_inner() |
| 6 | footgun | Non-reentrancy | std Mutex isn’t recursive; double-lock self-deadlocks |
| 7 | footgun | Lock-ordering ABBA | induce a deadlock, fix with a canonical lock order |
| 8 | real-world | Mutex + Condvar | bounded blocking queue; wait() in a while loop |
| 9 | capstone | Concurrent Bank | deadlock-free transfers + poison recovery under a thread storm |
The ideas, built up
1. The guard is the lock (rungs 1–2)
fn bump(m: &Mutex<i32>, by: i32) {
let mut guard = m.lock().unwrap(); // guard: MutexGuard<i32>
*guard += by; // DerefMut → &mut i32
} // guard drops here → unlock
Three things to notice:
&Mutex, not&mut Mutex. The mutex hands out mutability through a shared reference. That is what lets anArc<Mutex<T>>(which only ever gives you&) still mutate.let mut guard. The mutex turned a shared&into mutable access, but the “mut-ness” has to reappear somewhere — it reappears on the guard binding, because*guard += bygoes throughDerefMut, which needsmut.- Unlock is
Drop. There is nounlock()method. The lock is released when the guard goes out of scope. This is the single most important fact about locks in Rust, and rung 3 is entirely about controlling when that happens.
To share across threads, wrap in Arc and clone one handle per thread:
let counter = Arc::new(Mutex::new(0));
for _ in 0..n_threads {
let c = Arc::clone(&counter); // each thread gets its own handle
s.spawn(move || {
for _ in 0..per_thread { bump(&c, 1); }
});
}
Why both
ArcandMutex? They are orthogonal.Arcanswers “who owns it?” (shared ownership, so the data lives as long as any thread needs it).Mutexanswers “who can touch it right now?” (exclusive access). You need both:Arcto share the handle,Mutexto coordinate the mutation.
The assertion 8 * 1000 == 8000 is the data-race detector. The lock must be held
across the whole read-modify-write. If you ever did read, unlock, +1, lock, write, two threads could read the same value and one update would be lost — and
the total would land below 8000.
Note on
Arcvsscope: rung 2 usesthread::scope, which also lets threads borrow locals (the scope guarantees they join first), so theArcis technically redundant there. With plainthread::spawn(which requires'static), theArcis load-bearing — each thread genuinely needs its own owning handle.
2. Guard lifetime = critical section length (rung 3)
Because the guard holds the lock until it drops, holding it across slow work serializes every other thread behind you. The fix is to shrink the critical section: grab what you need, release, then do the slow part unlocked.
fn slow_sum(data: &Mutex<Vec<i32>>, expensive: impl Fn(i32) -> i32) -> i32 {
let guard = data.lock().unwrap();
let snapshot = guard.clone(); // copy the data out
drop(guard); // release the lock BEFORE the slow work
snapshot.iter().map(|x| expensive(*x)).sum()
}
The lock is held for microseconds (one clone) instead of for the entire
expensive pass. The ladder enforces this: the expensive closure tries to
try_lock() the same mutex and panics if it can’t — so holding the guard across
the loop fails the test.
Two tools to release early, equivalent in effect:
drop(guard); // explicit
{ let g = m.lock()...; /* use g */ } // inner scope: g drops at the brace
Real-world echo: this is why production code clones out of the lock, or computes a new value and then takes a brief lock to store it, rather than holding a mutex across I/O or heavy CPU.
3. RwLock — split the lock when reads dominate (rung 4)
A Mutex gives exclusive access even to readers — two threads that only want
to read still serialize. When reads vastly outnumber writes (config, caches,
routing tables), that is wasted parallelism. RwLock<T> splits the lock:
| Method | Guard | Access | Concurrency |
|---|---|---|---|
read() | RwLockReadGuard | &T | many at once |
write() | RwLockWriteGuard | &mut T | one, blocks all readers |
fn reader_sum(rw: &RwLock<Vec<i32>>) -> i32 {
let guard = rw.read().unwrap(); // &Vec<i32> — shared
guard.iter().sum()
}
fn writer_push(rw: &RwLock<Vec<i32>>, v: i32) {
let mut guard = rw.write().unwrap(); // &mut Vec<i32> — exclusive
guard.push(v);
}
The asymmetry mirrors the borrow rules exactly: read() needs only let guard
(it’s &T), write() needs let mut guard (it’s &mut T). The rung proves the
sharing is real — 4 reader threads all hold read guards simultaneously, and the
max-overlap counter reaches 4. With a Mutex it would never exceed 1.
Caveat — writer starvation: std’s
RwLockgives no fairness guarantee. On some platforms a steady stream of readers can starve a waiting writer. That is why “read-heavy” is the rule of thumb; under write pressure a plainMutexcan actually be faster and fairer.
4. Poisoning — the lock as a tripwire (rung 5)
Now the question rung 1 deferred: why does lock() return a Result?
If a thread panics while holding the guard, the data might be half-updated — an
invariant could be broken mid-mutation. Rust records this: the mutex becomes
poisoned, and every later lock() returns Err(PoisonError).
// A thread dies mid-mutation:
let mut g = m.lock().unwrap();
*g = 999;
panic!("boom"); // guard's Drop runs during unwind → mutex is now poisoned
After that, a plain .lock().unwrap() would itself panic. To keep going, recover
the guard out of the error:
fn recover(m: &Mutex<i32>) -> i32 {
let guard = m.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
*guard
}
PoisonError::into_inner() hands you the guard anyway. Poisoning is advisory,
not a wall: it says “the invariant might be broken,” and you decide whether the
data is still usable. In rung 5 the 999 was fully written before the panic, so
the data is fine and recovery is correct.
Poisoning is contested.
parking_lot::Mutexandtokio::sync::Mutexdon’t poison at all — they decided the ergonomic tax wasn’t worth it. So.unwrap()on a std lock is really an assertion that no holder ever panics; code that must survive panics handles thePoisonError. (Recent Rust also addsMutex::clear_poison().)
5. Non-reentrancy — the single-thread deadlock (rung 6)
Unlike some languages, std’s Mutex is not recursive. If a thread holds the
guard and calls lock() on the same mutex again, it blocks forever waiting for
itself.
fn would_self_deadlock(m: &Mutex<i32>) -> bool {
let _guard = m.lock().unwrap(); // hold it
m.try_lock().is_err() // second attempt → Err(WouldBlock)
}
This isn’t an oversight — it’s required for soundness. A reentrant lock would
hand you a second &mut to data you already hold a &mut to, which is
aliasing UB. Non-reentrancy is what keeps the guard’s &mut exclusive.
The rung proves the deadlock without hanging by using try_lock(), which returns
immediately (Err(WouldBlock)) instead of blocking. The lesson: a real lock()
on line 2 would freeze the program invisibly — no panic, no error, just a
hung thread. try_lock turns an unobservable hang into a returnable Err, which
is also the real tool when you genuinely might re-enter: detect and back off
instead of wedging.
6. Lock ordering — the ABBA deadlock (rung 7)
The classic multi-lock deadlock. Two accounts, each behind its own mutex:
Thread 1 (A→B): lock A, then lock B
Thread 2 (B→A): lock B, then lock A
T1 holds A, waiting for B ┐
T2 holds B, waiting for A ┘ → cycle → neither proceeds → hang
This is ABBA: a cycle in the “who-waits-for-whom” graph. The fix is a global
lock order. If every thread always acquires locks in the same order, no cycle
can form. Here, order by ascending account id:
fn transfer_ordered(from: &Account, to: &Account, amt: i64) {
if from.id < to.id {
let mut fg = from.balance.lock().unwrap(); // lower id first
let mut tg = to.balance.lock().unwrap();
*fg -= amt; *tg += amt;
} else {
let mut tg = to.balance.lock().unwrap(); // lower id first
let mut fg = from.balance.lock().unwrap();
*fg -= amt; *tg += amt; // still from→to
}
}
The trick is to keep two concerns independent:
- Acquisition order is always lower-id-first (deadlock avoidance).
- Mutation still subtracts from
from, adds toto(correctness).
You bind the guards to the right roles in each branch, but lock in the canonical order. The harness runs 100k transfers each way at once — the exact ABBA setup — and a 5-second watchdog catches a wrong ordering instead of hanging your terminal.
When there’s no natural
id, order by the mutex’s memory address (std::ptr::from_ref(m) as usize) or any stable total order. The content of the order doesn’t matter — only that every thread agrees on it.Deeper point: deadlock-freedom is a property of the whole system, not one function.
transfer_orderedis safe only because every caller obeys the same order. One rogue lock-in-argument-order site reintroduces the cycle.
7. Condvar — waiting for a condition (rung 8)
A Mutex lets you read shared state safely, but it can’t make you wait for
that state to become a certain way. Busy-looping while q.lock().is_empty() {}
burns a core. A Condvar (condition variable) is a parking lot tied to a
mutex: a thread can sleep until another thread notifies it.
The one method that matters:
guard = self.cv.wait(guard).unwrap();
It atomically (a) unlocks the mutex and parks the thread, then (b) on wakeup re-locks the mutex and returns the guard. The atomic unlock-and-sleep is the whole point: it closes the race where you check the condition, then sleep, and miss a notify that lands in between.
fn push(&self, v: T) {
let mut guard = self.inner.lock().unwrap();
while guard.len() == self.cap { // WHILE, not if
guard = self.cv.wait(guard).unwrap();
}
guard.push_back(v);
self.cv.notify_all(); // notify AFTER mutating
}
fn pop(&self) -> T {
let mut guard = self.inner.lock().unwrap();
while guard.is_empty() {
guard = self.cv.wait(guard).unwrap();
}
let item = guard.pop_front().unwrap();
self.cv.notify_all();
item
}
Two rules that define correct Condvar use:
- Wait in a
while, never anif. Afterwait()returns you only know you were woken, not that the condition holds. Spurious wakeups happen, and with one shared condvar anotify_allwakes every parked thread — including other poppers. If two poppers wake on one item, thewhilemakes the loser re-checkis_empty()and go back to sleep instead of callingpop_front().unwrap()on an empty deque and panicking. The loop is what makes a shared condvar safe. - Notify after you mutate, so a parked thread wakes to re-test its predicate.
notify_onevsnotify_all:notify_oneis cheaper but only safe when any single waiter can make progress on the event. With mixed waiter kinds on one condvar (pushers + poppers),notify_onecan wake the wrong kind and stall;notify_allis the safe default. The throughput fix is two condvars (not_full,not_empty) so you only wake the relevant side — which is exactly howstd::sync::mpscand most bounded channels are built.
Footguns
| Trap | What bites | Fix |
|---|---|---|
| Holding the guard too long | every other thread serializes behind you | snapshot + drop(guard) before slow work (rung 3) |
.lock().unwrap() everywhere | a panicked holder poisons the lock → all later locks panic | recover via unwrap_or_else(|e| e.into_inner()) (rung 5) |
| Re-locking the same mutex in one thread | self-deadlock, hangs silently | don’t; std Mutex isn’t reentrant. Use try_lock to detect (rung 6) |
| Two locks in opposite orders | ABBA deadlock under concurrency | canonical lock order (lower id / address first) (rung 7) |
if cond { cv.wait() } | spurious wakeup or a raced predicate → act on a false condition | always while cond { cv.wait() } (rung 8) |
Forgetting notify after mutating | waiters sleep forever | notify_all() after every state change |
| Same-account transfer | from.lock(); to.lock(); is a double-lock = rung 6 | reject from == to up front (rung 9) |
Real-world patterns
Arc<Mutex<T>>is the canonical “shared mutable state across threads” handle.Arcfor shared ownership,Mutexfor exclusive access — orthogonal, both needed.- Fine-grained locking. One
Mutexper item (Vec<Mutex<i64>>in the capstone) lets disjoint operations run in parallel, unlike one coarseMutex<Vec<_>>that serializes everything. RwLockfor read-heavy state — config snapshots, caches, routing tables — with the writer-starvation caveat in mind.Mutex + Condvaris the primitive under channels, thread pools, and producer/consumer pipelines.std::sync::mpscis essentially this.parking_lotoffers faster, smaller, non-poisoningMutex/RwLockand is a common drop-in in production crates.
Capstone insight (rung 9)
The Bank fuses the entire ladder into one stress test: 8 threads × 50,000
random transfers against a bank with a deliberately poisoned account, an
8-second deadlock watchdog, and one invariant — money is conserved.
fn lock_recover(m: &Mutex<i64>) -> MutexGuard<'_, i64> {
match m.lock() { Ok(g) => g, Err(e) => e.into_inner() } // rung 5
}
fn transfer(&self, from: usize, to: usize, amt: i64) -> Result<(), TransferError> {
if from == to { return Err(TransferError::SameAccount); } // rung 6
if from >= self.accounts.len() || to >= self.accounts.len() {
return Err(TransferError::NoSuchAccount);
}
if from < to { // rung 7
let mut fg = Self::lock_recover(&self.accounts[from]); // lower index first
let mut tg = Self::lock_recover(&self.accounts[to]);
if *fg < amt { return Err(InsufficientFunds { have: *fg, need: amt }); }
*fg -= amt; *tg += amt; Ok(()) // rung 3
} else {
let mut tg = Self::lock_recover(&self.accounts[to]);
let mut fg = Self::lock_recover(&self.accounts[from]);
if *fg < amt { return Err(InsufficientFunds { have: *fg, need: amt }); }
*fg -= amt; *tg += amt; Ok(())
}
}
The structural “aha”: each safety property is an independent line of defense, and they compose. Three distinct failure modes, three distinct guarantees:
| If you… | You get… | The defense |
|---|---|---|
.unwrap() the poisoned account | a worker panics | lock_recover (poison recovery) |
| lock in inconsistent order | the bank hangs (watchdog fires) | lower-index-first (lock ordering) |
| check funds after mutating, or overflow | money created/destroyed | check-before-mutate, hold both guards |
Money is conserved (8000 → 8000) only because all three hold at once. The proof is in the numbers: ~340k transfers applied, ~58k denied for insufficient funds, the poisoned account survived, total unchanged. Correctness under concurrency is not one clever trick — it’s several disciplines layered, each closing one hole.
Explain it back
- Why does a
Mutexlet you mutate through a shared&, and where does the “mut” reappear? - There is no
unlock(). When exactly is the lock released, and how do you release it early? - Why does
lock()return aResult? What doesinto_inner()recover, and why is poisoning “advisory”? - Why is std’s
Mutexnon-reentrant, and why is that required for soundness rather than a limitation? - Draw the ABBA cycle. What single rule breaks it, and why must every caller obey it?
- Why must
cv.wait()live in awhileloop and never anif? Give two distinct reasons. - In the capstone, name the three independent defenses and the failure each one prevents.
See also
- Threads & scoped threads —
spawn,join,thread::scope, the'staticwall theArcclone works around. Send&Syncdeeply — whyArc<Mutex<T>>isSend + SyncandRc<RefCell<T>>is neither; the hand-rolledSpinLock.Cell/RefCell— interior mutability in a single thread;RefCell’s runtime borrow check is the non-atomic cousin of aMutex.
Channels
Ladder:
src/bin/channels.rs· Run:cargo run --bin channels· Phase 4 · 9 rungs
TL;DR
A channel is a typed pipe between threads: a Sender<T> end and a Receiver<T>
end. Instead of sharing memory behind a lock and coordinating who touches what when,
you move ownership through the pipe — send(value) gives the value away, recv()
takes it on the other side. Two facts carry the whole topic:
- Ownership moves through the pipe.
send(v)transfersvout of the sending thread. No aliasing, no lock at the call site — the type system already proved only one thread owns it. - The channel closes itself. When every
Senderdrops, theReceiverobserves “disconnected” and stops. That is how loops terminate cleanly — no sentinel value, no message count.
Almost every channel bug is a violation of fact 2: a Sender you forgot to drop, so
the receiver waits forever.
Why this exists (from first principles)
Shared-state concurrency (Arc<Mutex<T>>) answers “how do many threads safely touch
one piece of data?” Channels answer a different question: “how do threads hand work
to each other?” The distinction matters because shared state forces every participant
to agree on a locking protocol, and protocols are where deadlocks live.
A channel removes the protocol. The buffer is owned by the channel, not by any thread.
A producer’s only verb is send; a consumer’s only verb is recv. There is no “lock
this, then that” ordering to get wrong, because there is only ever one operation per
side. The classic slogan:
Do not communicate by sharing memory; instead, share memory by communicating.
Rust enforces the safety of this at the type level. Sender<T> and Receiver<T> are
Send only when T: Send — you can ship the ends to other threads precisely because
the values that flow through are themselves safe to move between threads. The “move
ownership through the pipe” model isn’t a convention; it’s what the borrow checker
already guarantees.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | Foundations | First pipe | mpsc::channel(), move the sender into a thread, recv() |
| 2 | Foundations | Multi-producer | tx.clone() — the “m” in mpsc; fan in from N threads |
| 3 | Mechanics | Receiver as iterator | for v in rx ends on disconnect; you must drop(tx) |
| 4 | Mechanics | Bounded & backpressure | sync_channel(k); send blocks when full; (0) = rendezvous |
| 5 | Footgun | The hang | stray Sender ⇒ recv blocks forever; RecvError vs SendError(v) |
| 6 | Footgun | Non-blocking | try_recv: split Empty (keep polling) from Disconnected (stop) |
| 7 | Real-world | Worker pool | Arc<Mutex<Receiver>> shared job queue + a results channel |
| 8 | Real-world | crossbeam | Receiver: Clone mpmc, and select! over multiple channels |
| 9 | Capstone | Build it | hand-rolled Channel<T> from Mutex + Condvar + VecDeque |
The ideas, built up
1. The pipe and the move
let (tx, rx) = mpsc::channel();
thread::spawn(move || {
for i in 1..=5 {
tx.send(i).unwrap();
}
});
let mut result = Vec::new();
for _ in 0..5 {
result.push(rx.recv().unwrap());
}
Two things to notice. First, move on the closure is mandatory: without it the closure
would only borrow tx, and that borrow would have to outlive the local tx in the
parent — a lifetime error. move hands ownership of tx into the thread.
Second, the signatures tell the whole story:
tx.send(value) -> Result<(), SendError<T>> // moves value in; Err if receiver gone
rx.recv() -> Result<T, RecvError> // blocks until a value arrives or all senders drop
send consumes value. After tx.send(i), the sending thread no longer owns i.
recv blocks — it parks the thread until something is in the buffer. A single sender
also preserves order, which is why recv’ing five times yields 1,2,3,4,5.
2. Many senders, one receiver (the “m” in mpsc)
mpsc = multi-producer, single-consumer. You get extra producers by cloning the
Sender. Every clone feeds the same Receiver.
let (tx, rx) = mpsc::channel();
for i in 0..n {
let tx = tx.clone(); // each thread gets its OWN handle
thread::spawn(move || {
tx.send(i * 10).unwrap();
});
}
The clone must happen inside the loop. If you tried to move the single tx into
the closure, it would be consumed on the first iteration and gone on the second. Cloning
first gives each thread a private handle while the original tx stays in the parent.
Order across threads is now nondeterministic (the OS schedules them however it likes), so the rung sums the results rather than asserting a sequence. The single receiver is the serialization point: whatever interleaving the senders produce, the consumer sees a well-defined stream of values one at a time.
3. Disconnect is the shutdown signal
The receiver is an iterator. for v in rx yields values until the channel is
disconnected — meaning every Sender (original and all clones) has dropped:
for i in 0..n {
let tx = tx.clone();
thread::spawn(move || { tx.send(i as i64).unwrap(); });
}
drop(tx); // <-- the linchpin
let mut result = 0;
for _ in rx { // ends by itself once every sender is gone
result += 1;
}
The drop(tx) is the entire lesson. After the loop, n clones live in threads (each
drops when its thread finishes), but the original tx is still held by main. The
iterator only ends when the sender count reaches zero. Leave the original alive and
for v in rx waits forever for a value that will never come.
Rule of thumb: the number of live senders is a reference count. The receiver’s loop terminates exactly when that count hits zero. Every
Senderyou hold is a promise “more might come” — drop the promise when it’s no longer true.
4. Bounded channels and backpressure
mpsc::channel() is unbounded: send never blocks, it just appends to the queue. A
fast producer feeding a slow consumer grows that queue without limit — a memory leak in
slow motion.
mpsc::sync_channel(k) is bounded to k buffered messages. When the buffer is full,
send blocks until the consumer frees a slot. That blocking is backpressure: the
producer is forced down to the consumer’s pace.
let (tx, rx) = mpsc::sync_channel(0); // capacity 0 = rendezvous
thread::spawn(move || {
tx.send("a").unwrap(); // blocks until a recv() is ready to take it
tx.send("b").unwrap();
tx.send("c").unwrap();
});
sync_channel(0) is the extreme case: a rendezvous channel with zero buffer. Every
send blocks until a recv is simultaneously ready — the value is handed across
thread-to-thread with no storage in between. send("b") literally cannot return until
someone has recv’d "a".
Testing note from the ladder: the rung’s assertion passes even with an unbounded
channel(), because it only inspects the consumer-side order. To actually witness backpressure you have to record the producer’s progress (push to a sharedArc<Mutex<Vec<_>>>right after eachsendreturns) and observe that with capacity 0 the producer can never get more than one value ahead of the consumer. A green test does not always prove the property you care about.
5. The edges of a channel’s life
When one half is gone, two symmetric errors report it:
rx.recv() -> Err(RecvError) // buffer empty AND every Sender dropped — nothing more can arrive
tx.send(v) -> Err(SendError(v)) // the Receiver dropped — nobody will take v, so it's handed BACK
RecvError is what ends for v in rx. You can also handle it explicitly:
let mut result = Vec::new();
while let Ok(value) = rx.recv() { // exits on Err(RecvError)
result.push(value);
}
SendError is the mirror image, and it carries the value with it. Since nobody can ever
receive v, send gives it back so you can do something else with it:
let (tx, rx) = mpsc::channel();
drop(rx);
let recovered = tx.send(99).unwrap_err().0; // SendError is a tuple struct; .0 is the value
assert_eq!(recovered, 99);
The footgun lives in the gap between these two errors: if a Sender never drops,
recv on an empty channel blocks forever. No RecvError is ever produced because, as
far as the channel knows, more values might still come. The infinite hang and the clean
RecvError are the same mechanism viewed from two sides — sender count zero vs not.
6. Receiving without blocking
recv() blocks, which is wrong for an event loop that must also do other work, or a
consumer with a deadline. try_recv() never blocks and returns a richer error:
loop {
match rx.try_recv() {
Ok(value) => result.push(value),
Err(TryRecvError::Empty) => thread::sleep(Duration::from_millis(100)), // keep polling
Err(TryRecvError::Disconnected) => break, // truly done
}
}
The two TryRecvError variants are the heart of the rung and must be handled
separately:
| Variant | Meaning | Correct response |
|---|---|---|
Empty | nothing right now, but senders are alive | back off and try again |
Disconnected | empty and all senders dropped | stop |
Collapse them and you get a bug either way: treat Empty as “done” and you quit early,
losing every later message; treat Disconnected as “try again” and you busy-spin
forever. A correct non-blocking drain must branch on both. (recv_timeout(dur) is the
middle ground: block up to a deadline, then return Timeout.)
7. The worker pool — channels as architecture
A fixed pool of N workers draining a shared job queue, with results flowing back over a
second channel. Two channels, two directions:
- jobs:
main --(many)--> workers(fan-out) - results:
workers --(many)--> main(fan-in)
The wall you hit: Receiver is not Clone (mpsc = single consumer). N workers can’t
each own the receiving end. The classic std thread-pool fix is to wrap it:
let (job_tx, job_rx) = mpsc::channel();
let (res_tx, res_rx) = mpsc::channel();
let job_rx = Arc::new(Mutex::new(job_rx)); // share one receiver behind a lock
for _ in 0..n_workers {
let job_rx = Arc::clone(&job_rx);
let res_tx = res_tx.clone();
thread::spawn(move || {
loop {
let job = { // lock held ONLY across recv
let job_rx = job_rx.lock().unwrap();
job_rx.recv()
};
match job {
Ok(x) => res_tx.send(x * x).unwrap(),
Err(_) => break, // job senders all dropped -> exit
}
}
});
}
for input in inputs { job_tx.send(input).unwrap(); }
drop(job_tx); // so workers see disconnect and exit
drop(res_tx); // so the result drain terminates
let mut results: Vec<i64> = res_rx.into_iter().collect();
results.sort();
Two subtleties decide whether this is correct and fast:
- Lock scope. The inner
{ ... }block releases the mutex before computingx * x. Hold the lock across the work and your N workers degrade to running one-at-a-time — the single most common mistake in hand-rolled pools. - Two independent drops.
drop(job_tx)lets workers see disconnect and stop;drop(res_tx)lets the result drain see disconnect and finish. These are two separate disconnect chains — rung 3’s and rung 5’s lessons resurfacing. Keep either original alive and you hang.
This is what threadpool and the work-distribution core of rayon look like underneath
(plus a vector of JoinHandles to join on shutdown).
8. crossbeam — what std channels structurally can’t do
std::sync::mpsc is single-consumer by design. crossbeam-channel lifts two limits.
True MPMC: the Receiver is Clone. Multiple consumers, no Arc<Mutex> wrapper.
The same worker pool collapses to:
use crossbeam_channel::{select, unbounded};
let (job_tx, job_rx) = unbounded();
for _ in 0..n_workers {
let job_rx = job_rx.clone(); // clone the RECEIVER itself
let res_tx = res_tx.clone();
thread::spawn(move || {
for job in job_rx { // shared iterator; ends on disconnect
res_tx.send(job * job).unwrap();
}
});
}
No mutex, no manual lock()/recv() dance, no inner block to scope the guard. The
workers share the iterator because the receiver is Clone + Sync.
select!: wait on several channels at once. std has no way to block on two receivers
simultaneously; crossbeam’s select! blocks until any arm is ready, then runs the
first one that fires:
while open_a && open_b {
select! {
recv(rx_a) -> msg => match msg {
Ok(value) => out.push(value),
Err(_) => open_a = false,
},
recv(rx_b) -> msg => match msg {
Ok(value) => out.push(value),
Err(_) => open_b = false,
},
}
}
if open_a { out.extend(rx_a); } // one closed -> drain the survivor to exhaustion
if open_b { out.extend(rx_b); }
The subtle correctness point: once a channel disconnects, recv on it returns Err
immediately and forever, so select! would keep picking the dead channel and busy-spin.
The fix here is to loop only while open_a && open_b, then the instant either closes,
fall out and out.extend(rx_other) — which consumes the surviving receiver as an iterator
until its senders drop. Guaranteed to terminate, no spin. This is exactly how you’d
merge a data stream against a shutdown signal in real code.
Footguns
| Trap | What bites | The fix |
|---|---|---|
Stray Sender | for v in rx / recv() blocks forever — no RecvError ever fires | drop(tx) the original after spawning producers |
move a single tx into a loop | consumed on iteration 1, won’t compile on iteration 2 | let tx = tx.clone() inside the loop |
| Unbounded channel, slow consumer | queue grows without limit (memory blowup) | sync_channel(k) for backpressure |
Collapsing try_recv errors | quit early on Empty, or spin forever on Disconnected | branch on both variants explicitly |
| Holding the job lock while computing | N workers serialize into one | scope the guard to just the recv, release before work |
Keeping the original res_tx alive | result drain never sees disconnect → hang | drop(res_tx) before draining |
select! on a disconnected channel | dead arm fires instantly, busy-spins | stop selecting it; drain the survivor with a plain for |
Real-world patterns
- Fan-out / fan-in worker pools are the bread and butter: one job channel out, one
result channel back.
threadpool, and the task-distribution layer ofrayon, are this pattern industrialized. Arc<Mutex<Receiver>>is the idiomatic way to give std’s single-consumer receiver to many workers when you don’t want a crossbeam dependency.crossbeam-channelis the go-to when you need real MPMC orselect!. It’s also faster than std’smpscand is what many production systems reach for.select!for shutdown — merge a work channel with a “stop” channel so a worker can be told to quit between jobs. The same shape asmerge_two.- Async mirrors this exactly:
tokio::sync::mpscis the same model with.awaitinstead of blocking, andtokio::select!is the async sibling of crossbeam’sselect!. Learn the threaded version and the async one is a renaming.
Capstone insight
Rung 9 rebuilds a blocking mpsc channel from three safe primitives, and the payoff is
seeing that the “magic” of recv is just a condition variable:
struct Shared<T> { items: VecDeque<T>, senders: usize } // buffer + live-sender count
struct Inner<T> { queue: Mutex<Shared<T>>, available: Condvar }
The receiver doesn’t busy-wait; it sleeps on the Condvar and is woken by whoever
changes the state it cares about:
fn recv(&self) -> Result<T, Disconnected> {
let mut shared = self.inner.queue.lock().unwrap();
loop {
if let Some(item) = shared.items.pop_front() { // 1. value ready -> take it
return Ok(item);
}
if shared.senders == 0 { // 2. drained AND no senders -> done
return Err(Disconnected);
}
shared = self.inner.available.wait(shared).unwrap(); // 3. sleep; wait() unlocks+parks
}
}
Three details make this correct, and each maps onto a behavior you used as a black box:
- Check
pop_frontbeforesenders == 0. If the last sender drops while items remain, the receiver must drain them first, and only then reportDisconnected. Reverse the two checks and you silently lose buffered messages on shutdown — this is precisely theRecvErrorsemantics from rung 5: empty AND disconnected, in that order. waitin aloop, not anif.Condvar::waitcan return spuriously (woken with no real change). Re-checking the predicate in a loop absorbs that. The loop body is the “while the thing I want isn’t true, keep sleeping” pattern.- Every state change a receiver waits on is followed by a notify.
senddoespush_backthennotify_one. The lastSender::dropdoessenders -= 1thennotify_all— that final notify is the entire disconnect mechanism: it wakes a parked receiver so it can re-check, see zero senders, and returnErrinstead of sleeping forever.
fn send(&self, value: T) {
{ let mut shared = self.inner.queue.lock().unwrap(); shared.items.push_back(value); }
self.inner.available.notify_one(); // notify AFTER releasing the lock
}
impl<T> Drop for MySender<T> {
fn drop(&mut self) {
if self.update_senders(-1) == 0 { // sender count is a manual refcount
self.inner.available.notify_all(); // wake the receiver to see disconnect
}
}
}
Notifying after unlocking is the polite habit: the woken receiver won’t immediately
re-block on a mutex you’re still holding. And the senders field is a hand-rolled
reference count — Clone increments it, Drop decrements it, and the receiver’s
termination condition is “count reached zero.” That is the same bookkeeping std’s real
mpsc does, minus the lock-free fast paths. Once you’ve written this, for v in rx
ending on disconnect is no longer magic; it’s a usize reaching 0 and a notify_all.
Explain it back
Future-you should be able to answer these cold:
- Why does
for v in rxsometimes hang forever, and what one line fixes it? - Why must you
tx.clone()inside the spawn loop instead of moving onetxin? - What does
SendError(v)carry thatRecvErrordoesn’t, and why? - In a non-blocking drain, what goes wrong if you treat
TryRecvError::Emptyas “done”? As “the channel is broken, stop”? - Why must the worker-pool lock be released before the worker does its computation?
- Why does the std worker pool need
Arc<Mutex<Receiver>>but the crossbeam version doesn’t? - In
merge_two, why would a naiveselect!busy-spin once one channel closes? - In the capstone
recv, why is the order of the two checks (pop_frontthensenders == 0) load-bearing? Why mustwaitsit inside aloop? - What is the single invariant that, if violated anywhere, makes the hand-rolled receiver sleep forever?
See also
Mutex/RwLock— the lock andCondvarthe capstone is built on.- Threads & scoped threads —
spawn,move,join; what channels connect. Send&Syncdeeply — why the channel ends areSendiffT: Send.Rc/Arc— theArcthat lets both channel ends share oneInner.
Data parallelism with rayon
Ladder:
src/bin/rayon_parallel.rs· Run:cargo run --bin rayon_parallel(add--releasefor honest timings) · Phase 4 · 9 rungs
TL;DR
Rayon turns a sequential iterator chain into a parallel one over a thread pool:
where you wrote .iter(), write .par_iter() and the work spreads across cores.
The engine underneath is work-stealing fork-join — every worker thread owns a
task deque, and an idle worker steals tasks from a busy one, so uneven work
still balances itself. The whole library is built from one primitive,
rayon::join(a, b), which runs two closures potentially in parallel.
Two lessons separate “I sprinkled par_ everywhere” from actually understanding it:
- Parallelism has overhead. It loses on small or cheap or memory-bound work.
par_iterpays off only whentotal_work / coresclearly exceeds rayon’s ~hundreds-of-microseconds setup cost. reduce/foldneed an associative operation. Partial results recombine in an unspecified tree shape, so a non-associative op (subtraction, float+) gives a different, non-deterministic answer every run.
The type system you already drilled (Send/Sync) is what makes all of this
sound: data races become compile errors, not crashes.
Why this exists (from first principles)
You have a million items and N cores. You want to use all N. The naive plan — “spawn N threads, give each a chunk” — has three problems:
'staticand ownership.std::thread::spawnneeds'staticclosures, so borrowing a local slice across threads doesn’t compile without scoped threads.- Load imbalance. Equal-sized chunks are not equal work. If chunk 3 happens to contain all the expensive items, three cores finish early and idle while one grinds. Static partitioning wastes exactly the parallelism you wanted.
- Boilerplate. Handles, joins, chunk math, result reassembly — every time.
Rayon answers all three. It runs on a thread pool sized to your core count
(created lazily on first use), so there are no per-call thread spawns. It splits
work recursively and dynamically, and its work-stealing scheduler means a
core that runs dry grabs pending work from a busy core — load balances itself, no
matter how lumpy the per-item cost. And it exposes all of it as a drop-in
parallel Iterator.
What keeps it safe is the same thing that keeps std::thread::scope safe:
closures handed to the pool must satisfy Send/Sync, so the compiler rejects
any sharing that would be a data race. Parallel bugs that are runtime disasters
in C++ are type errors here.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | foundations | par_iter first contact | .iter().sum() → .par_iter().sum(); same answer |
| 2 | foundations | adapter zoo | map/filter/collect; collect preserves input order |
| 3 | mechanics | reduce & fold | identity closure; fold-then-reduce (local acc → combine) |
| 4 | mechanics | rayon::join | the fork-join primitive par_iter is built on |
| 5 | footgun | when parallelism loses | measure the overhead; find the break-even |
| 6 | footgun | non-associative reduce | subtraction ⇒ a different answer every run |
| 7 | footgun | the shared-state wall | for_each push won’t compile; collect vs Mutex |
| 8 | real-world | par_sort & par_bridge | parallel sort; adapt any sequential Iterator |
| 9 | capstone | hand-rolled fork-join | parallel_map + parallel quicksort from join |
The ideas, built up
1–2. par_iter is the iterator you know, parallelized
The entire entry point is one import and one method swap:
use rayon::prelude::*;
fn parallel_sum(data: &[u64]) -> u64 {
data.par_iter().sum() // was: data.iter().sum()
}
use rayon::prelude::* brings the par_iter() method and the ParallelIterator
adapters (map, filter, reduce, collect, …) into scope. The chain reads
identically to the sequential version — that’s the design goal.
The adapter zoo behaves the same, with one subtlety worth internalizing:
fn even_squares(data: &[u64]) -> Vec<u64> {
data.par_iter()
.filter(|x| *x % 2 == 0)
.map(|x| x.pow(2))
.collect() // results land back IN INPUT ORDER
}
collectpreserves order;for_eachdoes not. Threads finish in whatever order they finish, butcollecttracks each item’s index and reassembles a deterministicVecmatching the sequential result. If you need ordered output, reach formap(...).collect(), neverfor_eachwith a side effect.
filter’s closure receives a double reference (&&u64) — one & from
par_iter yielding &u64, another from filter borrowing it — same as
sequential iterators.
3. reduce and fold: why an identity closure?
sum() is a special case of reduce. The general tool looks like this:
fn word_count_total(words: &[&str]) -> usize {
words.par_iter()
.map(|w| w.len())
.reduce(|| 0_usize, |a, b| a + b)
// ^^^^^^^^^^ ^^^^^^^^^^^^^^
// identity combine
}
The first argument is a closure that returns the identity, not a single value
like Iterator::fold takes. Why? Because rayon splits the data into an unknown
number of independent chunks and must seed each one separately. There is no
single starting accumulator threaded left-to-right; each chunk starts from the
neutral element and the partials get combined. So rayon calls || 0 possibly
many times — once per chunk it decides to create.
fold-then-reduce makes the two-level structure explicit:
fn concat_lengths(words: &[&str]) -> usize {
words.par_iter()
.fold(|| 0_usize, |acc, w| acc + w.len()) // per-thread local accumulator
.reduce(|| 0_usize, |a, b| a + b) // merge the few partials
}
Rayon’s
foldis notIterator::fold. It returns another parallel iterator of partial results — one accumulator per chunk — which is why you chain.reduce(...)after it to collapse those partials to a scalar. The win: the per-item hot loop touches only a thread-local accumulator (cheap, no cross-thread coordination), and only the handful of partials pay the merge cost.
4. rayon::join: the primitive everything is built on
par_iter is sugar. Underneath, rayon recursively splits work with a single
primitive. rayon::join(a, b) runs closures a and b potentially in
parallel and returns (a_result, b_result):
fn sum_split(data: &[u64]) -> u64 {
if data.len() <= 1024 {
return data.iter().sum(); // base case: go sequential
}
let (left, right) = data.split_at(data.len() / 2);
let (l, r) = rayon::join(|| sum_split(left), || sum_split(right));
l + r
}
The word potentially is the whole magic. join pushes task b onto the
current thread’s deque and runs a itself. If another worker is idle, it
steals b and runs it concurrently. If no one is free, the current thread just
runs b after a. Either way there is zero wasted scheduling — that is
work-stealing, and it is why a recursion tree of joins automatically uses
however many cores happen to be free, with no manual chunk math.
The base-case cutoff (
len <= 1024) matters: recursing all the way down to single elements would drown the actual work injoinoverhead. This same “go sequential below a threshold” pattern reappears in the capstone.
5. When parallelism actually helps (and when it loses)
Rule of thumb: speedup ≈ (work_per_item × item_count) / overhead. The ladder
makes per-item work tunable and sweeps it:
fn expensive(x: u64, iters: u64) -> u64 { // tunable, pure, CPU-bound
let mut acc = x;
for _ in 0..iters { acc = acc.wrapping_mul(31).wrapping_add(7); }
acc
}
A representative --release run summing expensive over 100,000 items:
iters= 0: seq 20µs par 376µs -> 0.06x loss <- work ~ 0, pure overhead
iters= 1: seq 141µs par 426µs -> 0.33x loss
iters= 10: seq 162µs par 421µs -> 0.39x loss
iters= 100: seq 633µs par 1.60ms -> 0.39x loss
iters=1000: seq 8.57ms par 2.25ms -> 3.81x WIN <- work finally dominates
Read it like this:
- The parallel column has a floor (~400µs). That is rayon’s fixed cost: splitting, deque pushes, steal coordination, recombination. Below that floor, parallel can never win no matter how you write it.
- The crossover is between 100 and 1000 iters. At
iters=100, sequential is still cheaper (633µs) than parallel’s overhead-laden 1.6ms. Only when one pass costs ~8.5ms does dividing it across cores swamp the coordination cost. - 3.81×, not Ncores×. Perfect linear scaling never happens — memory bandwidth, the serial recombine step, and hyperthreads all skim off the top. ~4× on a typical machine is a healthy real result.
Takeaway. Use
par_iterwhentotal_work / coresclearly exceeds rayon’s ~hundreds-of-µs setup. Tiny collections or trivial per-item work → stay sequential. Summing a million plain integers is the worst showcase: the work is one add per item and the loop is memory-bound, so extra threads just fight over the memory bus. When unsure, measure exactly like the table above.
6. Non-associative reduce is a silent bug
Because reduce recombines partials in a tree shape that depends on how rayon
split the work — which depends on runtime scheduling — the combine operation
must be associative: (a ∘ b) ∘ c == a ∘ (b ∘ c). Subtraction is the classic
violator:
fn par_diff(data: &[i64]) -> i64 {
data.par_iter().copied().reduce(|| 0, |a, b| a - b) // BUG: not associative
}
fn seq_diff(data: &[i64]) -> i64 {
data.iter().fold(0, |a, b| a - b) // deterministic meaning
}
The root cause is provable without any threads at all:
assert_ne!((10 - 5) - 3, 10 - (5 - 3)); // 2 != 8 — grouping changes the answer
Running par_diff over a 200,000-element vector 200 times produced 200
distinct answers in 200 runs, and not one matched seq_diff. Every run, rayon
made slightly different steal decisions, grouped the subtractions differently, and
returned a different number.
This is the nightmare class of bug: it compiles, runs, and returns a plausible-looking value that is wrong and never the same twice. The fix is never “rearrange the reduce” — it is only feed
reduce/foldan associative op. Note floating-point+is technically non-associative too (rounding depends on order), so parallel float sums can differ slightly from the sequential sum.
7. The shared-state wall
The reflex from other languages — “make an empty list, have each task push into it” — does not compile in Rust, and the rejection is the lesson:
let mut out = Vec::new();
data.par_iter().for_each(|&x| out.push(x * x)); // WRONG: does not compile
out
for_each calls its closure from many threads at once, so the closure must be
Fn (shareable, borrowing captures by & only). But out.push needs
&mut out, and two threads mutating one Vec simultaneously is a data race — so
the borrow checker refuses (the closure would have to be FnMut, and &mut out
can’t be shared). Rust turns the data race into a compile error.
Two fixes, with a clear preference:
// OK, idiomatic: don't share state at all. Each task returns a value;
// collect reassembles them in order. Lock-free, race-free, deterministic.
fn squares_collect(data: &[u64]) -> Vec<u64> {
data.par_iter().map(|&x| x * x).collect()
}
// Works, but worse: serialize pushes behind a lock.
fn squares_mutex(data: &[u64]) -> Vec<u64> {
let out = Mutex::new(Vec::new());
data.par_iter().for_each(|&x| out.lock().unwrap().push(x * x));
out.into_inner().unwrap()
}
The Mutex version compiles and is correct as a set, but:
- order is lost — threads finish in any order, so you must sort to compare;
- every push contends on one lock, serializing the very work you parallelized.
Note it needs only a plain Mutex, no Arc: for_each merely borrows out,
and rayon guarantees all tasks finish before it returns, so a shared &Mutex
across the scoped tasks suffices — the same reasoning as scoped threads.
into_inner() then consumes the mutex to hand back the Vec with no clone.
If you find yourself locking to collect results,
collectwas the better tool.
8. Real-world APIs: par_sort and par_bridge
v.par_sort_unstable(); // parallel sort, std-identical API
Rayon adds par_sort / par_sort_unstable to slices — a drop-in parallel
quicksort/mergesort with the same signature as the std sort.
par_iter only works on things rayon can split by index (slices, ranges,
Vec). A plain sequential Iterator — like str::split_whitespace, which yields
tokens one at a time and can’t be indexed — has no .par_iter(). par_bridge
adapts any Iterator: Send into a parallel one:
fn bridge_word_sum(text: &str) -> u64 {
text.split_whitespace()
.par_bridge() // adapt sequential Iterator -> parallel
.map(|w| w.parse::<u64>().unwrap())
.sum()
}
par_bridgehas workers pull items from the shared sequential source behind a lock, so it has a serial pull-bottleneck and does not preserve order. When you can get a slice orVec, nativepar_iteris faster. Reach forpar_bridgeonly when the source is fundamentally sequential — lines from a reader, an FFI iterator, a generator.
Footguns
| Footgun | What bites | Fix |
|---|---|---|
| Parallelizing cheap work | par_iter slower than iter on small/memory-bound work | Measure; stay sequential below the break-even |
Non-associative reduce/fold | silent, non-deterministic wrong answers | Only use associative ops; beware float + |
Shared mutable state in for_each | won’t compile (Fn/&mut conflict) | map().collect(); Mutex only if forced |
Expecting for_each order | runs in arbitrary completion order | use collect for ordered output |
par_bridge for an indexable source | serial pull-bottleneck, unordered | use native par_iter on the slice/Vec |
| Benchmarking unused results | dead-code elimination deletes the work | std::hint::black_box(result) |
The benchmark footgun is worth a closer look. In the capstone’s timing block:
let _: Vec<u64> = data.iter().map(|&x| expensive(x, 500)).collect(); // result dropped
In --release, the optimizer proved expensive is pure and the result unused, so
it deleted the entire sequential loop — the timer reported ~88ns, a lie. The
parallel side survived only because rayon::join is an opaque call the optimizer
can’t see through. To benchmark honestly you must consume the result (e.g.
black_box), or assert on it as rung 5 does.
Real-world patterns
- Embarrassingly parallel transforms.
data.par_iter().map(expensive).collect()is the bread and butter — image pixels, rows of a dataset, files to process. - Parallel aggregation.
par_iter().map(...).sum()/.reduce(...)for stats over large collections, as long as the combine is associative. par_sort_unstablefor large in-memory sorts.par_bridgeto parallelize work over a streaming source you can’t index.- Custom thread pools (
rayon::ThreadPoolBuilder) when you need to bound parallelism or isolate workloads — the samepar_iter/joinAPI runs inside.
Capstone insight
The capstone rebuilds rayon-style machinery from the single primitive join — no
par_iter, no par_sort. The structural “aha”: every parallel algorithm here
is the same shape — recurse, fork the two halves with join, fall back to
sequential below a cutoff.
fn parallel_map<T, R, F>(data: &[T], f: &F) -> Vec<R>
where
T: Sync, // both halves read &[T] from different threads
R: Send, // each half's Vec<R> travels back to the joiner
F: Fn(&T) -> R + Sync, // the SAME closure is shared across threads
{
if data.len() <= THRESHOLD {
return data.iter().map(f).collect(); // sequential base case
}
let (left, right) = data.split_at(data.len() / 2);
let (mut left, right) = rayon::join(
|| parallel_map(left, f),
|| parallel_map(right, f),
);
left.extend(right); // left first -> input order
left
}
The bounds are the real lesson, and they fall straight out of what crosses threads:
T: Sync—&[T]is read concurrently by both recursive calls, and&T: Send ⟺ T: Sync.R: Send— each half builds aVec<R>on its worker and ships it back to the thread that calledjoin.F: Fn(&T) -> R + Sync— the same closure runs on many threads, so it must beSyncand must beFn(no shared mutable state;FnMutwould be a race). Passingfas&Fdown the recursion avoids needingF: Clone.
Parallel quicksort is the same skeleton, but the disjointness that makes parallel
mutation sound comes from split_at_mut:
fn parallel_quicksort<T: Ord + Send>(data: &mut [T]) {
if data.len() <= THRESHOLD { data.sort_unstable(); return; }
let len = data.len();
data.swap(len / 2, len - 1); // mid as pivot: avoids O(n^2) on sorted input
let mut p = 0; // Lomuto partition
for i in 0..len - 1 {
if data[i] <= data[len - 1] { data.swap(i, p); p += 1; }
}
data.swap(p, len - 1); // pivot to its final resting place
let (left, pivot_and_right) = data.split_at_mut(p);
let (_, right) = pivot_and_right.split_at_mut(1); // skip the pivot
rayon::join(|| parallel_quicksort(left), || parallel_quicksort(right));
}
split_at_muthands back two disjoint&muthalves. That non-overlap is exactly what lets rayon sort both sides in parallel safely — the borrow checker knows the two&mut [T]can’t alias, so there is no data race, and nounsafeis needed. This is the same trick assplit_at_mutin the scoped-threads ladder, now powering a parallel sort. Choosing the middle element as pivot (swap(len/2, len-1)) is the standard defense against quicksort’s O(n²) worst case on already-sorted or reversed input — which the test deliberately feeds it.
Explain it back
- Why does
reducetake an identity closure instead of a single initial value, whileIterator::foldtakes a value? - What is work-stealing, and why does it beat statically chunking a slice into N equal pieces?
rayon::join(a, b)“potentially” runs in parallel. What does it actually do when no worker is idle, and why is that not a waste?- You parallelized a sum over a million
u64s and it got slower. Give two reasons and the rule for when to expect a speedup. - Why does
par_iter().for_each(|x| vec.push(x))fail to compile? What two fixes exist and which is better? - A parallel reduce with subtraction gives a different answer every run. What law is broken and why does the parallel split expose it?
- In the capstone
parallel_map, justify each bound:T: Sync,R: Send,F: Fn(&T) -> R + Sync. - What makes parallel in-place quicksort sound without
unsafe?
See also
- Threads & scoped threads —
thread::scope,split_at_mut, the hand-rolledparallel_mapthat this ladder rebuilds onrayon::join. Send&Syncdeeply — the bounds that make all of rayon sound.Mutex/RwLock— the lock used (and avoided) in rung 7.- Iterators end-to-end — the sequential adapters
par_itermirrors. - Closures &
Fn/FnMut/FnOnce— whyfor_eachneedsFn.
Shared state vs message passing
Ladder:
src/bin/concurrency_models.rs· Run:cargo run --bin concurrency_models· Phase 4 · 9 rungs
TL;DR
Two threads need to agree on some data. There are exactly two ways to arrange that:
- Shared state — one piece of memory, many pointers into it, access serialized
by a lock (
Arc<Mutex<T>>/Arc<RwLock<T>>) or atomics. “Communicate by sharing memory.” - Message passing — the data has one owner at a time and is handed off down a channel. No lock, because there is nothing shared. “Share memory by communicating.”
Neither is “better.” The skill is reading a workload and picking — and the senior move is combining them: an actor thread privately owns the state (message-passing mechanics) while presenting a single logical store to everyone (shared-state semantics).
The deepest one-liner from the whole ladder:
A lock serializes access at the critical section. An actor serializes access at the queue. Both give you mutual exclusion — they just put the “one-at-a-time” gate in a different place.
Why this exists (from first principles)
A data race is two threads touching the same memory with at least one writing, and no
synchronization between them. Rust makes data races a compile error via Send/Sync
and the borrow rules — but it doesn’t pick your architecture. You still have to choose
how threads coordinate, and that choice has two fundamentally different answers.
The reason both exist is that they fail in opposite ways:
- Shared state is cheap to read and write (a lock is just a flag), but every thread contends for the same lock, so a fat critical section silently serializes your whole program (rung 5).
- Message passing has no lock to contend, and ownership transfer means there’s nothing to race over — but channels add per-message overhead, can deadlock under backpressure, and let you accidentally re-share state if you send the wrong type (rung 6).
Understanding both, and when each bites, is the entire topic.
The ladder at a glance
| # | Tier | Rung | The lesson |
|---|---|---|---|
| 1 | foundations | Two roads, one counter | The same sum via Arc<Mutex> and via an mpsc aggregator |
| 2 | foundations | Ownership transfer | Owned jobs through a channel — the Mutex guards the queue, not the data |
| 3 | mechanics | Pipeline of stages | produce → ×3 → keep-even → collect, each its own thread; EOF cascades |
| 4 | mechanics | Fan-out / fan-in both ways | Shared VecDeque queue vs mpsc job queue, side by side |
| 5 | footgun | Lock held too long | Slow work inside the critical section serializes N threads to 1 |
| 6 | footgun | Message-passing footguns | Stray sender hang, clean disconnect, bounded backpressure, the Arc-through-channel trap |
| 7 | real-world | The actor | One thread owns a HashMap; others send commands + a one-shot reply |
| 8 | real-world | Hybrid | Writes through the actor, lock-light reads from a published snapshot |
| 9 | capstone | One trait, two impls | KvStore backed by Arc<RwLock<HashMap>> vs an actor; same tests pass both |
The ideas, built up
1. The same problem, two shapes
Summing 1..=N across 8 threads. The threads do identical work; the only difference is
how their partial sums become one total.
Shared state — every worker locks one accumulator and adds into it:
fn sum_shared(n: u64) -> u64 {
let total = Arc::new(Mutex::new(0));
for i in 0..THREADS {
let total = Arc::clone(&total);
thread::spawn(move || {
let (lo, hi) = chunk_bounds(n, i);
let mut total = total.lock().unwrap(); // lock ONCE per thread
*total += (lo + hi) * (hi - lo + 1) / 2;
});
}
// join all ...
*total.lock().unwrap()
}
The discipline already shows: lock once per thread, not once per number. The lock is a coordination point, so you touch it as rarely as possible — 8 acquisitions, not a million.
Message passing — each worker sends its partial; main is the sole owner of the total:
fn sum_message(n: u64) -> u64 {
let (tx, rx) = mpsc::channel();
for i in 0..THREADS {
let tx = tx.clone();
thread::spawn(move || {
let (lo, hi) = chunk_bounds(n, i);
tx.send((lo + hi) * (hi - lo + 1) / 2).unwrap();
});
}
drop(tx); // <-- critical: see below
let mut total = 0;
while let Ok(partial) = rx.recv() { total += partial; }
total
}
There is no Mutex anywhere, yet this is perfectly correct under concurrency. The
channel is the synchronization, and the total only ever lives in one thread.
The
drop(tx)is not optional.rx.recv()returnsErronly when all senders are gone. The originaltxlives insum_message’s scope; if you don’t drop it, the loop waits forever for a value from a sender that will never send. (Rung 6 makes this its own lesson.)
2. Ownership transfer = no lock on the data
Send owned Vec<u8> payloads to a worker pool. Each payload is owned by exactly one
worker while processed, then its result is handed back.
let job_rx = Arc::new(Mutex::new(job_rx)); // mpsc Receiver is single-consumer
// worker:
let job = {
let job_rx = job_rx.lock().unwrap();
job_rx.recv()
}; // <-- guard dropped HERE
match job {
Ok(payload) => {
let sum = payload.into_iter().map(u64::from).sum(); // processed OFF the lock
res_tx.send(sum).unwrap();
}
Err(_) => break,
}
Two things make this work:
- The only lock guards the queue (the shared
Receiver), never the payloads. EachVec<u8>is moved into one worker and owned by it for its entire life. - The lock guard is bound in a tight inner scope so it drops before processing.
Footgun preview. Had you written
while let Ok(job) = job_rx.lock().unwrap().recv(), the temporary guard would live until the end of the loop body — so you’d process while holding the receiver lock, serializing your whole pool back to one worker. Bind the guard, then drop it.
3. Pipelines: ownership flows down the pipe
Message passing shines for pipelines — each stage is its own thread, linked only by channels, and an item flows stage → stage owned by one stage at a time:
[produce] --tx--> rx --[×3] --tx1--> rx1 --[keep even] --tx2--> rx2 --[collect]
thread::spawn(move || { for x in input { tx.send(x).unwrap(); } });
thread::spawn(move || { for x in rx { tx1.send(x * 3).unwrap(); } });
thread::spawn(move || { for x in rx1 { if x % 2 == 0 { tx2.send(x).unwrap(); } } });
let out: Vec<_> = rx2.into_iter().collect();
Why this is elegant:
- The EOF cascade. When
inputis exhausted, stage 1’s closure ends →txdrops → stage 2’sfor x in rxends →tx1drops → … → main’s loop ends. One “done” signal at the source propagates the whole way down with zero extra code. This is the same shutdown mechanism the actor uses (rung 7). - All stages run concurrently — a true assembly line, throughput gated by the slowest stage.
- Order is preserved for free, because each channel has exactly one feeder thread.
The price of “for free” here: the channels are unbounded, so a fast producer can outrun a slow consumer and grow memory. Bounded channels (rung 6c) add backpressure.
4. Fan-out / fan-in, both ways — the comparison
The same worker pool built twice, differing only in how a worker gets its next task.
Shared-state queue — workers drain a pre-loaded Arc<Mutex<VecDeque>>:
let task = { let mut q = queue.lock().unwrap(); q.pop_front() };
match task { None => break, Some(t) => partial += work(t) }
Message-passing queue — workers recv() from a shared channel until disconnect.
Both give the same total. The difference you should feel:
Shared VecDeque | mpsc channel | |
|---|---|---|
| “no more work” | you decide: pop_front() returns None | recv() returns Err on disconnect |
| empty queue, work still arriving | busy-spins on the lock (CPU burn) | parks the thread (no CPU) |
| backpressure | build it yourself | bounded channel gives it for free |
The decision rule this rung hands you:
Pre-loaded, bounded work → a shared queue is fine. Open-ended / streaming arrival → a channel, because blocking and end-of-stream come for free instead of being hand-rolled.
5. The shared-state tax: lock held too long
This is the footgun that defines shared state. The same sum, computed two ways, where
expensive(task) sleeps ~2 ms (standing in for parsing / hashing / I/O):
// WRONG: expensive() runs while the accumulator lock is held
let mut total = total.lock().unwrap();
*total += expensive(task);
// OK: expensive() runs off-lock; lock only for the +=
let v = expensive(task);
*total.lock().unwrap() += v;
Measured with 40 tasks, one thread per task:
held = 87 ms shrunk = 3 ms
The punchline is brutal: in the bad version the number of threads didn’t matter at all.
Forty threads delivered the throughput of one, because each held the lock for its whole
2 ms sleep and they queued single-file. Runtime is N × (time under lock) regardless of
core count.
The entire discipline of shared-state concurrency in one sentence: do the work outside the lock; touch the shared state for as short as possible. Snapshot-then-release, compute-then-commit.
6. The message-passing footguns
Channels have their own sharp edges. The ladder makes each observable (timeouts and
try_* so “it would hang” becomes a testable Err):
// (a) A live sender — even an unused clone — keeps recv blocking forever
let (tx, rx) = mpsc::channel();
let _tx_clone = tx.clone();
rx.recv_timeout(Duration::from_millis(50)) // Err(Timeout)
// (b) Drop EVERY sender -> the clean shutdown signal
drop(tx); drop(tx_clone);
rx.recv() // Err(RecvError)
// (c) Bounded buffer full -> backpressure made visible
let (tx, _rx) = mpsc::sync_channel::<i32>(2);
tx.send(1).unwrap(); tx.send(2).unwrap();
tx.try_send(99) // Err(Full(99))
The most insidious one is (d), the aliasing trap:
let arc = Arc::new(Mutex::new(0));
let arc_clone = arc.clone();
tx.send(arc_clone).unwrap(); // "sent" — but main still holds `arc`
// worker: *arc.lock().unwrap() += 10
*arc.lock().unwrap() // == 10 <- main SEES the worker's write
Sending an Arc<Mutex<T>> through a channel does not transfer ownership of the data.
Moving an Arc moves a pointer; both Arcs still alias the same Mutex. You’ve
re-introduced shared state behind a message-passing facade — and every shared-state hazard
with it.
The transfer-vs-share distinction lives in the type you send, not in the fact that you used a channel. Send a
Vec<u8>→ genuine handoff. Send anArc<Mutex<_>>→ you’re back in lock-land. This is how teams convince themselves they’ve “gone lock-free with channels” while quietly shipping shared state down those channels.
7. The actor: combining both models
The senior pattern. One thread privately owns a HashMap — no Mutex, no Arc
around the map. Everyone else holds a cheap clonable handle (just a Sender) and sends
command messages; reads carry a one-shot reply channel.
enum Command {
Get { key: String, reply: mpsc::Sender<Option<String>> },
Set { key: String, value: String },
}
impl KvActor {
fn spawn() -> Self {
let (tx, rx) = mpsc::channel();
thread::spawn(move || {
let mut map = HashMap::new(); // plain local — owned by one thread
for cmd in rx {
match cmd {
Command::Get { key, reply } => { reply.send(map.get(&key).cloned()).unwrap(); }
Command::Set { key, value } => { map.insert(key, value); }
}
}
});
KvActor { tx }
}
fn get(&self, key: &str) -> Option<String> {
let (reply_tx, reply_rx) = mpsc::channel(); // fresh one-shot
self.tx.send(Command::Get { key: key.into(), reply: reply_tx }).unwrap();
reply_rx.recv().unwrap() // blocks until the answer
}
}
Why this is the combining pattern:
- No lock on the map, because exactly one thread ever touches it. The borrow checker never even has to consider it shared — it isn’t.
- Concurrent correctness is automatic. Ten threads can clone the handle and fire commands; they can’t corrupt the map because they never touch it. The actor applies commands one-at-a-time off its queue.
getis a synchronous round-trip over two messages — send request with a reply channel, block on the reply. Reads feel like function calls.- Shutdown is free. When the last handle drops, its
txdrops, thefor cmd in rxloop ends, the thread exits — the same EOF cascade as the pipeline.
This is shared-state semantics (one logical store everyone uses) delivered through message-passing mechanics (zero locks on the data).
8. Hybrid: writes through the actor, reads from a snapshot
The plain actor has one weakness: reads queue behind writes (a Get is a command on
the same channel). For read-heavy workloads, decouple them.
The actor stays the single writer, but after each write it publishes an immutable
snapshot into a shared Arc<RwLock<Arc<HashMap>>>. Readers bypass the actor entirely:
// writer (inside the actor loop):
map.insert(key, value);
*snapshot.write().unwrap() = Arc::new(map.clone()); // publish: atomic pointer swap
ack.send(()).unwrap(); // ack so set() is read-your-writes
// reader:
let snap = Arc::clone(&self.snapshot.read().unwrap()); // bump refcount under a brief read-lock
snap.get(key).cloned() // lookup with NO lock held
Two subtleties make this correct and fast:
- The
RwLockReadGuardis a temporary that drops at the end of theletstatement, so the lookup runs lock-free on a privateArcclone. The read-lock is held for exactly one refcount bump. - Publishing is an atomic pointer swap of the whole map, so a reader holds either the old map or the new one — never a torn, half-applied state.
The tradeoff: publishing clones the whole map per write. That’s the price of
lock-light, never-changes-under-you reads. In production you’d reach for arc-swap (a
single atomic pointer swap, no RwLock) and/or persistent maps (im) so the clone is
structural-sharing-cheap.
9. Capstone: one trait, two architectures
The proof that the model is an implementation detail behind a stable interface:
trait KvStore: Send + Sync {
fn get(&self, key: &str) -> Option<String>;
fn set(&self, key: &str, value: &str);
fn delete(&self, key: &str);
fn len(&self) -> usize;
}
SharedStore=Arc<RwLock<HashMap>>; every op takes the appropriate lock.ActorStore= a worker thread owns the map; ops areGet/Set/Delete/Lencommands, reads/lencarry a reply channel, writes carry an ack so the caller gets read-your-writes.
The same exercise() and an 8-thread hammer() storm run against both via
Arc<dyn KvStore> and pass identically. A lock-based store and a lock-free actor store
are observably indistinguishable to callers.
One enabler worth noting:
mpsc::Sender<T>isSyncin current std, soActorStore(which holds aSender) isSend + Syncand fits behindArc<dyn KvStore>exactly like the lock-based store.
Footguns
| Trap | What happens | Fix |
|---|---|---|
Forgot drop(tx) | rx.recv() blocks forever — a live sender means “more might come” | drop every extra sender before draining |
| Stray sender clone | same hang, but harder to spot (an unused clone counts) | give each sender a clear owner/scope; drop deliberately |
| Lock across slow work | N threads serialize to 1; thread count stops mattering | compute off-lock, lock only to commit |
| Guard held too long | lock().recv() in a while let holds the guard across the body | bind the guard in a tight scope so it drops first |
| Bounded channel full, no consumer | blocking send parks forever = deadlock | size the buffer; use try_send; ensure a draining consumer |
Arc<Mutex> through a channel | “message passing” that secretly shares state | send owned data for real handoff; if you must share, own it |
Real-world patterns
- mpsc worker pool —
Arc<Mutex<Receiver>>shared by N workers, lock only to dequeue (rungs 2, 4). This is the bones of a thread pool. - Pipelines — stage-per-thread linked by channels (rung 3); the threaded form of Unix pipes / streaming ETL.
- The actor —
tokio’s recommended pattern for shared mutable state in async code is exactly rung 7 (a task owning the state + an mpsc command channel). TheCommandenum is a protocol; swap the transport (channel → socket) and callers don’t change. - Read-optimized snapshots —
arc-swapand copy-on-write config publishing are rung 8 in production form. - Strategy behind a trait — rung 9 is how you keep a concurrency choice swappable; the same shape lets you A/B a lock-based and actor-based backend.
Capstone insight
The whole ladder collapses to a single realization:
Mutual exclusion has to live somewhere. A lock puts the gate at the critical section — threads queue to touch the data. An actor puts the gate at the message queue — threads queue to send a command, and one owner touches the data.
Once you see that, “shared state vs message passing” stops being a religious debate and becomes an engineering trade-off: where do you want the queue, how expensive is each crossing, and what does the data’s ownership story actually look like.
Explain it back
- Why does
sum_messageneed noMutex, and why isdrop(tx)mandatory? - In the worker pool, what exactly does the one
Mutexprotect — and what does it not? - Why did 40 threads run at the speed of one in the “lock held too long” rung?
- A coworker says “we use channels so we’re lock-free.” They send
Arc<Mutex<State>>down those channels. What’s wrong? - Why does the actor’s
HashMapneed no lock? Where did the serialization go? - In the hybrid store, why is a reader guaranteed never to see a half-applied write?
- State the “where is the gate?” insight in one sentence.
See also
- Channels — the mpsc /
sync_channel/ crossbeam mechanics this builds on Mutex/RwLock— guards, poisoning, deadlock, lock orderingSend&Syncdeeply — whyArc<Mutex>crosses threads andRccan’t- Threads & scoped threads —
spawn,join, the'staticwall - Data parallelism with
rayon— when the answer is “neither, justpar_iter”
Adding a new note
The workflow for capturing a concept the moment a ladder is done.
Steps
- Copy the template into
docs/src/concepts/<concept>.md(kebab-case, e.g.cell-refcell.md). - Fill every section — TL;DR, why, the ladder table, signatures, footguns,
“explain it back”. Pull the rung list straight from the
// Ladder:comment at the top ofsrc/bin/<concept>.rs. - Register it in
SUMMARY.mdunder the right Phase heading. A page not listed inSUMMARY.mdis not built. - Tick the row in the completed-concepts table on the Introduction.
- Commit & push to
master— theDeploy docsGitHub Action rebuilds and publishes the site automatically.
Local preview
# one-time: install mdBook
cargo install mdbook # or: brew install mdbook
# live-reloading preview at http://localhost:3000
mdbook serve docs --open
# one-off build into docs/book/ (gitignored)
mdbook build docs
Just say the word
In Claude Code you can simply say “write up the <concept> note” after a ladder
is finished — it reads the source file and drafts the page in this format.
One-time GitHub setup
For the deploy workflow to publish, set Settings → Pages → Build and deployment
→ Source = GitHub Actions once on the repo. After that, every push to master
that touches docs/ redeploys to https://utkarsh5026.github.io/rust-scratch/.
Note template
Copy this into docs/src/concepts/<concept>.md when a ladder is finished, then
fill every section. Keep it tight — this is a reference you’ll re-read, not a
transcript of the ladder.
# <Concept name>
> Ladder: [`src/bin/<concept>.rs`](https://github.com/utkarsh5026/rust-scratch/blob/master/src/bin/<concept>.rs) ·
> Run: `cargo run --bin <concept>` · Phase N · M rungs
## TL;DR
<The one-paragraph mental model. If you can't write this, you're not done.>
## Why it exists
<The concrete problem this concept solves. What goes wrong without it?>
## The ladder
| # | Tier | Rung | The lesson |
|---|------|------|------------|
| 1 | foundations | <rung> | <one line> |
| … | | | |
| M | capstone | <rung> | <one line> |
## Signatures to know
<The std types / trait defs / bounds worth memorizing, in fenced rust blocks.>
## Footguns
- **<trap>.** <what bites you and the fix.>
## Explain it back
- <question you should answer cold>
- …
## See also
- <related concept notes>