Tracking issue for the OOM hook (alloc_error_hook)

@SimonSapin

PR #50880 added an API to override the std OOM handler, similarly to the panic hook. This was discussed previously in issue #49668, after PR #50144 moved OOM handling out of the Alloc/GlobalAlloc traits. The API is somewhat similar to what existed before PR #42727 removed it without an explanation. This issue tracks the stabilization of this API.

Defined in the std::alloc module:

~~pub fn set_oom_hook(hook: fn(Layout) -> !); pub fn take_oom_hook() -> fn(Layout) -> !;~~

pub fn set_alloc_error_hook(hook: fn(Layout));
pub fn take_alloc_error_hook() -> fn(Layout);

CC @rust-lang/libs, @SimonSapin

Unresolved questions

Name of the functions. The API before rustc: Implement the #[global_allocator] attribute #42727 used _handler, I made it _hook in OOM handling changes #50880 because that's the terminology used for the panic hook (OTOH, the panic hook returns, contrary to the OOM hook). Make the OOM hook return () rather than ! #51264
Should this move to its own module, or stay in std::alloc?
Interaction with unwinding. alloc::alloc::oom is marked #[rustc_allocator_nounwind], so theoretically, the hook shouldn't panic (except when panic=abort). Yet if the hook does panic, unwinding seems to happen properly except it doesn't.
Should we have this, or Tracking issue for the #[alloc_error_handler] attribute (for no_std + liballoc) #51540, or both?
Who is responsible for preventing reentrancy issues through the allocator? unsafe-code-guidelines#506

We have an accepted RFC #48043 that includes adding a way to opt into of panicking instead of aborting on OOM (so that catch_unwind could be used to recover from OOM at coarse granularity). Perhaps this could be the mechanism for that?

Oh right, I had forgotten about that. And in fact, it works:

#![feature(allocator_api)]

use std::alloc::set_oom_hook;
use std::alloc::Layout;

fn oom(layout: Layout) -> ! {
    panic!("oom {}", layout.size());
}

fn main() {
    set_oom_hook(oom);
    let result = std::panic::catch_unwind(|| {
        let v = Vec::<u8>::with_capacity(100000000000000);
        println!("{:p}", &v[..]);
    });
    println!("{:?}", result);
}

cargo run with last nightly:

thread 'main' panicked at 'oom 100000000000000', src/main.rs:7:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.
Err(Any)

(despite the #[rustc_allocator_nounwind] annotation on alloc::alloc::oom)

Added item about interaction with unwinding.

@glandium

@glandium While this example works, the #[rustc_allocator_nounwind] annotation will (likely) prevent some drops from running during unwinding (in particular those that would have to be run inside the function that performs the oom call).

Picture me puzzled.

#![feature(allocator_api)]

use std::alloc::set_oom_hook;
use std::alloc::Layout;

struct Foo;

impl Drop for Foo {
    fn drop(&mut self) {
        println!("Foo");
    }
}

fn oom(layout: Layout) -> ! {
    let f = Foo;
    panic!("oom {}", layout.size());
}

fn main() {
    set_oom_hook(oom);
    let result = std::panic::catch_unwind(|| {
        let f = Foo;
        let v = Vec::<u8>::with_capacity(100000000000000);
        println!("{:p}", &v[..]);
    });
    println!("{:?}", result);
}

thread 'main' panicked at 'oom 100000000000000', src/main.rs:16:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.
Foo
Foo
Err(Any)

Or

#![feature(allocator_api)]

use std::alloc::set_oom_hook;
use std::alloc::Layout;

struct Foo;

impl Drop for Foo {
    fn drop(&mut self) {
        println!("Foo");
    }
}

fn oom(layout: Layout) -> ! {
    let f = Foo;
    panic!("oom {}", layout.size());
}

fn main() {
    set_oom_hook(oom);
    let result = std::panic::catch_unwind(|| {
        let f = Foo;
        std::alloc::oom(Layout::new::<u8>());
        println!("after oom");
    });
    println!("{:?}", result);
}

thread 'main' panicked at 'oom 1', src/main.rs:16:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.
Foo
Foo
Err(Any)

Ah, I forgot --release, and with it, in both cases, the Foo instance in the closure is not dropped (the one in oom is, though).

With panic hooks they notably return () instead of ! which allows you to chain them (if necessary). I think if we want to use the terminology "hook" for OOM we probably want the same (not returning !) as that'll allow us to also justify take_oom_hook as otherwise I think you can't actually call that and/or delegate in an order where you don't go first?

OOM hooks could always panic/abort themselves of course but the signature may just want to be that by default we don't require it and then the fallback of OOM is to abort as OOM does today

take_oom_hook is still useful albeit annoying, since hooks can't be closures. So you'd have to store the value in a global, but then you can call it from your own function. I can see how returning () would make things more flexible, though.

Make the OOM hook return `()` rather than `!`

Auto merge of #51264 - glandium:oom, r=alexcrichton

Auto merge of rust-lang#109507 - Amanieu:panic-oom-payload, r=davidtwco

This tracking issue was closed when PR #109507 merged to remove the feature, but that was reverted in PR #110782. Reopening to reflect that.

PR #112331 is open to remove the feature again, and is already marked as closing this.

We discussed this in the @rust-lang/libs-api meeting today. Considering the recent progress towards externally implementable items (#146348), we think that the best way forward would be to get rid of the OOM hooks entirely and instead just allow users to override the standard library's #[alloc_error_handler].

docs(alloc): explain why ReportingAllocator wraps mimalloc on stable …

Per yesterday's libs-api meeting, we'd like to split set_alloc_error_hook in twain so that setters can explicitly opt-in to unwinding semantics. There have been several soundness issues resulting from unwinds on OOM, and the effective safety requirements for it are incredibly strict (auditing one's entire dependency graph, since the vast majority of unsafe code is unsound in the presence of any spurious unwinds at all), so we'd like to have a version of set_alloc_error_hook that wraps the hook in an abort-on-unwind guard. I'll send a PR implementing this, but naming is open to bikeshedding ^^

Rollup merge of rust-lang#159702 - nia-e:alloc-error-unwind-hook, r=j…

With #159702, the docs for set_alloc_error_hook_unwinding now say

/// Unwinding from the allocation error hook is not unsafe per se; however, large swathes of
/// library code assume this never happens and may cause UB. Therefore, upon setting a hook
/// that may unwind, care should be taken to audit code in dependencies for possible unsoundness
/// in the presence of unwinds and ensure UB is not triggered.

This is unprecedented and IMO plain inacceptable. No other safety comment in std or anywhere else that I have seen says "audit the entire codebase". Has anyone even given a plausible argument for how that could be practically done? I doubt it can be practically done so I think this safety comment is equivalent to saying "this is not safe, there will probably be UB later, good luck have fun". This safety-comment is unformalizable in verification tools as it does not actually have a meaning (not in the same sense that our other safety comments do). This isn't a safety comment, it's an admission of defeat.

I am very strongly opposed to creating an ecosystem precedent for such "safety" reasoning (it doesn't deserve that name IMO). If other crates follow this example, the social contract that keeps unsafe Rust usable at scale will start eroding and risks destroying one of the key values of Rust -- an ecosystem of sound libraries that one can reason about in isolation and compose to large applications without worrying about that composition introducing UB.

Navigation Menu

Repository navigation

Tracking issue for the OOM hook (`alloc_error_hook`) #51245

Unresolved questions

30 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Tracking issue for the OOM hook (alloc_error_hook) #51245

Description

Unresolved questions

Activity

SimonSapin commented on May 31, 2018

glandium commented on May 31, 2018

glandium commented on May 31, 2018

glandium commented on May 31, 2018

nikic commented on May 31, 2018

glandium commented on May 31, 2018

glandium commented on May 31, 2018

glandium commented on May 31, 2018

alexcrichton commented on May 31, 2018

glandium commented on May 31, 2018

30 remaining items

SimonSapin commented on Dec 9, 2023

Amanieu commented on Dec 9, 2025

nia-e commented on Jul 22, 2026

RalfJung commented on Jul 22, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Tracking issue for the OOM hook (`alloc_error_hook`) #51245