Go 1.0.1: The Escaping Bug

Go 1.0.1 shipped on 25 April 2012, twenty-eight days after Go 1. It was the first point release, and it existed to fix one thing. The release note lists it in a single clause: an escape-analysis bug that can lead to memory corruption.¹

Escape analysis is a compiler optimization. It decides whether a value lives in a function's stack frame or on the heap. Get that decision wrong in the cheap direction and you pay one needless allocation. Get it wrong in the other direction and you leave a pointer aimed at stack memory that has already been handed back and reused, so the program reads or writes through it and sees garbage. So a memory-corruption fix ships as a change to an optimization pass: in Go, where a value lands is part of what keeps the language memory-safe, and for one shape of code 1.0 put it in the wrong place.

The report#

The report is issue #3545, "cmd/gc: escape analysis bug," opened by Russ Cox on 18 April 2012 against the Go1.0.1 milestone.² His description is two sentences:

In the program below, 'i' is treated as not escaping, yet it does escape. This can cause crashes and memory corruption.

Here is that program's shape, cut down to the moving parts. Hit Run.

package main

import "fmt"

type box struct{ v *int }

func send(ch chan *box, n int) {
	i := n
	b := box{&i}
	defer func() { ch <- &b }()
}

func main() {
	ch := make(chan *box, 2)
	send(ch, 1)
	send(ch, 2)
	close(ch)
	for b := range ch {
		fmt.Println(*b.v)
	}
}

1
2

It prints 1 and 2, the correct result on a current toolchain.

Here is the same program compiled with Go 1.0 itself, the release that shipped the bug. I built that toolchain from its source tag in a period container and ran the program unchanged:³

$ go version
go version go1
$ go run corruption.go
2
0

Not 1 and 2. The first value the channel hands back is wrong and the second is zeroed garbage, the same on every run. Both pointers were left aimed at send's stack frame, and the second call reused that frame before the range loop read through them. The exact wrong values depend on what the reused stack held; that they are wrong does not. This is the bug Russ Cox reported, on the compiler that still had it.

Look at what has to escape. Inside send, the local i has its address taken into b. The deferred closure sends &b on a channel that is buffered and outlives the call. So both i and b genuinely outlive send's frame. The compiler's one job is to notice that and put them somewhere that survives the return. In Go 1.0 it did not notice.

What escape analysis decides#

Escape analysis runs at compile time. For every value a function creates, it asks one question: does any reference to this value survive past the function's return? If the compiler can prove the answer is no, the value stays in the stack frame and is reclaimed for free when the frame pops, zero garbage-collector involvement. If a reference can outlive the call, the value escapes and goes on the heap, where the collector keeps it alive as long as something points at it.

This is what makes return &local legal in Go. In C that is a dangling pointer and undefined behavior. In Go the compiler sees the address leaving the frame and silently promotes the variable to the heap, so the pointer stays valid. The analysis is what lets you take the address of a local without thinking about its lifetime.

The two ways to be wrong are not symmetric:

• A false positive is the cheap mistake. The compiler decides a value escapes when it does not, so it lands on the heap when the stack would have done. You pay one allocation, and the program is still correct.
• A false negative is the dangerous one. The compiler decides a value does not escape when it does. It stays on the stack, a pointer to it outlives the return, the frame is reclaimed, the next call reuses that region, and every read or write through the stale pointer hits whatever now lives there. That is memory corruption, up to a pointer field sitting over bytes that were never a pointer.

So the analysis has one hard constraint: it may over-approximate escaping as much as it wants, and it may never under-approximate. Heap-allocating something that was stack-safe is a slowdown. Stack-allocating something that escapes is a hole. Issue #3545 was an under-approximation, the only kind of escape mistake that can corrupt memory, and that is what made it worth a point release.

You can watch the decision happen at runtime. testing.AllocsPerRun is an ordinary exported function, so it runs anywhere, no build flags. It counts heap allocations per call.

package main

import (
	"fmt"
	"testing"
)

// makeOnHeap returns the address of a local, so n must live on the heap.
func makeOnHeap() *int {
	n := 42
	return &n
}

// sumOnStack computes a value that never leaves the frame: zero heap allocs.
func sumOnStack() int {
	total := 0
	for i := 0; i < 10; i++ {
		total += i
	}
	return total
}

// Package-level sinks keep the compiler from optimizing the calls away.
var sink *int
var stackSink int

func main() {
	escaping := testing.AllocsPerRun(100, func() { sink = makeOnHeap() })
	stacked := testing.AllocsPerRun(100, func() { stackSink = sumOnStack() })
	fmt.Printf("makeOnHeap (returns &local): %.0f alloc(s)/call\n", escaping)
	fmt.Printf("sumOnStack (stays in frame): %.0f alloc(s)/call\n", stacked)
}

makeOnHeap (returns &local): 1 alloc(s)/call
sumOnStack (stays in frame): 0 alloc(s)/call

The 1 versus the 0 is the escape decision. makeOnHeap returns a pointer to its local, so n is forced onto the heap and every call allocates. sumOnStack keeps everything in the frame and allocates nothing. The package-level sinks matter: without them the compiler proves the calls are dead and deletes them, and you measure nothing.

To see the reasoning instead of the cost, ask the compiler directly. Pass -m through -gcflags and it prints its escape verdicts to stderr at build time.⁴ The -m output prints to a build terminal, not the browser, so this block is a local build rather than the Playground.

package main

var sink any

//go:noinline
func escapes(x int) {
	sink = x // stored in a package-level interface: must outlive the call
}

//go:noinline
func stays(p *int) int {
	return *p + 1 // only reads through p; nothing leaves the frame
}

func main() {
	escapes(42)
	n := 7
	_ = stays(&n)
}

$ go build -gcflags=-m escape.go
./escape.go:7:9: x escapes to heap
./escape.go:11:12: p does not escape

The verdicts follow the code. escapes stores its argument in a package-level interface that outlives the call, so the value goes to the heap. stays only reads through its pointer parameter, so nothing leaves the frame. escapes to heap and does not escape are the answers the analysis exists to produce, and #3545 was 1.0 producing the wrong one for a value that left through a closure.

Why this pattern fooled the 2012 compiler#

The trigger in #3545 is narrow. The local's address escapes only through a closure that is invoked in place, here the deferred func literal. The 1.0 analyzer got exactly that corner wrong.

A closure captures the locals it uses by reference. Inside the compiler that captured local is represented as a PPARAMREF, a reference node standing in for the original variable. When code inside the closure took the address of a captured local and let that address escape, the 1.0 analyzer walked the escaping value back as far as the PPARAMREF and stopped. It never followed the last edge, from the reference back to the variable it referred to. So the original local was still classed as non-escaping and left on the stack, even though the closure had just leaked its address to a channel.

The corruption follows from stack reuse:

1. send(ch, 1) returns. Its frame, holding i and b, is free stack again.
2. send(ch, 2) is called and reuses that same stack region for its own i and b.
3. Both *box pointers now sitting in the channel alias that reused memory. The value the first call queued no longer exists at the address the channel remembers.
4. The range loop reads *b.v through those pointers and gets whatever the second call, or anything after it, left in the slot.

The fix is to heap-allocate i and b, so each call gets its own storage that stays valid for as long as the channel holds a pointer to it.

The fix#

Luuk van Dijk, who wrote the escape analyzer, fixed it in CL 6061043, "cmd/gc: fix addresses escaping through closures called in-place," with Russ Cox reviewing. The description is one line: "Fixes issue 3545."⁵ The change is in escwalk(), the function that walks the value-flow graph, and it adds the missing edge: when it reaches a captured reference, follow it back to the original.

// Follow the missing edge: a captured reference back to its original.
if(src->class == PPARAMREF) {
	if(leaks && debug['m'])
		warnl(src->lineno, "leaking closure reference %hN", src);
	escwalk(level, dst, src->closure);
}

With that edge in place, the escape flooding reaches the original variables and tags them. Run the bug program through the same -m flag and the compiler now reports what it missed in 1.0:

$ go build -gcflags=-m corruption.go
./corruption.go:8:2: moved to heap: i
./corruption.go:9:2: moved to heap: b
./corruption.go:19:15: *b.v escapes to heap

Lines 8 and 9 are i and b, the two locals the 1.0 compiler wrongly left on the stack, now moved to the heap where they belong. The CL also added regression tests foo124 through foo137 to test/escape2.go, covering the in-place closure across every form it can take: immediate call, defer, go, nested, and inside a loop. Those tests are why no later rewrite reintroduced #3545.

A young pass#

Escape analysis was new. It entered the tree on 24 August 2011, off by default behind a flag, and was turned on for everyone four days later.⁶ Go 1.0 tagged seven months after that. What shipped in March 2012 was a single file, src/cmd/gc/esc.c, about eight hundred lines, doing flow-based analysis with a deliberately coarse set of escape states and no notion of partial or per-field escape.

It was conservative on purpose, and it carried author TODOs marking the places it gave up. Closures were one of the known rough spots. The commit log for the seven months before 1.0 has more than one entry along the lines of "fix escape analysis bug," including one landing about a month before the release that dealt specifically with escape analysis interacting badly with closures. Issue #3545 is the same corner, found a month after release instead of a month before. It fits the maturity of the code exactly. Escape analysis has been rewritten many times since, and none of those rewrites touched a line of anyone's Go source, because escape analysis is an implementation detail, not part of the Go 1 spec.

The first point release#

Go 1.0.1 also carried a handful of smaller standard-library fixes, among them a text/template typecheck on pipelined arguments and a panic in encoding/base64 on input whose length was not a multiple of four. The escape-analysis bug is the one the release was issued for.

Go 1 had shipped its compatibility promise four weeks earlier: source written to the Go 1 spec would keep compiling. Go 1.0.1 was the first live test of how that promise absorbs a bug. The fix landed on tip, was cherry-picked onto the release branch, and the release was cut by tagging that branch. No API changed. No source needed editing. A memory-corruption bug in the compiler was closed by recompiling with a fixed compiler. You cannot see it on any toolchain you can install today; the only way to watch it happen is to rebuild the 2012 one, which is what printed the 2 and the 0.