Learning Go: Stale slices explained

October 18, 2023
Massimiliano Ziccardi
Related topics:
GoLinux
Related products:
Red Hat Enterprise Linux

This article focuses on some weird Go slice behaviors that the developer might encounter without adequate knowledge of slice internals.

I will begin by illustrating the problem with a fictional company called ACME Corporation and its software engineer, Wile E. Then I will explain why that happened and how to avoid it.

Wile meets the Go slices

During the company's day of learning, Wile decides to learn the Go language. He read about arrays and slices and wants to do some practice, beginning with a straightforward exercise—a piece of code that:

  • creates a slice containing all the digits from 1 to 0
  • derives from the previous one, another slice containing all the digits from 1 to 9.

Wile thinks: "Wow, that's really easy! Let's try some code!"

He starts writing the code and ends up with the first exercise:

package main

import "fmt"

func main() {
	// STEP1: create the first slice with the digit from 1 to 0
	fmt.Println("[STEP1] Create a slice containing all the decimal digits (1,2,3,4,5,6,7,8,9,0)")
	charSlice1 := []byte{'1', '2', '3', '4', '5', '6', '7', '8', '9', '0'}
	fmt.Println("[STEP1]", "charSlice1", string(charSlice1))

	fmt.Println("\n[STEP2] Create a slice containing all digits from the first slice but the last one")
	// STEP2: create a new slice with all the digits from slice 1 but the last one (the 0)
	charSlice2 := charSlice1[:len(charSlice1)-1]
	fmt.Println("[STEP2]", "charSlice2", string(charSlice2))

	fmt.Println("\nFinal Result:")
	// Print both slices
	fmt.Println("charSlice1", string(charSlice1))
	fmt.Println("charSlice2", string(charSlice2))
}

He runs the code and gets the following output:

[STEP1] Create a slice containing all the decimal digits (1,2,3,4,5,6,7,8,9,0)
[STEP1] charSlice1 1234567890

[STEP2] Create a slice containing all digits from the first slice but the last one
[STEP2] charSlice2 123456789

Final Result:
charSlice1 1234567890
charSlice2 123456789

Wile thinks: "Well, that's been easy! From what I learned, a slice is just a view on a backing array, so charSlice1 and charSlice2 are pointing to the same backing array. Let's try to change the content of one: if that's true, the other one will also change!"

Wile ends up with the following code:

package main

import "fmt"

func main() {
	// STEP1: create the first slice with the digit from 1 to 0
	fmt.Println("[STEP1] Create a slice containing all the decimal digits (1,2,3,4,5,6,7,8,9,0)")
	charSlice1 := []byte{'1', '2', '3', '4', '5', '6', '7', '8', '9', '0'}
	fmt.Println("[STEP1]", "charSlice1", string(charSlice1))

	// STEP2: create a new slice with all the digits from slice 1 but the last one (the 0)
	fmt.Println("\n[STEP2] Create a slice containing all digits from the first slice but the last one")
	charSlice2 := charSlice1[:len(charSlice1)-1]
	fmt.Println("[STEP2]", "charSlice2", string(charSlice2))

	// STEP3: change the numbers 3 and 4 with an X in charSlice2
	fmt.Println("\n[STEP3] Changing the numbers 3 and 4 to the 'X' character in charSlice2")
	charSlice2[2] = 'X'
	charSlice2[3] = 'X'
	fmt.Println("[STEP3]", "charSlice2", string(charSlice2))

	// Print both slices
	fmt.Println("\nFinal Result:")
	fmt.Println("charSlice1", string(charSlice1))
	fmt.Println("charSlice2", string(charSlice2))
}

He runs it and gets the following output:

[STEP1] Create a slice containing all the decimal digits (1,2,3,4,5,6,7,8,9,0)
[STEP1] charSlice1 1234567890

[STEP2] Create a slice containing all digits from the first slice but the last one
[STEP2] charSlice2 123456789

[STEP3] Changing the numbers 3 and 4 to the 'X' character in charSlice2
[STEP3] charSlice2 12XX56789

Final Result:
charSlice1 12XX567890
charSlice2 12XX56789

As expected, changing charSlice2 changed charSlice1 too.

Wile thinks: "That's too easy. I'm starting to love Go! Now I want to add a fourth step that appends a value to the slices."

After a few seconds, Wile ends up with the snippet below:

package main

import "fmt"

func main() {
	// STEP1: create the first slice with the digit from 1 to 0
	fmt.Println("[STEP1] Create a slice containing all the decimal digits (1,2,3,4,5,6,7,8,9,0)")
	charSlice1 := []byte{'1', '2', '3', '4', '5', '6', '7', '8', '9', '0'}

	// STEP2: create a new slice with all the digits from slice 1 but the last one (the 0)
	fmt.Println("\n[STEP2] Create a slice containing all digits from the first slice but the last one")
	charSlice2 := charSlice1[:len(charSlice1)-1]

	fmt.Println("\n[STEP2] Result:")
	fmt.Println("[STEP2]", "charSlice1", string(charSlice1))
	fmt.Println("[STEP2]", "charSlice2", string(charSlice2))

	// STEP3: change the numbers 3 and 4 with an X in charSlice2
	fmt.Println("\n[STEP3] Changing the numbers 3 and 4 to the 'X' character in charSlice2")
	charSlice2[2] = 'X'
	charSlice2[3] = 'X'

	fmt.Println("\n[STEP3] Result:")
	fmt.Println("[STEP3]", "charSlice1", string(charSlice1))
	fmt.Println("[STEP3]", "charSlice2", string(charSlice2))

	// STEP4: append a '-' to charSlice2
	fmt.Println("\n[STEP4] Append '-' to charSlice2")
	charSlice2 = append(charSlice2, '-')

	fmt.Println("\n[STEP4] Result:")
	fmt.Println("[STEP4]", "charSlice1", string(charSlice1))
	fmt.Println("[STEP4]", "charSlice2", string(charSlice2))
}

However, this time, running it doesn't show what Wile was expecting:

[STEP1] Create a slice containing all the decimal digits (1,2,3,4,5,6,7,8,9,0)

[STEP2] Create a slice containing all digits from the first slice but the last one

[STEP2] Result:
[STEP2] charSlice1 1234567890
[STEP2] charSlice2 123456789

[STEP3] Changing the numbers 3 and 4 to the 'X' character in charSlice2

[STEP3] Result:
[STEP3] charSlice1 12XX567890
[STEP3] charSlice2 12XX56789

[STEP4] Append '-' to charSlice2

[STEP4] Result:
[STEP4] charSlice1 12XX56789-
[STEP4] charSlice2 12XX56789-

"In Step 4, why does appending a character in charSlice2 change the last character of charSlice1?" Wile wonders. "Weird..."

He is a bit confused, so he tries to add a further step (Step 5) that adds three more characters to charSlice2.

He writes the following code he wrote:

package main

import "fmt"

func main() {
	// STEP1: create the first slice with the digit from 1 to 0
	fmt.Println("[STEP1] Create a slice containing all the decimal digits (1,2,3,4,5,6,7,8,9,0)")
	charSlice1 := []byte{'1', '2', '3', '4', '5', '6', '7', '8', '9', '0'}

	// STEP2: create a new slice with all the digits from slice 1 but the last one (the 0)
	fmt.Println("\n[STEP2] Create a slice containing all digits from the first slice but the last one")
	charSlice2 := charSlice1[:len(charSlice1)-1]

	fmt.Println("\n[STEP2] Result:")
	fmt.Println("[STEP2]", "charSlice1", string(charSlice1))
	fmt.Println("[STEP2]", "charSlice2", string(charSlice2))

	// STEP3: change the numbers 3 and 4 with an X in charSlice2
	fmt.Println("\n[STEP3] Changing the numbers 3 and 4 to the 'X' character in charSlice2")
	charSlice2[2] = 'X'
	charSlice2[3] = 'X'

	fmt.Println("\n[STEP3] Result:")
	fmt.Println("[STEP3]", "charSlice1", string(charSlice1))
	fmt.Println("[STEP3]", "charSlice2", string(charSlice2))

	// STEP4: append a '-' to charSlice2
	fmt.Println("\n[STEP4] Append '-' to charSlice2")
	charSlice2 = append(charSlice2, '-')

	fmt.Println("\n[STEP4] Result:")
	fmt.Println("[STEP4]", "charSlice1", string(charSlice1))
	fmt.Println("[STEP4]", "charSlice2", string(charSlice2))

	// STEP5: append '+*/' to charSlice2
	fmt.Println("\n[STEP5] Append '+*/' to charSlice2")
	charSlice2 = append(charSlice2, '+', '*', '/')

	fmt.Println("\n[STEP5] Result:")
	fmt.Println("[STEP5]", "charSlice1", string(charSlice1))
	fmt.Println("[STEP5]", "charSlice2", string(charSlice2))

}

Another surprise hits Wile. Now the output is:

[STEP1] Create a slice containing all the decimal digits (1,2,3,4,5,6,7,8,9,0)

[STEP2] Create a slice containing all digits from the first slice but the last one

[STEP2] Result:
[STEP2] charSlice1 1234567890
[STEP2] charSlice2 123456789

[STEP3] Changing the numbers 3 and 4 to the 'X' character in charSlice2

[STEP3] Result:
[STEP3] charSlice1 12XX567890
[STEP3] charSlice2 12XX56789

[STEP4] Append '-' to charSlice2

[STEP4] Result:
[STEP4] charSlice1 12XX56789-
[STEP4] charSlice2 12XX56789-

[STEP5] Append '+*/' to charSlice2

[STEP5] Result:
[STEP5] charSlice1 12XX56789-
[STEP5] charSlice2 12XX56789-+*/

Wile: "That can't be! In Step 5, it didn't change charSlice1! How can it be that the same function, called two times, has two different behaviors?"

He then decides to add another step (Step 6) where he will change back the X characters of the charSlice2 and produces the following code:

package main

import "fmt"

func main() {
	// STEP1: create the first slice with the digit from 1 to 0
	fmt.Println("[STEP1] Create a slice containing all the decimal digits (1,2,3,4,5,6,7,8,9,0)")
	charSlice1 := []byte{'1', '2', '3', '4', '5', '6', '7', '8', '9', '0'}

	// STEP2: create a new slice with all the digits from slice 1 but the last one (the 0)
	fmt.Println("\n[STEP2] Create a slice containing all digits from the first slice but the last one")
	charSlice2 := charSlice1[:len(charSlice1)-1]

	fmt.Println("\n[STEP2] Result:")
	fmt.Println("[STEP2]", "charSlice1", string(charSlice1))
	fmt.Println("[STEP2]", "charSlice2", string(charSlice2))

	// STEP3: change the numbers 3 and 4 with an X in charSlice2
	fmt.Println("\n[STEP3] Changing the numbers 3 and 4 to the 'X' character in charSlice2")
	charSlice2[2] = 'X'
	charSlice2[3] = 'X'

	fmt.Println("\n[STEP3] Result:")
	fmt.Println("[STEP3]", "charSlice1", string(charSlice1))
	fmt.Println("[STEP3]", "charSlice2", string(charSlice2))

	// STEP4: append a '-' to charSlice2
	fmt.Println("\n[STEP4] Append '-' to charSlice2")
	charSlice2 = append(charSlice2, '-')

	fmt.Println("\n[STEP4] Result:")
	fmt.Println("[STEP4]", "charSlice1", string(charSlice1))
	fmt.Println("[STEP4]", "charSlice2", string(charSlice2))

	// STEP5: append '+*/' to charSlice2
	fmt.Println("\n[STEP5] Append '+*/' to charSlice2")
	charSlice2 = append(charSlice2, '+', '*', '/')

	fmt.Println("\n[STEP5] Result:")
	fmt.Println("[STEP5]", "charSlice1", string(charSlice1))
	fmt.Println("[STEP5]", "charSlice2", string(charSlice2))

	// STEP6: change back the X characters of charSlice2
	fmt.Println("\n[STEP6] replace the X characters with '2' and '3'")
	charSlice2[2] = '3'
	charSlice2[3] = '4'

	fmt.Println("\n[STEP6] Result:")
	fmt.Println("[STEP6]", "charSlice1", string(charSlice1))
	fmt.Println("[STEP6]", "charSlice2", string(charSlice2))
}

But again, the output is not what he was expecting:

[STEP1] Create a slice containing all the decimal digits (1,2,3,4,5,6,7,8,9,0)

[STEP2] Create a slice containing all digits from the first slice but the last one

[STEP2] Result:
[STEP2] charSlice1 1234567890
[STEP2] charSlice2 123456789

[STEP3] Changing the numbers 3 and 4 to the 'X' character in charSlice2

[STEP3] Result:
[STEP3] charSlice1 12XX567890
[STEP3] charSlice2 12XX56789

[STEP4] Append '-' to charSlice2

[STEP4] Result:
[STEP4] charSlice1 12XX56789-
[STEP4] charSlice2 12XX56789-

[STEP5] Append '+*/' to charSlice2

[STEP5] Result:
[STEP5] charSlice1 12XX56789-
[STEP5] charSlice2 12XX56789-+*/

[STEP6] replace the X characters with '2' and '3'

[STEP6] Result:
[STEP6] charSlice1 12XX56789-
[STEP6] charSlice2 123456789-+*/

Wile analyzes what happened:

  1. In Step 3, changing charSlice2 changed charSlice1 too, as expected.
  2. In Step 4, appending a character to charSlice2 changed the last character of charSlice1.
  3. in Step 5, appending 3 characters to charSlice2 didn't change charSlice1.
  4. in Step 6, changing charSlice2 didn't change charSlice1.

Wile: "What is going on? Is this a bug?"

Explanation

I can understand why Wile is confused, but the behavior is expected. He's missing some knowledge of how slices work with append function.

Let's look at the steps of his code one by one.

Step 1

charSlice1 := []byte{'1', '2', '3', '4', '5', '6', '7', '8', '9', '0'}

This code is straightforward but does not exactly do what Wile was expecting: charSlice1 is not a 10-byte array.

Instead, charSlice1 is a view on ten bytes of a ten bytes backing byte array. Its definition is as follows:

type SliceHeader struct {
    Data uintptr
    Len int
    Cap int
}

In that structure:

  • Data is a pointer to the backing array
  • Len is the length of the slice
  • Cap is the size (capacity) of the backing array

That means that, internally, charSlice1 will have these values:

  • Data: pointer to a 10 bytes array
  • Len: 10
  • Cap: 10

Graphically, the situation looks as below:

                               charSlice1 (len: 10, cap: 10)
                                    |
                |---------------------------------------|
                |                                       |
backing_array:  | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 0 |

Step 2

charSlice2 := charSlice1[:len(charSlice1)-1]

Again, this is different from what Wile expects it to do. It isn't copying the array.

That assignment creates a new slice pointing to the same backing array as charSlice1, but reducing the slice length by 1.

The values of the SliceHeader for charSlice2 will be:

  • Data: pointer to the same backing array as charSlice1
  • Len: 9
  • Cap: 10

Since Len is 9, fmt.Println will print only the first nine characters of the array:

                               charSlice1 (len: 10, cap: 10)
                                    |
                |---------------------------------------|
                |                                       |
backing_array:  | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 0 |
                |                                   |
                |-----------------------------------|
                                  |
                             charSlice2 (len: 9, cap: 10)

Step 3

charSlice2[2] = 'X'
charSlice2[3] = 'X'

In this code, we are changing the third and the fourth characters of charSlice2. However, since the backing array for charSlice1 and charSlice2 is the same, charSlice1 will be changed, too.

​
                               charSlice1 (len: 10, cap: 10)
                                    |
                |---------------------------------------|
                |                                       |
backing_array:  | 1 | 2 | X | X | 5 | 6 | 7 | 8 | 9 | 0 |
                |                                   |
                |-----------------------------------|
                                  |
                             charSlice2 (len: 9, cap: 10)

​

Step 4

charSlice2 = append(charSlice2, '-')

Here the exciting part starts: appending a character to charSlice2 change the last character of charSlice1. Why?

Let's see what we have in memory at this point:

                               charSlice1 (len: 10, cap: 10)
                                    |
                |---------------------------------------|
                |                                       |
backing_array:  | 1 | 2 | X | X | 5 | 6 | 7 | 8 | 9 | 0 |
                |                                   |
                |-----------------------------------|
                                  |
                             charSlice2 (len: 9, cap: 10)

As you can see, both charSlice1 and charSlice2 refer to the same backing array. The only difference is that charSlice2 has a length of 9.

When we append a new character to charSlice2, the append function will increase the charSlice2 length and set its last character (the tenth byte of the backing array) with the value to append. Since charSlice1 points to the same backing array as charSlice2, changing the backing array of one slice will also affect the other.

This is how the situation appears now:

                               charSlice1 (len: 10, cap: 10)
                                    |
                |---------------------------------------|
                |                                       |
backing_array:  | 1 | 2 | X | X | 5 | 6 | 7 | 8 | 9 | - |
                |                                       |
                |---------------------------------------|
                                    |
                               charSlice2 (len: 10, cap: 10)

Step 5

charSlice2 = append(charSlice2, '+', '*', '/')

In Step 5, we saw that appending three characters to charSlice2 doesn't change charSlice1 anymore. Let's see why.

In Step 4, we increased the size of charSlice2, so now we have this situation:

                               charSlice1 (len: 10, cap: 10)
                                    |
                |---------------------------------------|
                |                                       |
backing_array:  | 1 | 2 | X | X | 5 | 6 | 7 | 8 | 9 | - |
                |                                       |
                |---------------------------------------|
                                    |
                               charSlice2 (len: 10, cap: 10)

What happens is that when we try to append the three characters to charSlice2, we exceed the capacity of the backing array. To accommodate the new data, append creates a bigger backing array, copies the data from charSlice2 to the new array, and returns a fresh slice pointing to the new backing array.

That means that from now on, charSlice1 is a stale slice and won't be in sync with charSlice2 anymore (since they now refer to 2 different backing arrays).

The new situation is now like this:

                               charSlice1 (len: 10, cap: 10)
                                    |
                |---------------------------------------|
                |                                       |
backing_array:  | 1 | 2 | X | X | 5 | 6 | 7 | 8 | 9 | - |
                |                                       |
                |---------------------------------------|


                               charSlice2 (len: 13, cap: 20)
                                    |
                |-------------------------------------------------------------------------------|
                |                                                                               |
backing_array:  | 1 | 2 | X | X | 5 | 6 | 7 | 8 | 9 | - | + | * | / |   |   |   |   |   |   |   |
                |                                                                               |
                |-------------------------------------------------------------------------------|

Step 6

charSlice2[2] = '3'
charSlice2[3] = '4'

In Step 6, we observe that changing charSlice2 doesn't change charSlice1 anymore.
After reading the previous paragraph, it should be clear why that happens: charSlice2 now points to a different backing array and has no links anymore with charSlice1.

This is how the two slices appear after the assignment:

                               charSlice1 (len: 10, cap: 10)
                                    |
                |---------------------------------------|
                |                                       |
backing_array:  | 1 | 2 | X | X | 5 | 6 | 7 | 8 | 9 | - |
                |                                       |
                |---------------------------------------|


                               charSlice2 (len: 13, cap: 20)
                                    |
                |-------------------------------------------------------------------------------|
                |                                                                               |
backing_array:  | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | - | + | * | / |   |   |   |   |   |   |   |
                |                                                                               |
                |-------------------------------------------------------------------------------|

Conclusion

The append function will try to fit the new data into the current backing array. However, if we exceed the current capacity, append will create a new backing array. That means that when working with different slices generated from the same backing array, we have to pay particular attention when we use it since it could modify other slices or diverge totally.