As a software engineer, there comes a time when you encounter a problem that a single thread or process might not be enough. So you decide to solve it with multi-threading, or in Go’s case, goroutines. Although, Go makes it easier to create concurrent programs compared to other languages, debugging them is still a big pain. In this post, we will explore a tool that can help us debug them, called Delve debugger.

Debugging with print statements (the ugly way)

Before exploring the Delve debugger, let’s see how most of us debug programs. Let us see the printf debugging technique. For example, let’s assume the following simple Go program, which divides two numbers and prints them to the standard output.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

package main

import (
	"fmt"
)

func main() {
	fmt.Printf("100/2=%d\n", div(100, 2))
}

func div(a, b int) int {
	res := a / a
	return res
}

When we run the program it will print the following output:

100/2=1

The program prints the wrong result. For most, the error would be obvious, but let’s see how someone would debug it using print statements.

11
12
13
14
15
16
17

func div(a, b int) int {
	fmt.Printf("a=%d", a)
	fmt.Printf("b=%d", b)
	res := a / a
	fmt.Printf("res=%d\n", res)
	return res
}

When debbuging with print statements, we usually end up adding many of them in our code to verify the variables, and as you can see our code quickly becomes cluttered.

Now let’s run our program again.

Output:

a=100
b=2
res=1
100/2=1

We have printed on the console the function arguments, as well as the local value res. We see that although the args have been passed corectly, the variable res doesn’t hold the expected value. Using the print statement debugging technique, we managed to pinpoint the culprit. In our case, it is line 14.

Enter the goroutines

That was pretty easy, now let’s add some goroutines and try the same technique. We are going to modify the code so that it will work with goroutines. We are using a WaitGroup to sync our program and so that the maing goroutine does not exit before the goroutines we have spawned finished their job. We are just printing the result on the console with the fmt package to keep it as simple as possible, although it’s not safe and you should never do this.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

package main

import (
	"fmt"
	"sync"
)

func main() {
	var wg sync.WaitGroup
	for i := 1; i <= 5; i++ {
		wg.Add(1)
		go div(100, i, &wg)
	}
	wg.Wait()
}

func div(a, b int, wg *sync.WaitGroup) {
	defer wg.Done()
	res := a / a
	fmt.Printf("%d/%d=%d\n", a, b, res)
}

If we run the program, we see the same bug again:

100/5=1
100/2=1
100/1=1
100/4=1
100/3=1

Now let’s add some print statements to debug the goroutines.

17
18
19
20
21
22
23
24

func div(a, b int, wg *sync.WaitGroup) {
	defer wg.Done()
	fmt.Printf("a=%d\n", a)
	fmt.Printf("b=%d\n", b)
	res := a / a
	fmt.Printf("res=%d\n", res)
	fmt.Printf("%d/%d=%d\n", a, b, res)
}

If we run the program after we have added the print statements, we get the following output:

a=100
b=5
res=1
100/5=1
a=100
b=4
res=1
100/4=1
a=100
b=2
res=1
100/2=1
a=100
b=1
res=1
100/1=1
a=100
b=3
res=1
100/3=1

As you can see, now that we introduced goroutines debugging with print statements has become super hard. Even for such a small program as the above. The output order from the print statements is pretty much random. And because of that, it is hard to figure out which goroutine printed it.

A few more reasons to avoid debugging with print statements:

  • the fmt package is not thread-safe
  • debugging with print statements makes the whole debugging process slower since you have to recompile each time you add a new print statement.
  • after you are done with debugging, it is easy to forget some print statements in your code.

Debugging with Delve (the good way)

Now let’s debug our goroutines with Delve. To use the Delve debugger, we’ll have to install it first. I installed on my Linux system with the go get method:

go get github.com/go-delve/delve/cmd/dlv

If you have a different system, you can check the installation process for your system here.

We are going to debug the program with the goroutines we introduced earlier:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

package main

import (
	"fmt"
	"sync"
)

func main() {
	var wg sync.WaitGroup
	for i := 1; i <= 5; i++ {
		wg.Add(1)
		go div(100, i, &wg)
	}
	wg.Wait()
}

func div(a, b int, wg *sync.WaitGroup) {
	defer wg.Done()
	res := a / a
	fmt.Printf("%d/%d=%d\n", a, b, res)
}

Now that you have installed Delve successfully, you can load the program into Delve with dlv debug main.go, and you should be able to see the (dlv) prompt. Here you can use run commands to help you debug your Go program. You can run help to see the options Delve offers.

The most common option you will use is break, which sets a breakpoint. And you can list all your breakpoints with breakpoints. For example, let’s set a breakpoint at line 19 in our code.

(dlv) break main.go:19
Breakpoint 1 set at 0x4ba9cf for main.div() ./main.go:19

To run our program until the next breakpoint (or until termination, if none is set), we can use continue.

(dlv) continue
> main.div() ./main.go:19 (hits goroutine(7):1 total:3) (PC: 0x4ba9cf)
> main.div() ./main.go:19 (hits goroutine(8):1 total:3) (PC: 0x4ba9cf)
> main.div() ./main.go:19 (hits goroutine(10):1 total:3) (PC: 0x4ba9cf)
    14:		wg.Wait()
    15:	}
    16:	
    17:	func div(a, b int, wg *sync.WaitGroup) {
    18:		defer wg.Done()
=>  19:		res := a / a
    20:		fmt.Printf("%d/%d=%d\n", a, b, res)
    21:	}

Delve is now running our program and has paused at the breakpoint on line 19. We can see that Go has created three goroutines, all of which have halted at line 19. The entire Go program was paused at that point time, and that is why the main goroutine didn’t manage to spawn the rest of the goroutines. Delve has also printed the source code near that line.

Here we have many options. For example, we can list the function’s arguments with the command args.

(dlv) args
a = 100
b = 5
wg = (*sync.WaitGroup)(0xc00001a0b0)

You can print the local variables with locals.

(dlv) locals
(no locals)

We see that there is no local variable in the div function yet. That’s because we are at line 19, and the command at that line hasn’t been executed yet. Let’s execute it by stepping over to the next line with next.

(dlv) next
> main.div() ./main.go:20 (PC: 0x4baa01)
    15:	}
    16:	
    17:	func div(a, b int, wg *sync.WaitGroup) {
    18:		defer wg.Done()
    19:		res := a / a
=>  20:		fmt.Printf("%d/%d=%d\n", a, b, res)
    21:	}

Ok, let’s check the local variables now.

(dlv) locals
res = 1

Using goroutines we can list the goroutines of the program.

(dlv) goroutines
  Goroutine 1 - User: /usr/local/go/src/runtime/sema.go:56 sync.runtime_Semacquire (0x46a725)
  Goroutine 2 - User: /usr/local/go/src/runtime/proc.go:307 runtime.gopark (0x43b835)
  Goroutine 3 - User: /usr/local/go/src/runtime/proc.go:307 runtime.gopark (0x43b835)
  Goroutine 4 - User: /usr/local/go/src/runtime/proc.go:307 runtime.gopark (0x43b835)
  Goroutine 5 - User: /usr/local/go/src/runtime/mfinal.go:161 runtime.runfinq (0x41a460)
  Goroutine 6 - User: ./main.go:17 main.div (0x4ba960)
  Goroutine 7 - User: ./main.go:19 main.div (0x4ba9d7) (thread 559883)
  Goroutine 8 - User: ./main.go:20 main.div (0x4baa01) (thread 559881)
  Goroutine 9 - User: ./main.go:17 main.div (0x4ba960)
* Goroutine 10 - User: ./main.go:20 main.div (0x4baa01) (thread 558969)
[10 goroutines]

Delve has not only printed our main program’s goroutine, but also the Go runtime goroutines. The goroutine we are currently inspecting has an asterisk symbol * prepended. In our case, it is Goroutine 10. We can switch to another goroutine using the command goroutine.

(dlv) goroutine 8
Switched from 10 to 8 (thread 559881)

Now that we switched to Goroutine 8 let’s print the function arguments.

(dlv) args
a = 100
b = 3
wg = (*sync.WaitGroup)(0xc00001a0b0)

We see that this goroutine has different args. But if we print the local variable res we will see that it has the same value:

(dlv) locals
res = 1

One last command we are going to see is set. This is a very powerful and useful command, since it let’s you change the value of your program at runtime.

(dlv) set res=10
(dlv) locals
res = 10

Conclusion

Debugging concurrent programs and goroutines can get very complicated, and because of that, you should use all the tools available that can make the whole process easier. Delve offers many great features for debugging your Go programs, and it will help you a lot when debugging.

Delve also has many integrations with editors, such as GoLand (which I currently use), VSCode, Vim, etc. There is also a nice (if you prefer that) GUI interface for Delve, called Gdlv.

Feel free to share below in the comment section how you are debbuging your Go programs.