Use cilium/ebpf to compile and load TC BPF code
5 min read

Use cilium/ebpf to compile and load TC BPF code

Introduction

cilium/ebpf is one of the best Golang BPF libraries currently on the market. It can read, modify and load eBPF programs and maps and attach them to various hooks in the Linux Kernel.

Compared to dropbox/goebpf, another Golang eBPF library, cilium/ebpf is advanced in compiling eBPF code directly from Go’s go generate ability. The library provides a handy tool called bpf2go that manages the compiling process of eBPF code as well as the generation of Golang bindings. Due to the generation of Golang bindings, its user doesn’t need to even know where the eBPF objects reside in. It conceals the detail of managing elf object files and gives its user the Golang interface to interact with directly.

Sample TC BPF code

We use the snippet below to illustrate the process of composing, compiling, and loading a TC BPF program. The BPF program is written in C language, and for the curious readers who wonder how a BPF code is compiled into an ELF file, I will give the equivocal llvm command for building.

The sample code can be found in my Github Repo:

https://github.com/d0u9/blog_samples/tree/master/bpf/cilium_ebpf_basic

Helper Header Files

BPF framework has provided a bunch of helper functions that can be invoked in BPF code directly to call for some specific OS abilities, such as redirecting a packet to another interface. The full list of these functions can be found in man 7 bpf-helpers manual. It is worth mentioning that the list varies from version to version of the Linux Kernel. So it is better to consult your kernel’s source code before taking it.

The good news is that the Linux Kernel provides a script to generate the C header files specific to that version automatically:

# Run this command at the root of Linux Kernel source.

make -C tools/lib/bpf

Run the command above at the root of your kernel source, and the header files will be generated as bpf_helper_defs.h in tools/lib/bpf directory. User should use tools/lib/bpf/bpf_helpers.h header file instead of referencing bpf_helper_defs.h directly. The tools/lib/bpf/bpf_helpers.h file defines many macros and constants that are essential to any BPF code, and it includes bpf_helper_defs.h as well.

#include <linux/bpf.h>
#include <linux/pkt_cls.h>
#include "include/helpers.h"

SEC("tc_prog")
int tc_main(struct __sk_buff *skb)
{
	char hello_str[] = "hello egress pkt";
	bpf_trace_printk(hello_str, sizeof(hello_str));
	return TC_ACT_OK;
}

char __license[] SEC("license") = "GPL";
bpf/bpf.c

Save the snippet above as bpf.c, and then compile it into elf object file directly by llvm :

clang -O2 -emit-llvm -c bpf.c -o - | llc -march=bpf -filetype=obj -o bpf.o

Inspect the output binary file:

file bpf.o

## Output
bpf.o: ELF 64-bit LSB relocatable, eBPF, version 1 (SYSV), not stripped

Load the TC object binary by tc command

It is handy to attach a TC BPF binary to a specific interface by using tc command manually. We have to create a clsact Qdisc for our target interface firstly:

tc qdisc show dev eth0

Then, attach our BPF binary to that interface’s ingress direction as a classifier.

tc filter add dev eth0 ingress bpf object-file bpf.o section tc_prog direct-action

Check if it is successful:

tc filter show dev eth0 ingress

To view the log message dumped by BPF code, read /sys/kernel/debug/tracing/trace_pipe file by cat or other tools.

Compile BPF code by bpf2go command

Prepare

Create a new Golang module, and put the BPF C files in bpf subdirectory.

mkdir cilium_ebpf_basic
cd cilium_ebpf_basic
go mod init ebpf
go get github.com/cilium/ebpf/cmd/bpf2go

The Golang Part

// main.go
package main

//go:generate go run github.com/cilium/ebpf/cmd/bpf2go tc bpf/bpf.c -- -I./bpf
cilium_ebpf_basic/main.go

Generate BPF object files by running go generate. Then, Four new files can be found in the same directory named as ebpf_bpfeb.go, ebpf_bpfeb.o, ebpf_bpfel.go and ebpf_bpfel.o respectively. The last char, e or b, of file names stands for the byte-order the file caters for. The .go files contain bindings to BPF programs and maps. The .o files are BPF ELF object binaries.

Load BPF into the Kernel

It is time to write our main function which attaches the BPF objects we just generated to the interface. The same as above, the ingress direction of eth0 is chosen as our target.

Load BPF programs and maps in BPF object binary into the kernel is pretty simple:

// Load bpf programs and maps into the kernel
objs := tcObjects{}
if err := loadTcObjects(&objs, nil); err != nil {
    log.Fatalf("loading objects: %v", err)
}
defer objs.Close()
cilium_ebpf_basic/main.go

The tcObjects type and loadTcObjects() function are generated by bpf2go tool. loadTcObjects() function loads the programs and maps in BPF object file into the kernel. This function will fail if the BPF code cannot pass the verification of the BPF verifier in the kernel space.

Every BPF program and map loaded into the kernel are granted a handler, i.e. a file descriptor, which is an identifier to that BPF object. The file descriptor is an integer index the same as an opened file. For our scenario, to get the file descriptor of our tcObjects, use the line of code below.

progFd := objs.tcPrograms.TcMain.FD()

or

progFd := objs.TcMain.FD()

Attaching the BPF program to a tc classifier is different from attaching BPF program to Kprobe point. To attach a BPF program to an interface, we have to ask netlink for help.

netlink is a communication method provided by the Linux Kernel to bridge the userspace and kernel space for high-efficiency data transfer. ip command and its relatives, e.g. tc command, are netlink based.  To Attach our BPF program to the eth0 interface, we need to take advantage of netlink as well.

There has already been a mature Golang library named netlink that provides almost full functional netlink abilities to communicate with the kernel.

go get github.com/vishvananda/netlink

As we have operated before, a clsact Qdisc must be created as an attach point. If you followed my doc from the beginning, the clsact Qdisc should have been created. If so, please delete it to prevent getting errors afterward:

tc qdisc del dev eth0 clsact

Then, the Golang code. I have commented on the code detailedly to tell the process.

// Get the netlink handler of eth0 interface.
// ETH_NAME is defined as a const:
//     const ETH_NAME = "eth0"
eth0, err := netlink.LinkByName(ETH_NAME)
if err != nil {
    log.Fatalf("cannot find %s: %v", ETH_NAME, err)
}


// Declare the Qdisc attributes.
// These attributes describes how our Qdisc will be added.
attrs := netlink.QdiscAttrs{
    LinkIndex: eth0.Attrs().Index, 	// Interface Index
    Handle:    netlink.MakeHandle(0xffff, 0),
    Parent:    netlink.HANDLE_CLSACT,
}

// Then, declare the Qdisc
qdisc := &netlink.GenericQdisc{
    QdiscAttrs: attrs,
    QdiscType:  "clsact",
}

// Add our clsact Qdisc
if err := netlink.QdiscAdd(qdisc); err != nil {
    log.Fatalf("cannot add clsact qdisc: %v", err)
}

// Create tc filter attributes
filterAttrs := netlink.FilterAttrs{
    LinkIndex: eth0.Attrs().Index,
    Parent:    netlink.HANDLE_MIN_INGRESS,   // The direction
    Handle:    netlink.MakeHandle(0, 1), 
    Protocol:  unix.ETH_P_ALL,
    Priority:  1,
}

// Declare the BPF filter
filter := &netlink.BpfFilter{
    FilterAttrs:  filterAttrs,
    Fd:           progFd, 	// This filed links our bpf program with netlink library
    Name:         "hi-tc",
    DirectAction: true,
}

// Add filter
if err := netlink.FilterAdd(filter); err != nil {
    log.Fatalf("cannot attach bpf object to filter: %v", err);
}
cilium_ebpf_basic/main.go

Put all together, compile and run

The complete code of main.go is pasted below with comments stripped:

package main

import (
        "log"

        "github.com/vishvananda/netlink"
        "golang.org/x/sys/unix"
)

//go:generate go run github.com/cilium/ebpf/cmd/bpf2go tc bpf/bpf.c -- -I./bpf

const ETH_NAME  = "eth0"

func main() {
    var err error

    objs := tcObjects{}
    if err := loadTcObjects(&objs, nil); err != nil {
        log.Fatalf("loading objects: %v", err)
    }
    defer objs.Close()

    progFd := objs.tcPrograms.TcMain.FD()

    eth0, err := netlink.LinkByName(ETH_NAME)
    if err != nil {
        log.Fatalf("cannot find %s: %v", ETH_NAME, err)
    }

    attrs := netlink.QdiscAttrs{
        LinkIndex: eth0.Attrs().Index,
        Handle:    netlink.MakeHandle(0xffff, 0),
        Parent:    netlink.HANDLE_CLSACT,
    }

    qdisc := &netlink.GenericQdisc{
        QdiscAttrs: attrs,
        QdiscType:  "clsact",
    }

    if err := netlink.QdiscAdd(qdisc); err != nil {
        log.Fatalf("cannot add clsact qdisc: %v", err)
    }

    filterAttrs := netlink.FilterAttrs{
        LinkIndex: eth0.Attrs().Index,
        Parent:    netlink.HANDLE_MIN_INGRESS,
        Handle:    netlink.MakeHandle(0, 1),
        Protocol:  unix.ETH_P_ALL,
        Priority:  1,
    }

    filter := &netlink.BpfFilter{
        FilterAttrs:  filterAttrs,
        Fd:           progFd,
        Name:         "hi-tc",
        DirectAction: true,
    }

    if err := netlink.FilterAdd(filter); err != nil {
        log.Fatalf("cannot attach bpf object to filter: %v", err);
    }
}
cilium_ebpf_basic/main.go

Compile and run it on your Linux host:

go generate && go build -o a.out main.go tc_bpfel.go && sudo ./a.out

References

https://pkg.go.dev/github.com/newtools/ebpf
https://pkg.go.dev/github.com/vishvananda/netlink