Load pinned maps littered in different directories for BPF programs by cilium/ebpf library
6 min read

Load pinned maps littered in different directories for BPF programs by cilium/ebpf library

Introduction

I previously used the foniod/redbpf library, implemented in Rust language, as a tool collection to write BPF programs using Rust. It has a handy function that replaces a map in an ELF with another by the map’s name.

pub fn replace_map(&mut self, map_name: &str, new: Map) -> Result<&mut Self>

However, to fulfill the same task by using the cilium/ebpf library is filled with pain. There is no equivalent function in the cilium/ebpf library that has the same ability to replace a map with another. And that is why I verbose all the words in this article.

Maps are pinned in the same directory

This is the simplest scenario for replacing maps. For example, one BPF program references two maps pinned in the same directory.

The generated loading function by the cilium/ebpf library has an argument that has a type of *CollectionOptions. There are various configurable options in this type that controls how a program/map loads. Among these control options, a field named PinPath exists. This option tells the BPF loader where the pinned map resides if the map is declared with LIBBPF_PIN_BY_NAME.

I posted the sample code here:

blog_samples/bpf/cilium_ebpf_pinned_map/pin_to_same_dir at master · d0u9/blog_samples
Contribute to d0u9/blog_samples development by creating an account on GitHub.

In this example, two maps, map_1 and map_2, are declared with pinning flag set to LIBBPF_PIN_BY_NAME:

struct {
        __uint(type, BPF_MAP_TYPE_ARRAY);
        __type(key, __u32);
        __type(value, __u64);
        __uint(max_entries, 10);
        __uint(pinning, LIBBPF_PIN_BY_NAME);
} map_1 SEC(".maps");
pin_to_same_dir/../include/maps/map_1.h
struct {
        __uint(type, BPF_MAP_TYPE_ARRAY);
        __type(key, __u32);
        __type(value, __u64);
        __uint(max_entries, 10);
        __uint(pinning, LIBBPF_PIN_BY_NAME);
} map_2 SEC(".maps");
pin_to_same_dir/../include/maps/map_2.h

Then we reference these two maps in the BPF program:

SEC("kprobe/sys_execve")
int prog_a(struct __sk_buff *skb)
{
	__u32 key = 2;
	__u64 initval = 1, *valp = NULL;
	valp = (__u64 *)bpf_map_lookup_elem(&map_1, &key);
	if (!valp) {
		bpf_map_update_elem(&map_1, &key, &initval, BPF_ANY);
	} else {
		__sync_fetch_and_add(valp, 1);
	}

	valp = (__u64 *)bpf_map_lookup_elem(&map_2, &key);
	if (!valp) {
		bpf_map_update_elem(&map_2, &key, &initval, BPF_ANY);
	} else {
		__sync_fetch_and_add(valp, 1);
	}

	return 0;
}
pin_to_same_dir/bpf/bpf.c

We have specified these two maps are LIBBPF_PIN_BY_NAME, so a pinning directory must be provided or the program loading will fail. The generated interface function in prog_bpfel.go (or prog_bpfeb.go if on a big-endian host) as a prototype of:

func loadProgObjects(obj interface{}, opts *ebpf.CollectionOptions) error {} 

Since we want these two maps to be pinned in the same directory, the PinPath option can be set once and the same:

opt := ebpf.CollectionOptions{
    Maps: ebpf.MapOptions{
        // Note: The pin file must be placed in `bpf` filesystem.
        PinPath: PIN_PATH,
   },
}

It is worth mentioning that all pin files of maps and programs must be in a filesystem of a particular type, bpf. Usually, a bpf filesystem is mounted on a path of /sys/fs/bpf for a typical Linux distribution. Curious readers can verify this by running the command below:

df -a | grep bpf 

Build and run this code:

 make && sudo ./a.out

The pin files are created in the directory of /sys/fs/bpf. Use the ls command to check it out.

Maps are pinned in the same directory

It is hard for a production environment BPF programs to pin all maps in the same directory. Usually, the maps are pinned to different directories according to some hierarchy designs. We update the sample code in the previous section by adding another two maps, the map_3 and map_4. The map_3 is pinned in /sys/fs/bpf/map3_dir/, and the map_4 is pinned in /sys/fs/bpf/map4_dir/.

The complete code can be found in my repo:

blog_samples/bpf/cilium_ebpf_pinned_map/replace_with_another_map at master · d0u9/blog_samples
Contribute to d0u9/blog_samples development by creating an account on GitHub.

Create pinned map

We have to prepare the pinned maps before starting code. bpftool, a handy tool shipped with the Linux Kernel, is a collection of commands which manipulates different BPF objects in the system, such as maps, programs, net, BTF, etc. Some attention must be taken seriously to create our pinned map, including the key and value size and the number of entries the map can store.

The prototypes of map_3 and map_4 are:

struct {
	__uint(type, BPF_MAP_TYPE_ARRAY);
	__type(key, __u32);
	__type(value, __u64);
	__uint(max_entries, 10);
	__uint(pinning, LIBBPF_PIN_BY_NAME);
} map_3 SEC(“.maps”);

struct {
	__uint(type, BPF_MAP_TYPE_ARRAY);
	__type(key, __u32);
	__type(value, __u32);
	__uint(max_entries, 10);
	__uint(pinning, LIBBPF_PIN_BY_NAME);
} map_4 SEC(".maps");

These two maps are identical except for the type of key. map_3 has a value type of __u64 but __u32 for map_4. The difference here is reflected in the different sizes of key types of the map:

bpftool map create map_3 type array key 4 value 8 entries 10 name map_3
bpftool map create map_4 type array key 4 value 4 entries 10 name map_4

The Golang Code

It is not as simple as using one function to replace maps in the BPF program with another one. A few steps must be taken to toward this purpose.

First, load the specification of the BPF binary by using functions generated by go2bpf. For our example here, the function is loadProg().

specs, err := loadProg()

The CollectionSpec structure contains descriptions of both map and program. The specifications are parsed directly from the BPF ’s ELF file and structured into Golang types. The funny part of the program specification is that the whole BPF assembly codes are stored in the field of Instructions in ProgramSpec. We will reference this field later when detailing the mechanism of map replacement later.
The CollectionSpec type has a member function called RewriteMpas:

func (cs *CollectionSpec) RewriteMaps(maps map[string]*Map) error {}

This function is the key to our problem of map replacement. This function accepts a map whose key is a string, and a value is *Map object. The *Map object can be created via different functions in many ways.

Here, we use the function NewMapWithOptions() to load and create map object. It is also the function used by generated interface functions behind the scenes:

opt := ebpf.MapOptions{
    PinPath: path.Join(BPF_FS, "map4_dir"),
}
map3, err := ebpf.NewMapWithOptions(specs.Maps["map_4"], opt);

opt := ebpf.MapOptions{
    PinPath: path.Join(BPF_FS, "map3_dir"),
}
map4, err := ebpf.NewMapWithOptions(specs.Maps["map_3"], opt);

The MapOptions mentioned in the previous section specifies the pin directory in which the loader searches for a pinned map. We fill the PinPath field with the parent directory path.

After that, we hold two map objects in hand, which the BPF program can reference. The replacement happens after providing a rewriteMap in which pairs of map name and map object are specified:

rewriteMap := map[string]*ebpf.Map {
    "map_3": map3,
    "map_4": map4,
}

Then pass this map as a parameter to RewriteMaps() function:

err := specs.RewriteMaps(rewriteMap)

This function deletes map specifications mentioned in rewriteMap from the collection. And replace it with the file description of opened BPF map. Then it is time to load the BPF into the Kernel by the collection specification modified before:

objs := progPrograms{}
opts := ebpf.CollectionOptions {
    Maps: ebpf.MapOptions {
		PinPath: BPF_FS,
    },
}
specs.LoadAndAssign(&objs, &opts)

We meet the CollectionOptions again. This time, the option controls which directories the remaining maps are pinned in. Finally, load all specs by function LoadAndAssign().

How the map replacement works

Curious readers must want to know what is going on when the RewriteMap() function is invoked. How does the BPF program reference another opened map instead of creating a new one? The answer is simple, the file descriptor.

Any successfully loaded BPF object is attached with a file descriptor that acts as a handler or a reference to that object. The implementation of map replacement is simple: iterate over each BPF instruction in the BPF program, test if it is a load instruction, and replace the operand with the file descriptor value.

Remember the Instruction filed in the program specification mentioned before? Yes, we operate it directly.

References

https://github.com/cilium/ebpf/blob/3418f341ac0460cf4164146e0c1ec6c40156b4fc/asm/instruction.go#L139