Cannot write in /sys/kernel/debug/tracing/kprobe_events

Adalann

Hi everyone,

I'm wondering if it's possible in Solus to enable debug probes for kernel modules ? I'm playing with the Wireguard VPN protocol and I'd like to be able to see the decrypted packets in wireshark. Long story short, it's possible by using a tool provided by the Wireguard maintainer, tool that I can compile without any problem so the Wireguard side of things seems ok.

Things starts to get complicated when I try to launch this tool, since the script is tring to append the needed probes in the file /sys/kernel/debug/tracing/kprobe_events via echo, and it fails (write error, No such file or directory), despite the fact that file /sys/kernel/debug/tracing/kprobe_events actually returns something (empty).

I did some digging around, and here is what I know :

the user must root (of course)
degubfs needs to be mounted
the option CONFIG_KPROBE_EVENTS needs to be set to "y" at the kernel compilation time, to enable kprobes

debugfs is actually mounted, and I checked the content of /usr/src/$(uname -r)/.config, and it indicates that CONFIG_KPROBE_EVENTS=y.

Does anyone have an idea why /sys/kernel/debug/tracing/kprobe_events is not writable by the script ? I'd be really grateful if someone had an clue ! This may not be related specificly to Solus but still, maybe someone have something in mind to help !

Regards

YakoYakoYokuYoku

This is what I can scratch from the (bit fragmented for me) documentations about eBPF.

The /sys/kernel/debug/tracing/kprobe_events pseudo-file is writable indeed, but there are conditions for a successful write.

The supplied arguments for kprobe_events must follow the correct syntax.

# This adds a probe to the do_sys_open event, i.e. this will record the open(2) syscall arguments
$ echo 'p:myprobe do_sys_open dfd=%ax filename=%dx flags=%cx mode=+4($stack)' > /sys/kernel/debug/tracing/kprobe_events

The event that we want to probe must be available and enabled for probing.

$ vim /sys/kernel/debug/tracing/available_events

On a failed write the kernel will promptly return ENOENT or another errno.

AFAICT Idk how to enable those events for tracing yet, but hopefully a good analysis on the kernel documentation (or elsewhere) could yield the answer.

YakoYakoYokuYoku

Narrowed down to the following.

Although that kprobing events is (obviously) possible in Solus, I've couldn't find a way to enable the wireguard events nor confirm their availabilty in debugfs. Following the Wireshark guide on WireGuard it mentions that is required to build an out-of-tree module (with the source already included in the wireguard-tools package) and extracting .rodata values from it (?), but anything apart (as you've mentioned that you could compile the requirement from the extract-handshake.sh tool) from that it does not mention anything beyond (both the Wireshark wiki and the WireGuard key extractor README), no kernel configs, any further tweaks in things like debugfs or procfs, so I'm afraid that this is as far as those docs can go. OTOH the WireGuard developers can be contacted regarding this.

Adalann

Sorry for not ansering sooner...
First up, thank you for your replies. I did not had time to go deeper on this subject for now but I'm still interessed on making this tool work.

Yes, reading the little documentation on the subject it seems to not require any kind of tweaking on the kernel side and should "simply run", this is why it's a little frustrating (and it becomes even more interesting at the same time 😛).

I need to try this again and it may be worth the shot to get in touch with the WireGuard teams, I agree. I'll post back here if I find anything.

zombieWalker

I was combing the internet trying to find solution to this issue.
The author made an assumption that every OS/ kernel will use the same symbol.
See https://git.zx2c4.com/wireguard-tools/tree/contrib/extract-handshakes/extract-handshakes.sh#n46
The symbol "index_hashtable_insert" is also nowhere to be found in the symbols table /proc/kallsyms.
Try replacing that with something that can be found in the table. It could be "wg_index_hashtable_insert" or something else.
Digging deeper, I can confirm that at least in my current version of linux and many others, wg_index_hashtable_insert is the only symbol I can find in /drivers/net/wireguard/ of the linux source.
Note: This may not even work for you if you're using userspace implementations. You'd have to find the appropriate functions/ method to debug/ trace. 😆