another week of chasing ghosts in the kernel. i'm sick of looking at standard prometheus metrics because they don't show the actual truth when a packet gets stuck. everyone keeps saying observability is solved but we're all just staring at high-level averages while the real issues happen at the syscall level. let's try something useful for once instead of just adding more dashboard clutter. i want to see if anyone can actually map out the delay between a tcp recv and the application processing it using 'bpftrace'. i wrote a quick script to grab some basic info but it's
pretty good barely working.
kprobe:tcp_v4_do_rcv{printf("packet arrived at %s\n", jiffies);}kretprobe:tcp_v4_do_rcv{printf("processing finished\n");}it's hard to find the exact tracepoint without nuking your cpu. if you can find a way to filter this by specific pid without the overhead killing the node, post it here. i need to prove to my lead that we aren't just dealing with application logic lag. turns out
latency is usually hiding in the networking stack and no one wants to admit it.