In one of our weekly office-hours calls, Shuah Khan said something I keep coming back to:
“A blog is a way for you to look back six months from now, a year from now, and ask: what did I work on? You are leaving breadcrumbs for yourself. If a year from now you schedule an interview with somebody and they want to know what you did, this enables you to give a complete answer.”
So this is that blog. Breadcrumbs.
Getting in#
I completed LFD103 - A Beginner’s Guide to Linux Kernel Development - on my own and got genuinely excited about the process. Excited enough to give a presentation on kernel development workflow to COLAB NU, a collaboration community at my university.
Someone in the audience asked whether I had any patches in the Linux kernel. I had to say no. I had learned how contributions worked, but I had not actually made one.
That answer was specific enough to act on. I found the Linux Kernel Mentorship Program and applied with one goal: close that gap. The program received 693 applications for the Spring 2026 cohort; 159 completed the prerequisite tasks. I was one of them.
The prerequisite phase#
The prerequisites were a month of education: building and booting the kernel,
decoding stack traces, writing commit messages, and learning the git + email
workflow that kernel development runs on. No GitHub, no pull requests - just
git format-patch, git send-email, and the mailing lists. That last part took
the most adjustment. I had spent my whole career inside Git forges. Suddenly the
workflow was plain text over email, and every patch, every review comment, every
Reviewed-by: was visible to anyone who cared to look, indefinitely.
It felt clunky at first. By the end it felt honest.
Finding a thread#
Shuah Khan’s first assigned task was simple: fix the CREDITS file. The
comment at the bottom of the file referenced specific people by name - names
that would go stale every time a new entry was added. I sent a patch to
simplify it.
Randy Dunlap’s reply came the next day. The first chunk of my fix had already
been applied in linux-next - I was patching the wrong tree. He told me to
always use the -next tree unless patching something that only applied to
mainline. Simple advice, but it was the kind of thing you only learn by
making the mistake.
I sent a v2. Randy gave an Acked-by:. Shuah then told me to reset the
version tag to [PATCH] - not [PATCH v2] - for the upstream submission,
because the internal review rounds shouldn’t count as public versions. That
lesson stuck: internal and upstream versioning are separate clocks.
The CREDITS patch went upstream. That was my first. But the more interesting
story started when I decided what to work on next.
I wanted to understand why certain kernel selftests failed to build on musl libc. musl is a lightweight, standards-focused alternative to glibc used in Alpine Linux, certain embedded systems, and static-binary toolchains. The kernel’s tools and selftests are supposed to be portable, but they had accumulated quiet glibc assumptions over the years - headers included transitively, macros that only existed in glibc, typedefs that glibc provided but the C standard never required.
Each one of those assumptions was a build failure waiting to happen. Fixing them became my thread for the entire mentorship.
The patches#
Here is the shape of that work:
| Patch | Outcome |
|---|---|
CREDITS: simplify the end-of-file alphabetical-order comment | Merged - Andrew Morton |
KVM: selftests: guard execinfo.h for non-glibc builds | Merged - Sean Christopherson / Paolo Bonzini |
KVM: selftests: elf: include <endian.h> instead of <bits/endian.h> | Merged - Paolo Bonzini |
selftests/rseq: replace glibc-specific __GNUC_PREREQ | Merged - Thomas Gleixner |
KVM: selftests: use KVM’s task pinning APIs in steal_time | Merged - Sean Christopherson |
KVM: selftests: replace ulong with unsigned long | linux-next |
vfio: selftests: include <libgen.h> for basename() | In review |
sys/mman.h: add MADV_COLLAPSE | Merged - musl libc, Rich Felker |
That last one is worth a sentence. While working through musl build failures
in the kernel’s KVM selftests, I hit a missing constant: MADV_COLLAPSE,
added to the Linux kernel in v6.1, was not yet in musl’s sys/mman.h. I
filed it upstream to musl libc. Rich Felker - musl’s maintainer - applied it
and told me he found three more missing flags nearby and added those too.
Contributing to the C library itself was not what I expected when I applied to a Linux kernel mentorship. But that is what the portability thread looked like in practice: fixing the kernel to build on musl, and fixing musl to support the kernel.
The patches that didn’t make it#
Not every patch worked. Two fell away entirely.
In May, I sent tools/include/linux/bitops.h: undef PAGE_SIZE after including limits.h - an attempt to fix a macro collision between musl’s limits.h and
KVM’s PAGE_SIZE definition. The patch was technically correct but the
approach was wrong. It received no replies and I dropped it.
In June, I tried a different angle on the ulong problem:
tools/include/linux/types.h: add ulong typedef for musl compatibility. The
Sashiko AI review bot flagged that adding the typedef to a shared header would
cause -Werror=redundant-decls failures in perf and other host tools. The
bot was right. I dropped the approach and spent the following weeks replacing
ulong directly across all the KVM selftest files instead - a more thorough
fix, but one that required touching thirteen files.
The dropped patches were not wasted. They taught me that the correct fix is often wider than you want, and that taking a shortcut to avoid touching thirteen files usually means the shortcut is wrong.
The back-and-forth#
The most instructive review came early, on the execinfo.h patch.
execinfo.h is a GNU extension - available in glibc but not in musl. Building
KVM selftests with musl-gcc failed because lib/assert.c included it
unconditionally. My first instinct was to guard the include with __GLIBC__
and provide a weak stub for non-glibc builds.
Shuah Khan pushed back. She said adding #ifdefs throughout the code was
generally not the right approach, and pointed me to __weak symbols in the
BPF selftests as a possible pattern.
Then Sean Christopherson weighed in and disagreed with Shuah’s direction:
“I disagree. If we didn’t need the
__GLIBC__#ifdef, then I would be in favor of__weak, but since the#ifdefferyis needed, using an#ifdefand a__weaksymbol is double the ugliness. IMO, the way to make this less ugly is to use a single#ifdefand a local stub.”
Two expert reviewers, two different opinions. I went with Sean’s approach: a
single #ifdef __GLIBC__ guarding both the include and the stack-dumping
logic, with a local stub for non-glibc builds. I sent a v2. Sean applied it
to kvm-x86 fixes on May 19.
What I learned: maintainers are not a monolith. They have different priorities and different aesthetic preferences. When they disagree on your patch, you have to read the reasoning, make a judgment call, and explain your choice. That is engineering, not just code submission.
The best moment#
In July I resent a patch that had been sitting dormant: KVM: selftests: Use KVM's task pinning APIs in steal_time. Sean Christopherson had originally
authored it as part of a musl-compatibility series that was never reposted
after initial feedback. I sent it again, fixed the changelog typos that had
been flagged, and attributed everything correctly.
Sean’s reply came the next day:
“Nope, which means we can simply drop the problematic
pthread_attr_setaffinity_np()… I’ll send a v2. Thanks for posting this, Hisam!”
He posted the revised series himself and merged it into kvm-x86 selftests.
My name is in the commit message as the person who picked it back up.
But the moment I remember most came in August. I was working on the ulong
cleanup - replacing the non-standard ulong typedef across thirteen KVM
selftest files - and Sean asked, in the middle of the review thread, whether
resolving this would finally unblock musl builds of the full KVM selftest
suite against kvm-x86/next. I tested it and gave him a detailed breakdown:
two remaining blockers, where they were, and why each one required __GLIBC__
rather than _GNU_SOURCE.
He wrote back:
“I don’t suppose you know the canonical way for checking for support of glibc-only functionality of this nature?”
I explained that __GLIBC__ is the in-tree idiomatic check - it announces
that glibc is present, rather than merely requesting its extensions. A few
days later, Sean sent the kernel a patch of his own:
KVM: selftests: Use __GLIBC__, not _GNU_SOURCE, to detect actual glibc.
The commit includes:
Suggested-by: Hisam Mehboob <hisamshar@gmail.com>
That tag is one line. But it represents something: a senior kernel maintainer read an explanation from a mentee, found it correct and worth formalizing, and put the mentee’s name in the permanent history of the kernel. That is what the program made possible.
What I learned#
From Shuah#
Two things Shuah told me early have shaped how I work:
First, the TO vs CC rule. Maintainers go in the TO field. Mailing lists go in CC. It sounds obvious until you realize that getting it backwards is the single most common reason a first patch gets no response - not rejection, just silence. She had to repeat this across multiple sessions because new contributors kept making the same mistake.
Second: “Don’t make this process sequential. Just keep sending patches.” I spent too much of March waiting for responses before moving to the next problem. Review takes time. The right posture is to keep the queue full and let feedback come to you.
From the community#
Watching other mentees receive review feedback - including rejections - taught
me as much as getting feedback myself. The most consistent lesson across every
thread I observed: maintainers care far more about the why than the how.
A patch whose commit message explains what broke, why it broke, and why this
fix is correct gets a Reviewed-by:. A technically sound patch with a vague
commit message gets questions.
Write the commit message first. If you cannot explain the problem clearly in prose, you do not understand it well enough to fix it.
Tips for future mentees#
Get TO and CC right. Maintainers on TO, mailing lists on CC. Run
get_maintainer.pland read it carefully. Silent patches are usually addressed to nobody.Find your thread. A coherent theme teaches you more than scattered patches. It also builds a relationship with a specific set of maintainers who start to recognize your name.
The commit message is the patch. Write the why. The code is just the proof.
Dropped patches are not wasted. They tell you the correct approach was wider than you thought.
Revive dead patches. The mailing list is full of correct fixes that got one round of feedback and went silent. Pick one up, fix the notes, send it again.
Test against the actual constraint. For portability work, that means building with musl, not reasoning that the change should work. For bug fixes, reproduce the bug first.
Don’t make it sequential. Keep the queue full.
What’s next#
The mentorship is a milestone, not an endpoint. There are still glibc
assumptions in tools/ and the KVM selftests. The ulong cleanup is in
linux-next. The vfio libgen.h patch is in review. I plan to keep working
through the list.
More than the patches, I want to keep the rhythm: read the mailing list, send patches, take review feedback, iterate. The mechanics are not complicated. The hard part is showing up week after week, long after the program has ended and there is no graduation requirement moving you forward.
Thank you to Shuah Khan for building this program and running it with the same care she gives to the kernel. Thank you to Brigham Campbell for being a co-mentor whose work set a visible standard. Thank you to Sean Christopherson, Paolo Bonzini, Thomas Gleixner, Andrew Morton, Randy Dunlap, David Matlack, and Rich Felker for reading patches from an unfamiliar name and engaging seriously with them.
The program took me from “kernel development is something other people do” to having my name in the commit history of the Linux kernel and musl libc. That is not a small thing.
