Keith Cirkel Software Cyber Shepherd

100 patches to 5 browsers in 18 months

Over the last two years I've been increasingly contributing to web browsers. I recently made my 100th patch to browser engines, which I consider somewhat of a milestone. It's rewarding and enjoyable, and it's something others seem interested by, as I'm frequently asked about it. It seems to have even taken some browser engineers by surprise: what's my deal? Why all the contributions? Am I looking for a job? I read the other day that if you had to repeat something 3 times you should write an article (that's an extremely niche reference that might get a chortle from someone). Hopefully there's some advice scattered throughout this diatribe for those aspiring to do the same.

The backstory

First the history; I've been working at GitHub since 2017, spending my first 5 years on the "Web Systems" team. This team managed essentially all of the JavaScript that was served to users on github.com. It has been one of the best teams I've ever worked on. Aside from the privilege of being surrounded by colleagues smarter than me, it was great to work on a team where we were all incredibly well aligned on our goal to ship as little client side code as possible to get the job done. We regularly looked for opportunities to remove code where we could, such as finding native functionality to lean on, polyfilling new features and removing old libraries, or carefully cutting back on code to support older browsers. We tried to promote new and upcoming browser features and maintained lists of browser bugs to watch for when we could remove fixes, and regularly spoke to the browser vendors about incorporating new features (fingers still crossed on a browser native emoji picker). We minimised the amount of external dependencies, at the time I think we had two third party dependencies that weren't authored by us. This wasn't because we thought we could write better code, but because it let us aggressively pare down code that didn't serve us, and kept us nimble because we only had to answer to ourselves. Even as recently as 2019 the entirety of GitHub's JavaScript, compressed, would fit inside a 1.44mb floppy disc. Small enough to internalize but large enough that we didn't want to accumulate much more complexity - we were incentivized to offload new problems. This let us think deeply about how problems got solved, and (because the entire teams output wasn't spent managing hundreds of thousands of lines of JavaScript) we had the time to make more strategic fixes to problems. One example of this; one of my team-mates - Mu-An - discovered a focus bug in Firefox was biting us and rather than introduce more JavaScript to work around this, Mu-An and our other team mate David just... fixed it. I didn't know you could just do that. I mean, I knew the major web browsers were all open source but they were intimidating and had hundreds of engineers working on them and didn't need to waste their time on fixing our issues... or did they? This piqued my interest, and Mu-An graciously guided through fixing a different focus bug we found.

Gradually, my colleagues on the Web Systems team left. Josh in 2020, David in 2021, Mu-An in 2022, Mislav & Kristján in 2023. Each departure re-affirmed my responsibility to retain some of this team's DNA and continue to extol the virtues of managing small teams with tight responsibilities, using The Web Platform as the foundational core. So I set myself a mission: contribute some code to all three of the major browser engines. There was just one problem. I didn't know a lick of C++ and all three major browsers were built on massive, sprawling C++ codebases. The first C++ I had ever altered was in that tiny focus bug patch I'd written a couple of years prior. Even without the barrier of an entirely new language to learn, these are still gigantic codebases with millions of lines of code. It was intimidating.

Luckily my time at GitHub has equipped me for this. Prior to joining, I had positioned myself as a front-end expert, specialising in JavaScript only. My colleagues at GitHub quickly dismantled the barriers of single discipline development. I fondly remember the first task given to me as a new employee back in 2017: fix our semver ordering on the releases page - the library used for this was written in C. I spoke to my manager at the time, and told them I probably wasn't the best person to fix this as I'd never written C before, an unconvincing argument I'd find out. GitHub is a place with lots of code written in lots of languages: Ruby, Go, C, Haskell, Rust, JS - there was a bug that needed fixing and I was given the support to fix it, but the programming language wasn't going to be a barrier. This wasn't the only time this sort of thing happened; in the 7 years that I've been at GitHub I've had to shift out of my comfort zone a lot. These days I regularly flip between Ruby and JavaScript with the occasional forays into C, Golang, and Rust, to complete my tasks for the quarter. Maybe jumping into a large C++ codebase isn't the barrier I thought it was?

The second important factor was that my team were already involved in standards. We'd attended TC39 meetings before, so I was quite familiar with the web, and with web standards, and the politics of those things. This meant I could slip somewhat effortlessly into new working groups and knew when it was a good idea to shut up and listen, but also knew when it was okay to talk. In my opinion it's important to ramp up slowly in a working group, do some ground work and build up to the bigger pieces. A fantastic way to establish yourself in a working group is to volunteer to take minutes (be sure to observe a few meetings first to understand the process). Taking meeting minutes is a job very few want to do, but it's necessary for each meeting and for newcomers it's a great way to learn the names of who is speaking, learn how people speak about problems, and learn the procedures of the working group. While I've met a never ending list of lovely and accommodating people during standards work, the work is difficult and nuanced and there's a process to get work done; coming in hot with your own style, your own ideas, and your own way of fixing problems probably isn't necessarily the smoothest way to have an impact.

Setting goals and achieving them

So, given my particular set of skills, my inaugural task: by the end of 2023, find one simple feature to specify and land in the three major browsers. In April during a meeting of the Web Components Community Group, I found the perfect bite-sized feature to add; customElements.getName(). So that month I wrote some Web Platform Tests for the feature and a spec, then git cloned the Chromium source and got stuck in. It took me a few days to get close to a patch in Chrome most of that time was spent compiling the damn thing (a full working day for a fresh compile), but then came another month of feedback, improving my poor attempt at a patch (thanks to the incredible patience of Xiaocheng and Mason guiding me through it all!) With that merged I got to work on the Firefox implementation and it ended up going much quicker and easier now I was finding my feet; a week all in and it was merged! I took so long to get these implementations done, though, that Ryosuke had beat me to the WebKit implementation! I'd have to find a new thing to land in WebKit. Luckily it wasn't very long before I found another small contribution to make to WebKit to get the trifecta. And so in the space of 2 months I'd gone from barely knowing C++ to having a contribution in each of the browsers, soon to be deployed to somewhere to the tune of 4 billion customers.

This first experience of merging patches was both thrilling and also overwhelming. Browser engines are massive sprawling C++ codebases, and each has a distinct flavour.

Firefox is probably the least familiar codebase to work on - there are a lot of C++ macros, some peculiar naming conventions, and a lot of legacy artefacts, but it's a joy to work on - fast compiles (the fastest of all three) and a great CI setup. The documentation is good and the codebase is easy to navigate. Using Mercurial rather than Git is a choice... it took a while to get used to it, and I'm told at some point they'll switch to Git but it remains to be seen when. Git-Cinnabar exists to ease the pain, but I didn't find it that helpful, instead opting to learn Mercurial. Firefox uses Phabricator for reviews, which can be a little challenging to work with but for the most part has been trouble free. SearchFox is an indispensable tool to use especially as local IDE features like autocomplete or go-to-definition seem to fall over on a codebase where a lot of code generation and macros reside.

Apple's WebKit (aka open source Safari) is the most aesthetically pleasing codebase and maybe the most similar looking to JavaScript. The codebase adheres to modern C++ standards (they use common C++ standard library functions like std::optional and std::variant, while the other browsers have home-grown variants), it's also the most challenging to get started with as it seems to require some amount of institutional knowledge to know how to do things. Finding people and asking questions is key here - luckily folks are helpful on the WebKit Slack channel. Their documentation could be improved, and the unified builds are a struggle because what compiles locally might not pass CI checks across all platforms. WebKit's manual syncing of Web Platform Tests is a chore and I wish they'd move to more automation, like Firefox and Chrome do. Adding a new file is tricky because it effectively requires XCode which isn't available on Linux, but you can fudge it. WebKit uses Git and GitHub, which is great as it's one less barrier to contribution - especially when trying to onboard colleagues and peers. It being hosted on GitHub means you can GitHub's code navigation features to navigate code when local IDE features inevitably fall over, but there's also a "WubKat" index on SearchFox which I often consult, though the index is usually a day or so behind.

Chrome is massive and as a non-Googler without access to the shared build server - "reclient" - you can expect fresh compiles to take a full working day. Chrome's documentation is great though, and Google's source.chromium.org is an indispensable resource for code search. While it doesn't have the esoteric macros like Firefox, the code isn't as "clean" as WebKit, and there are a lot of differences between their choices and standard C++ features. The main issue with development on Chrome is just the sheer size. Fetches take forever, the file tree is gargantuan, and compiles are by far and away the slowest of the three. They use Git which is nice, but in slightly strange ways - I think because of Gerrit, their review system. PRs (or "CLs" as they call them) are done via Chromium Review and you'll need to use the provided tooling to submit them, you can't simply push a branch.

Swapping between engines can be a little confusing as you'll have to swap idioms from naming conventions to APIs to build tools, but if you squint your eyes hard enough they're all kind of the same - certainly when it comes to the raw implementation which of course is derived from the various standards documents. Given they all use C++, they all make use of classes and inheritance, public and private fields, but they all have their own built in smart pointer and tracing mechanisms which as a non-C++ developer is still confusing, but I muddle along.

But enough critique, back to the mission. Eventually, in October-December 2023 I managed to get a couple of big ships under my belt with CustomStateSet. You see, a year prior I had written about CSS Classes, espousing CSS CustomStateSet as an alternative to classes. The problem being that this was only available in Chrome at the time, and the polyfills for other browsers had some fairly large trade-offs. Talking with my new found browser engineer colleagues, I had concluded that work on this had somewhat stalled, we were at an impasse around syntax. Chrome had been shipping the "old" double-dash syntax (e.g. --foo) for over a year, but WebKit wished for a different syntax - the now settled syntax of :state(foo). I understood it that if WebKit or Firefox landed an implementation it could sway Chrome to move to the new syntax. So I implemented CustomStateSet in Firefox and then again in Webkit. These both were a lot of work for me - the WebKit implementation was quite a bit simpler and was done in a month and shipped shortly after, but the Firefox implementation was more complex, and touched into parts of Stylo, the Rust based CSS engine that Servo and Firefox share. Stylo has some advanced dependency tracking which I presume is what makes it so fast, but it also means tracking changes deeply throughout the code. Some follow on patches were required to get everything working, but the net result is that I learned a great deal about how browsers handle style invalidation and how they generate code from IDL definition files (here be dragons). Sadly I still have no idea how to effectively use debuggers in these projects so I end up relying a lot more on tests, which is probably not the worst thing.

The contributions slowly kept building. Fixing accessibility bugs in WebKit, small patches for Dialogs in Firefox. By November, Mozilla granted me "commit access", allowing me to land my own patches. By March '24 I became a WebKit comitter, and a Chromium committer a week later. Meanwhile I was encouraging my team-mates to send patches and get involved. Lindsey worked with me on a few WebKit patches, Marais has been landing patches on getting Observables in WebKit, and Clay has been working on headingoffset in Chrome, and participating in the Aria Working Group. I continue to encourage colleagues and host office hours to pair with any GitHub employee on building out browser features (if you're a colleague reading this, and want to pair then check my Slack profile where you'll find a link to book an hour session with me). This all came back to the original goal - retaining the DNA of GitHub supporting the Web Platform. I was elated when seeing Brian Kardell's Mid 2024 browser contribution statistics post, and in particular this image:

a pie chart listing the 2024 mid-year contributions to the WebKit project. The top 10, in order: Apple, Igalia, Sony, Ubie, Red Had, softathome.com, Rose, Alexey Knyazev, GitHub, Ian Grunert.

Proof that all this work has been paying off! GitHub might not be the largest contributor to WebKit - we've got other priorities - but we've made a dent, and hopefully made the Web Platform a little better.

Looking ahead

I intended to write this post as I landed my 100th patch, but laziness prevails and so I've now tipped over 100; 43 patches to WebKit, 36 to Firefox, and 26 to Chrome. July 2024, just over a year since my first patch to Chrome - adding customElements.getName() - I celebrated with a victory lap; contributing the same API to servo and a patch for ladybird. By my count that's 105 Web Platform contributions in 18 months, not including all the additional spec work (for example the 40 or so patches to the html & css specs), community building, or work my colleagues have been doing (Marais has been a machine, merging 17 patches to WebKit in a few short months).

So what's next? I won't be slowing down, and I don't think my colleagues will be either. I'll continue to make the case that GitHub is a special place, in a good position to push the web forward, and I'll still be working on new ideas I consider important. I will tell anyone that will listen that the door is open for web developers in the standards community, and it's important that we build a good balance between those using the platform and those implementing it. It's great to see companies like Shopify, Salesforce, Bloomberg and Paypal represented at these venues, and it would be great to see more. If you think your company would be interested in making more contributions to the web platform, feel free to reach out. I'd happily talk more about this topic at any length - and can travel (distance and budgets permitting) if that's preferred.

I've got a few bigger ideas in the pipeline. Invoker Commands is making good progress (parts of it are available for testing in the 3 major browsers) and will likely be shipping soon, and I've got a slew of smaller puzzle pieces around Richer Text Fields which I intend to work on. Not to mention continuing to help make Observables a thing. You can see what projects I'm focussed on over at the "What I'm working on" page, which I use to track the visibility of my browser contributions and other projects.