-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 =============== kpatch delivery =============== Abstract ======== This is a proposal to create a delivery and build system to provide easy and automatic delivery of patches to a running linux kernel. CentOS 7 currently has support for live patching a running kernel (via kpatch and the surrounding tools), but crafting a patch and applying it is a very manual process. Making applying kernel fixes an easy and automatic process will make it easier to keep a CentOS installation more secure by default without having to schedule downtime, which will be a large benefit to users. Overview of proposed system =========================== - From a high level, the system will have a single RPM package for each individual kernel patch. This package will contain a kpatch kernel module built for each supported kernel. This allows for the greatest flexibility if end users have particular workflows and processes for deployment. It also allows us to utilise RPM package signing, which ensures users can trust these kernel patches originate from CentOS. RPM building - ------------ Given a manifest with the following information: - Path to kernel source patch - The range of kernel releases compatible with this patch (and which require the fix contained within the patch) - The version of the patch The rpm-building tool will then output a .spec file which can be built with the standard RPM toolchain. This spec file contains the scripts to: - install the kernel modules in /lib/modules - insmod the module into the running kernel - install the module into the initramfs, which makes sure each patch module is loaded when that kernel is booted There will be some magic in this tool to work out which functions the patch alters. `Provides:` and `Conflicts:` lines will be added to the output spec file for each function touched, with something like `Provides: kpatch-function-foo` (the actual naming scheme can be decided on later). This should ensure that only a single patch which modifies a certain function can be installed and active at any point in time. Installing - ---------- All the RPM packages will be added to a yum group. The packages can then be installed by running: $ yum group install kernel-livepatches (again, naming for this group can be decided on at a later date) To update the patches currently installed, that group can be upgraded. This will install new patches, if they are available, and updates any currently installed patches. $ yum group upgrade kernel-livepatches Only when a patch currently in the running system is updated will this process rmmod/insmod the patch's kernel module, since the existing RPM packages are unchanged. Higher level tool - ----------------- To assist with installing and maintaining these packages, a small CLI tool will be created which makes it easier to manage this process. This will be a small application using the yum API which chooses which packages to install based on the currently running and installed kernel versions. Patch QA - -------- This is an important part of ensuring the system is stable and production ready. Each patch which is built and packaged needs to be tested. This at a minimum means booting a VM with the affected kernel, then: 1. Run a small test suite 2. Install the patch package which is being tested 3. Run the same test suite again 4. Remove the patch package 5. Run the same test suite again The output of each of these test runs should be the same. Otherwise, the patch is defective. The test suite could be built by hand with an additional test case being added with each patch, exercising a code path which touches the functions which are modified in the patch. Benefits to Community ===================== Although the development effort is aimed at CentOS, since RHEL is binary compatible it will also benefit enterprise customers. Timeline ======== - Community bonding period - Week 0: - Make the plans in this document more concrete by getting input from a wider audience. - Week 1-5: Write the basic RPM building tool 1. Design manifest file format 2. Write scripts to build the kernel modules for each required kernel release, load and unload the modules, and install the modules in the initramfs 3. Write script to work out which functions a source patch touches and output virtual dependencies for the RPM spec files to avoid conflicts 4. Documentation for tool usage - Week 6-7 1. Work out any tasks the CLI tool should help with beyond install and update with the Using the background of having completed the generation of the packages, design 2. Implement install and update tasks in the CLI tool - Week 8-10: 1. Gather requirements and design the automation of the patch QA process, hopefully integrating with the current CentOS CI tooling 2. Implement this CI tooling - Week 11-12: Polishing and documentation for all parts of the project - Week 12-future: Maintenance About myself ============ Email: louis@kragniz.eu IRC nick on freenode: kragniz Github: https://github.com/kragniz Background: I'm a second year undergraduate at Aberystwyth University (in Wales, United Kingdom). Some of the experience I have outside of university work: - Currently an OpenStack core developer (on the image service) - Simple kernel modules - RPM packaging (for fedora) - Much fun with python Most of these can be seen on my github profile: https://github.com/kragniz Availability during the summer: Since I'm a core on OpenStack, I plan on spending a little time to maintain that (averaging up to a couple of hours a day). Beyond that, I'm free to put in 30-40 hours (more if necessary) per week. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBAgAGBQJVE7BHAAoJEI6BptrhPnCYbwQH/0yQFqR17H1BEWweDOdW3Q4W UhEjSSFvVRce+gcG2SifWMfxxsgNzRk1SLoriLwgtseR8+bLYPyOcVyqAom1+akW wYLAKFBSSC3+tt0+EagC0NRNuJxyPLdt9LMcD35Rfke1y/HUwGslCQbip5lIkE+h PgRtVYa5dN4s8NZGcYqFvDfxECD/NZZwivdSzxvqdEULsD+FDLhMzZBKLk7/T9QE 9IubbgNv6ykVNh4dqUjOEcttKY7akVKK+ELLyhSIPiRf5pdsHQcWbFrowqcjgen7 VrUIaO4c+xmYsIXrux/l5kW7wMfmgAEO2FH8uQc2EUTWYRHeRI28+KFQ70x7isg= =8EIv -----END PGP SIGNATURE-----