Stateless

Definition

A "stateless" distro is one where /etc and /var are entirely empty when booting. No state persists across reboots, hence the name. A more relaxed version of this preserves config files in /etc, but still allows them to be removed at any time to do a factory reset.

See http://0pointer.net/blog/projects/stateless.html for more information.

Purpose

In a stateless image, factory reset can be achieved by wiping out the content of /etc.

A system update can be done without touching local configuration files.

Limitations

Configuration files in /etc may become incompatible with an updated OS. That is a problem that needs to be solved by the person or tool which created these incompatible local configuration files. It can be mitigated a bit by keeping the configuration API of OS components stable across updates.

Implementation

Ideally, an OS component itself knows how to update the default configuration with small, independent configuration fragments in /etc, i.e. the actual configuration is constructed dynamically at runtime. systemd works like that.

Less suitable, but still "stateless" is an OS component which reads a configuration file from /etc if it exists and falls back to some read-only default when it doesn't. This is less suitable when a real human creates the configuration in /etc because that person then has to ensure that updated system defaults get copied into the config file in /etc, but it may work when the file is created using some tool that can be re-run after an update.

Finally, an OS component can be configured to ignore /etc entirely and just read its configuration files from a read-only location. It can then still be configured when building the OS, but not locally. Depending on the use-cases, such local modifications might not be needed.

Some legacy components use /etc as location for files created at runtime. This is undesirable because it makes it less obvious for an admin which files in /etc may be modified, but if the files get recreated without data loss after a factory reset or system update, then using /etc is acceptable.

Status and goals for "stateless" in Yocto

Stateless support does not exit in current Yocto. Work is under way to improve that. Related Bugzilla entries are https://bugzilla.yoctoproject.org/show_bug.cgi?id=9527, https://bugzilla.yoctoproject.org/show_bug.cgi?id=1593

It is not realistic to make all components in Yocto stateless. This will only work for some selected components and thus only those images using those components. The initial goal is to make the "refkit-image-common" from the IoT OS Reference Kit stateless. In that image, at least the following local configuration changes are meant to work:

• adding users and groups with user/groupadd
• replacing the default motd

The current prototype is hosted in https://github.com/pohly/intel-iot-refkit/tree/stateless. Previous work in Ostro that led to a considerably reduced /etc is in https://github.com/pohly/ostro-os/tree/stateless-rebased.

The generic mechanism is https://github.com/pohly/intel-iot-refkit/blob/stateless/meta-refkit/classes/stateless.bbclass. It supports STATELESS_RELOCATE_pn-<recipe> = "True" for compiling a recipe such that it uses a read-only sysconfig directory. STATELESS_MV is for cases where a recipe puts files into /etc in do_install that can also be elsewhere. STATELESS_RM is for items that can be removed. The same exists as STATELESS_RM/MV_ROOTFS for items created as part of package installation or rootfs functions.

The actual configuration used for Refkit is https://github.com/pohly/intel-iot-refkit/blob/stateless/meta-refkit/conf/distro/include/stateless.inc. It does not yet enforce an empty /etc. Instead one has to run "bitbake refkit-image-common:do_rootfs" and check the content of /etc manually.

Some BKMs for gradually moving content out of /etc:

• use STATELESS_MV_pn-<recipe> to avoid repackaging other recipes
• precede each entry with a detailed technical explanation of why the change is working
• sort entries in the file as follows:
  • header
  • STATELESS_ETC_WHITELIST
  • STATELESS_RM/MV sorted by pn-<reciped> (alphabetically)
  • STATELESS_RM_ROOTFS (no particular order, add at the bottom)
  • STATELESS_MV_ROOTFS (no particular order, add at the bottom)
  • special python functions

refkit-image-common is stateless once /etc is empty after do_rootfs and the resulting image works without problems. Right now, it still has the following content:

$ tree tmp-glibc/work/intel_corei7_64-refkit-linux/refkit-image-common/1.0-r0/rootfs/etc/
tmp-glibc/work/intel_corei7_64-refkit-linux/refkit-image-common/1.0-r0/rootfs/etc/
├── asound.conf
├── audisp
│   ├── audispd.conf
│   └── plugins.d
│       ├── af_unix.conf
│       └── syslog.conf
├── audit
│   ├── auditd.conf
│   ├── audit.rules
│   └── rules.d
│       └── audit.rules
├── bluetooth
│   ├── input.conf
│   └── network.conf
├── build
├── busybox.links.nosuid
├── busybox.links.suid
├── ca-certificates
│   └── update.d
├── ca-certificates.conf
├── dbus-1
│   ├── session.conf
│   ├── system.conf
│   └── system.d
│       ├── bluetooth.conf
│       ├── connman.conf
│       ├── dbus-wpa_supplicant.conf
│       ├── org.freedesktop.hostname1.conf
│       ├── org.freedesktop.locale1.conf
│       ├── org.freedesktop.login1.conf
│       ├── org.freedesktop.machine1.conf
│       ├── org.freedesktop.systemd1.conf
│       └── org.freedesktop.timedate1.conf
├── default
│   ├── auditd
│   ├── mountall
│   ├── usbd
│   ├── useradd
│   └── volatiles
│       ├── 99_dbus
│       ├── 99_pam
│       ├── 99_sshd
│       └── 99_wpa_supplicant
├── depmod.d
├── environment
├── filesystems
├── fstab
├── group
├── gshadow
├── host.conf
├── hostname
├── hosts
├── init.d
│   └── smack
├── inputrc
├── issue
├── issue.net
├── ld.so.conf
├── libaudit.conf
├── libnl
│   ├── classid
│   └── pktloc
├── login.defs
├── machine-id
├── mke2fs.conf
├── modprobe.d
├── modules-load.d
│   ├── iwlwifi.conf
│   └── uio.conf
├── motd
├── mtab -> /proc/mounts
├── network
│   ├── if-down.d
│   ├── if-post-down.d
│   │   └── wpa-supplicant -> ../if-pre-up.d/wpa-supplicant
│   └── if-pre-up.d
│       └── wpa-supplicant
├── nsswitch.conf
├── os-release
├── pam.d
│   ├── chage
│   ├── chfn
│   ├── chgpasswd
│   ├── chpasswd
│   ├── chsh
│   ├── common-account
│   ├── common-auth
│   ├── common-password
│   ├── common-session
│   ├── common-session-noninteractive
│   ├── groupadd
│   ├── groupdel
│   ├── groupmems
│   ├── groupmod
│   ├── login
│   ├── newusers
│   ├── other
│   ├── passwd
│   ├── runuser
│   ├── runuser-l
│   ├── sshd
│   ├── su
│   ├── systemd-user
│   ├── useradd
│   ├── userdel
│   └── usermod
├── passwd
├── profile
├── protocols
├── request-key.conf
├── request-key.d
├── rpc
├── securetty
├── security
│   ├── access.conf
│   ├── group.conf
│   ├── limits.conf
│   ├── limits.d
│   ├── namespace.conf
│   ├── namespace.d
│   ├── namespace.init
│   ├── pam_env.conf
│   └── time.conf
├── services
├── shadow
├── shells
├── skel
├── smack
│   ├── accesses.d
│   └── cipso.d
├── ssh
│   ├── moduli
│   ├── ssh_config
│   ├── sshd_config
│   └── sshd_config_readonly
├── ssl
│   ├── certs
│   │   ├── 02265526.0 -> /usr/share/ca-certificates/mozilla/Entrust_Root_Certification_Authority_-_G2.crt
│   │   ├── 024dc131.0 -> /usr/share/ca-certificates/mozilla/Microsec_e-Szigno_Root_CA.crt
│   │   ├── 03179a64.0 -> /usr/share/ca-certificates/mozilla/Staat_der_Nederlanden_EV_Root_CA.crt
...
│   │   └── XRamp_Global_CA_Root.pem -> /usr/share/ca-certificates/mozilla/XRamp_Global_CA_Root.crt
│   ├── openssl.cnf
│   └── private
├── sysctl.d
├── systemd
│   ├── journald.conf
│   ├── logind.conf
│   ├── network
│   ├── system
│   │   ├── bluetooth.target.wants
│   │   │   └── bluetooth.service -> /lib/systemd/system/bluetooth.service
│   │   ├── ctrl-alt-del.target -> ../../../lib/systemd/system/reboot.target
│   │   ├── dbus-org.bluez.service -> /lib/systemd/system/bluetooth.service
│   │   ├── default.target -> /lib/systemd/system/multi-user.target
│   │   ├── getty.target.wants
│   │   │   └── getty@tty1.service -> ../../../../lib/systemd/system/getty@.service
│   │   ├── local-fs.target.wants
│   │   │   └── var-volatile-lib.service -> /lib/systemd/system/var-volatile-lib.service
│   │   ├── multi-user.target.wants
│   │   │   ├── auditd.service -> /lib/systemd/system/auditd.service
│   │   │   ├── connman.service -> /lib/systemd/system/connman.service
│   │   │   ├── machines.target -> ../../../../lib/systemd/system/machines.target
│   │   │   └── remote-fs.target -> ../../../../lib/systemd/system/remote-fs.target
│   │   ├── network.target.wants
│   │   │   ├── ip6tables.service -> /lib/systemd/system/ip6tables.service
│   │   │   └── iptables.service -> /lib/systemd/system/iptables.service
│   │   ├── sockets.target.wants
│   │   │   └── sshd.socket -> /lib/systemd/system/sshd.socket
│   │   ├── sysinit.target.wants
│   │   │   ├── run-postinsts.service -> /lib/systemd/system/run-postinsts.service
│   │   │   └── systemd-timesyncd.service -> ../../../../lib/systemd/system/systemd-timesyncd.service
│   │   └── systemd-random-seed.service.wants
│   │       └── var-volatile-lib.service -> /lib/systemd/system/var-volatile-lib.service
│   ├── system.conf
│   ├── timesyncd.conf
│   ├── user
│   └── user.conf
├── terminfo
│   ├── a
│   │   └── ansi
│   ├── d
│   │   └── dumb
│   ├── l
│   │   └── linux
│   ├── r
│   │   └── rxvt
│   ├── s
│   │   ├── screen
│   │   ├── screen-256color
│   │   └── sun
│   ├── v
│   │   ├── vt100
│   │   ├── vt102
│   │   ├── vt200
│   │   ├── vt220
│   │   └── vt52
│   └── x
│       ├── xterm -> xterm-color
│       ├── xterm-256color
│       ├── xterm-color
│       └── xterm-xfree86
├── timestamp
├── tmpfiles.d
│   ├── 00-create-volatile.conf
│   ├── audit-volatile.conf
│   └── connman_resolvconf.conf
├── udev
│   ├── hwdb.d
│   ├── rules.d
│   │   ├── touchscreen.rules
│   │   └── udev-smack-default.rules
│   └── udev.conf
├── udhcpc.d
│   └── 50default
├── version
├── wpa_supplicant.conf
└── xdg
    └── systemd
        └── user -> ../../systemd/user

62 directories, 500 files

Alternatives

There are other solutions with similar goals. However, stateless has some properties that, when it is applicable, make stateless a better solution.

overlayfs

The OS configuration files are in /etc. At runtime, an overlay is activated. Editing items in /etc then stores a new copy of the modified files in the overlay. A system update is done when the overlay is not active.

Drawback: files in the overlay continue to shadow an updated system configuration, i.e. new settings from the updated OS configuration files are ignored. Addressing this would require merging potentially complex files during an update.

config generator

Instead of allowing local modifications in the normal /etc configuration files, one or more additional config files are used. A tool takes those files and the default system configuration to produce the actual content of /etc, either by writing to it directly or into an overlay.

Drawback: custom solution, the normal tools like "useradd" do not work; no such generator exists (?) and thus would have to be written from scratch Advantage: full control over the format of allowed configuration changes; the generator can be updated together with the OS so that the output always matches what the OS components expect

Such a config generator can be combined with a stateless distro.