Android 13 Launch Devices Will Use Huawei’s EROFS for Fast, Efficient Storage Use

Huawei may not be selling as many phones internationally as it once did, but its influence in the Android world can still be felt. Case in point, a file system authored by engineer Xiang Gao while he was working at Huawei is set to be used by every upcoming Android device launching with Android 13, at least those that ship with GMS. This file system, called EROFS (short for “Enhanced Read-Only File System”), is already in use by a few OEMs outside of Huawei because of the significant storage space savings and performance benefits it offers. With support for the file system baked into Linux since version 5.4, Google feels ready to make EROFS Android’s preferred file system for read-only partitions.

In this week’s edition of Android Dessert Bites, I’ll be sharing what evidence there is pointing to EROFS becoming mandatory for Android 13 launch devices (it’s pretty much an open secret) as well as some of the improvements users can expect from the change in file systems.

Esper Device Management

EROFS, from Huawei to Linux to Android

EROFS was first introduced to the staging area of the Linux kernel with version 4.14 back in mid-2018, when author and maintainer Xiang Gao submitted the first patches for the driver. Xiang stated that EROFS, unlike other read-only file systems, “focuses more on performance than purely on saving storage space as much as possible.” In his post, Xiang shared some numbers comparing the sequential and random read/write speeds from a Kirin 970-powered Android phone with an EROFS versus EXT4-formatted partition, and the initial results were promising.

After a year of continued development, Xiang returned with new patches graduating the EROFS file system driver from staging to mainline in version 5.4. EROFS, he said, had proven to be stable enough to be deployed in over 10 million Huawei devices running Android-based EMUI 9.0.1. Huawei was even confident enough in it to promote it as a key feature of EMUI 9.1.

(As a side note, you’ll notice that in Xiang’s original post, as well as in some old marketing posts, EROFS is referred to as the “Extendable Read-Only File System” rather than the “Enhanced Read-Only File System. I’m not sure exactly when “Extendable” was changed to “Enhanced”, but the latter is what it officially goes by now.)

Although Huawei’s EROFS kernel driver became part of the mainline Linux kernel with version 5.4 in late 2019, the file system itself wasn’t put to use by other Android device makers until mid-2021. Part of the reason is the kernel version requirement. Android devices tend to run older Linux kernel versions, so OEMs had to backport some patches if they wanted to support the feature on devices with older Linux versions. In addition, OEMs had to build support for EROFS into their OTA update infrastructure, since update_engine, Android’s A/B update daemon, didn’t yet support EROFS payloads. Eventually, though, several prominent Android OEMs, such as OPPO and Xiaomi, began shipping products using EROFS.

Other OEMs will soon follow Huawei, OPPO, and Xiaomi in using EROFS, thanks in large part to Google. Since the EROFS kernel driver was mainlined in Linux kernel 5.4, the Android Common Kernel inherited the changes starting with android11-5.4. Devices launching with Android 13 have to ship with at least android12-5.4 (which is allowed because of Google Requirements Freeze), so every Android 13 launch device will have a kernel that’s compatible with EROFS. Google has also brought EROFS support to update_engine, added the user space tools to build EROFS images, and recently made it possible to build GSIs and cuttlefish AVDs with EROFS. EROFS has received robust support from Google in recent months, and the reason recently became clear: They plan to replace EXT4 with it, at least for Android’s read-only partitions.

raven:/ $ mount
/dev/block/dm-7 on / type ext4 (ro,seclabel,relatime)
/dev/block/dm-8 on /system_dlkm type ext4 (ro,seclabel,relatime)
/dev/block/dm-9 on /system_ext type ext4 (ro,seclabel,relatime)
/dev/block/dm-10 on /product type ext4 (ro,seclabel,relatime)
/dev/block/dm-11 on /vendor type ext4 (ro,seclabel,relatime)
/dev/block/dm-12 on /vendor_dlkm type ext4 (ro,seclabel,relatime)

What drew OPPO, Xiaomi, and now Google to pick EROFS over other file systems? As previously mentioned, EROFS offers better random read/write performance over existing solutions. In addition, EROFS also offers dramatic storage space savings. Building a cuttlefish image that uses EROFS instead of EXT4 saves about 800MB of space, for example.

With this reduction of size, the extra space could be used to pack even more system apps and files into those partitions. The extra space could alternatively be given to the user data partition to hold more of the user’s apps and files. The bandwidth savings also can’t be neglected. The size reduction really adds up when OEMs have to push full OTA updates to hundreds of thousands or even millions of users.

Exactly how it achieves these improvements is difficult to explain because file system architecture is a complex subject. The abstract of the paper (PDF warning) Xiang and co. submitted to the 2019 USENIX Annual Technical Conference offers a nice tl;dr:

Smartphones usually have limited storage and runtime memory. Compressed read-only file systems can dramatically decrease the storage used by read-only system resources. However, existing compressed read-only file systems use fixed-sized input compression, which causes significant I/O amplification and unnecessary computation. They also consume excessive runtime memory during decompression and deteriorate the performance when the runtime memory is scarce. In this paper, we describe EROFS, a new compression-friendly read-only file system that leverages fixed-sized output compression and memory-efficient decompression to achieve high performance with little extra memory overhead. We also report our experience of deploying EROFS on tens of millions of smartphones. Evaluation results show that EROFS outperforms existing compressed read-only file systems with various micro-benchmarks and reduces the boot time of real-world applications by up to 22.9% while nearly halving the storage usage.

^{Xiang Gao, Mingkai Dong, Xie Miao, Wei Du, Chao Yu, and Haibo Chen. 2019. EROFS: a compression-friendly readonly file system for resource-scarce devices. In Proceedings of the 2019 USENIX Conference on Usenix Annual Technical Conference (USENIX ATC '19). USENIX Association, USA, 149–162.}

^‍ you want to get down into the nitty-gritty technical details, then I recommend starting with Xiang’s presentation at USENIX or KubeCon + CloudNativeCon. You can also read the documentation and readme for EROFS on kernel.org.

EROFS, coming soon to an Android 13 device near you

Given the numbers Huawei provided, it was no surprise to hear that OEMs flirted with EROFS on their devices. It was surprising, though, to hear that Google planned to make EROFS mandatory for all read-only partitions of Android 13 launch devices. The company has openly signaled its intention since around December of last year, though, so it was possible to see this move coming.

Back in December, a Google engineer repeated what a colleague told him about Android’s upcoming file system requirements while implementing the new system_dlkm partition, which holds dynamically loadable kernel modules for the system partition. According to him, the file system of the system_dlkm partition should be EROFS instead of EXT4 since “all RO fs will move to erofs with A13.” This patch was under the cuttlefish branch, so it wasn’t clear at the time if this statement only applied to cuttlefish builds or if they signaled Google’s broader plans for Android.

While a later patch explicitly confirmed that all read-only partitions of cuttlefish builds would use EROFS, it wasn’t until last month that it became clear the statement applied to all Android (for handheld) builds. Last month’s AOSP patch that added support for building GSIs with EROFS stated that “EROFS is planned to replace … ext4 as the file system format of readonly partitions.” Earlier this month, a Googler asked how a patch would “work today on P22 [Pixel 7 series] devices given that we mandate erofs for T launch.” Lastly, one of the talks at this year’s Linux Plumbers Conference is titled “erofs as a replacement for f2fs and the deprecation of ext4”, and it’s being held by a Google engineer working on file systems for Android. I previously heard through the grapevine about Android 13’s EROFS requirement for read-only partitions, but it looks like it’s not much of a secret given the repeated public mentions.

Interestingly, Google is also working to make the Android user space aware of EROFS, though I’m not sure why. The PackageManager within the latest Android 13 beta includes two new feature declarations: android.software.erofs and android.software.erofs_legacy.

These feature declarations don’t appear in AOSP yet, but the Pixel 6 on Android 13 declares the former even though its partitions are formatted in EXT4. This is because the framework only checks if the kernel supports EROFS, not whether the partitions are formatted in it, by probing for the existence of /sys/fs/erofs.

Again, I don’t know why Android is checking for EROFS support in the kernel, because these feature declarations aren’t checked anywhere yet as far as I can tell. (erofs_legacy, by the way, seems to refer to older versions of Linux 5.4 and 4.19 that support a legacy compression mode for EROFS images.)

Thanks for reading this edition of Android Dessert Bites. I hope that with everything I’ve revealed about Android 13 through my newsletter column and the Esper blog you’re as excited as I am about the upcoming release! I’ll be on vacation starting early next week, so there won’t be another edition of Android Dessert Bites for the next two weeks.

If you are looking for a tool to manage Android devices, give us at Esper a shout. To get an overview of mobile device management tools, check out our guide: 

Everything You Need to Know about MDM