3 essentials for writing a Linux system library in Rust

Rust is a good choice for writing a Linux system library due to guarantees on memory and thread safety, a revolutionary approach to memory ownership, and a polished Foreign Function Interface. In spite of a slightly steep learning curve, you will come to love Rust and especially its compiler.

This article is the first in a series focusing on Rust for Linux. Check out the other three articles:

• Part 2: How to create C binding for a Rust library

• Part 3: How to create Python binding for a Rust library

• Part 4: Build trust in continuous integration for your Rust library

I have created a demo Git repository with simple code to help explain the points in this article. The repository contains:

• An echo server listening on the Unix socket /tmp/librabc
• A Rust crate that connects to the socket and sends a ping packet every 2 seconds
• A C/Python binding
• A command-line interface (CLI) for the client
• A CI system built on top of GitHub Action

The Rust community has many great documents and other resources that teach Rust to a variety of audiences. This article assumes you have already triumphed over the Rust learning curve and acquired basic Rust coding experience with:

• Memory and object ownership
• Traits
• Cargo and crates.io
• Async/await

If you lack experience with any of these topics, The Rust Programming Language could be a good starting point. The Rust API Guidelines lists best practices for several aspects of Rust programming.

From Python to Rust

The content of this series is inspired by our work rewriting the Nmstate networking library as a Rust library. Nmstate was originally in Python, and we rewrote it in the hope that it could be included in Fedora CoreOS, where Python is not allowed.

Since one major dependency of Nmstate is written in Rust, we decided to rewrite Nmstate in Rust. (Of course, we never thought about rewriting Nmstate in C/C++, considering the 100+ options of support in nmstate and complex yaml/json parsing workflow. It could be a headache doing that in C and C++.)

Because Nmstate already has an API user in oVirt and Red Hat OpenShift, we needed to provide an identical Python API with this Rust rewrite. To minimize the Rust dependencies and save Rust compile time, we decided to just ship a C library in Rust and wrap that C library in a Python library. The Go binding of Nmstate is also a wrapper of this C library.

3 essential practices for writing a Linux system library in Rust

This article focuses on these three Rust practices that benefit a Linux system library: backward compatibility, event-driven asynchronous programming, and logging.

1. Backward compatibility

In addition to the ironclad features that help keep Rust programs from breaking, are rules that allow programs to keep working after a library has added new elements to a struct (structure) or enum. The best practice, when you code your library, is to decorate your public structs and enums with the non_exhaustive attribute as follows:

#[derive(Clone, Copy, Debug, PartialEq, Eq)]
#[non_exhaustive]
pub enum RabcEvent {
    IpcIn = 1,
    Timer,
}

#[derive(Clone, Debug, PartialEq, Default)]
#[non_exhaustive]
pub struct RabcFoo {
    foo1: String,
    foo2: u32,
}

"Non-exhaustive" means that you might add a new element in the future. The attribute has no effect within your own crate. You can refer to your own RabcEvent enum using code such as:

match event {
    RabcEvent::IpcIn => {},
    RabcEvent::Timer => {},
}

However, if another crate that imports the library tries to issue the same code, the dependent crate gets a compiler error such as:

7  | pub enum RabcEvent {
   | ^^^^^^^^^^^^^^^^^^
   = note: the matched value is of type `RabcEvent`, which is marked as
           non-exhaustive
help: ensure that all possible cases are being handled by adding a match arm
with a wildcard pattern or an explicit pattern as shown
   |
16 ~                 RabcEvent::Timer => (),
17 ~                 _ => todo!(),

The reason for this message is that the code handled the two values that exist in the enum now, but might break in the future if the library adds another value.

In short, the non_exhaustive attribute in a library requires the dependent crate to add a default handler. An example referring to the previous struct is:

let foo = RabcFoo {
    foo1: "Abc".into(),
    foo2: 0,
    ..Default::default()
};

let foo = RabcFoo::new(foo1, foo2);

2. Event-driven asynchronous programming

Sockets are widely used in Linux for interprocess communication (IPC). There are two ways to use sockets in Rust:

• rust async/await: Asynchronous and feature-rich
• epoll: Synchronous and lightweight

An example of asynchronous communication is the well-known tokio library, whereas synchronous communication is illustrated by the mio library.

The async/await-based echo server

The echo server cannot just fork a new thread for each client that connects because this places too much burden on CPU and memory resources and even opens up the risk of Denial-of-Service (DoS) attacks. Instead, the server creates a worker queue to limit resource use. tokio builds on Rust's async/await feature to provide an asynchronous runtime with numerous control options. This makes tokio the perfect fit for building an asynchronous echo server.

The code shown in this section comes from the full example in GitHub.

In addition to writing an async function to handle the client connection, add the tokio macro to build the tokio runtime, as shown in the following example:

#[tokio::main(flavor = "current_thread")]
async fn main() {
    // Many lines omitted
    loop {
        match listener.accept().await {
            Ok((stream, _)) => {
                tokio::spawn(async move {
                    process_client(stream).await;
                });
            }
            Err(e) => {
                log::error!("Failed to accept connection {}", e);
            }
        }
    }
}

The tokio::main(flavor = "current_thread") macro sets up a single-threaded tokio runtime within the current thread. If you prefer multithreaded execution, you can use:

#[tokio::main(flavor = "multi_thread", worker_threads = 10)]

For more control over the async runtime, you can use tokio::runtime::Builder. For example:

let runtime = tokio::runtime::Builder::new_multi_thread()
    .worker_threads(10)
    .build()?;
runtime.block_on(your_async_function())?;

With the help of the async runtime, you can easily wrap a complex async action. You could even wrap complex async actions into a synchronized API using async/await along with async runtime, but I would strongly recommend against it. Exposing a synchronous function using the tokio block_on() method inside and the developers writing to your API use tokio::runtime::Runtime::block_on over your wrapped API, triggers this error: Cannot start a runtime from within a runtime.

However, we are fine using tokio in the binary for the rabcd server.

The epoll-based echo client

When talking about asynchronous communication, most old-school C programmers instantly come up with the idea of using select or epoll. They find these mechanisms cozier than the new tokio aysnc runtime shown in the previous section.

Some reasons for using this classic epoll approach include:

• We save the Rust crate or API consumer from dealing with threading, especially memory sharing and communication between threads.
• It's easy to expose this synchronous Rust function to a C API.

You can get the full code for this sample application and library on GitHub.

To build the client library, you need to:

• Create a file descriptor for Unix socket communication.
• Create a TimerFD timer to generate an event every 10 seconds.
• Create an epoll instance to the poll the two previous file descriptors.
• Use epoll_event to distinguish socket communication from the repeat timer. The struct epoll_event is C structure in epoll. It should be RabcEvent in this Rust context.

Wrapping the two file descriptors into a single opaque struct with a process() call simplifies the library without the need for multithreading or blocking. The library user can choose how to process the asynchronous events. The RabcClient::process() will take RabcEvent and choose the correct file descriptor to proceed with the action base on the specified event type.

If you used mio to implement this strategy, you would have to implement mio::event::Source for your TimerFd file descriptor, which would be just as complex as just wrapping epoll yourself. But if your project could benefit from the mio library, please consider using it and contributing to the project.

IPC between server and client

In our example, the server accepts client connections on a Unix stream socket at /tmp/librabc. The socket does not place boundaries around each transaction. To distinguish messages, I normally append the data size before the data transfer. For example, the ipc_send() function of librabc/src/lib/ipc.rs has:

self.stream.write(&data.len().to_ne_bytes())?;
self.stream.write(data.as_bytes())?;

But there is a potential problem with this approach. A malicious client could set the size to u32::MAX causing the server to run out of memory. Instead, place a limit to the size through the RabcConnection.max_size variable:

pub fn ipc_send(&mut self, data: &str) -> Result<(), RabcError> {
    if data.len() > self.max_size {
        return Err(RabcError::new(
            ErrorKind::ExceededIpcMaxSize,
            format!(
                "Specified data exceeded the max size {} bytes, \
                 please change the limitation by set_ipc_max_size()",
                self.max_size
            ),
        ));
    }

The RabcClient struct

A struct named RabcClient is the interface used by a client to communicate with the server without considering the underlying IPC. The client has three public functions:

pub fn new() -> Result<Self, RabcError>;

pub fn poll(&mut self, wait_time: u32) -> Result<Vec<RabcEvent>, RabcError>;

pub fn process(
    &mut self,
    event: &RabcEvent,
) -> Result<Option<String>, RabcError>;

The client uses poll() to get a list of events and process() to process each one.

3. Logging

A system library should never print any message to stdout or stderr. Hence, a good logging system is mandatory. Rust has divided the logging system into a front end and back end:

• The logging facade: The set of APIs and traits abstracting the logging implementation.
• The logging implementation: The executor that saves the logs using the chosen method (printing to stderr, saving to journald, etc).

The library should depend only on the logging facade, allowing the library's consumer to define the logging actions. I use the log crate in the library and the env_logger crate in the CLI.

In general, the library simply uses the log::warn!() macro for logging:

log::warn!("This is a warning message");

Then the library consumer uses env_logger to print the logs:

let mut log_builder = env_logger::Builder::new();
log_builder.filter("foo", log::LevelFilter::Debug);
log_builder.init();

Robust libraries use asynchronous communication and flexible logging

This article offered specific guidelines for writing a Linux system library in Rust. The next articles will cover other aspects of integrating the system library into a larger system and the development cycle. Part 2 in this series describes the C binding and contains our custom-built, memory-based logging implementation.