Several months ago, the LLVM project blog published an article, Closing the gap: cross-language LTO between Rust and C/C++. In it, they explained that link-time optimization can improve performance by optimizing throughout the whole program, such as inlining function calls between different objects. Since Rust and Clang both use LLVM for code generation, we can even achieve this benefit across different programming languages.
In Red Hat Developer Tools, we have the Rust and LLVM Toolsets, which can easily be used together for cross-language link-time optimization (LTO), so let's try it out.
Setup
One of the caveats mentioned in the LLVM blog is toolchain compatibility, because cross-language LTO needs a close match between the LLVM libraries used by rustc and clang. The upstream Rust builds use a private copy of LLVM, which is sometimes even a snapshot rather than a final release, so it can be challenging to get a corresponding build of clang.
Red Hat's Rust Toolset uses the shared LLVM library from the same LLVM Toolset we get Clang from, so we always know these compilers are using the same code generation backend, yielding compatible bytecode for LTO.
On Red Hat Enterprise Linux 8 (RHEL 8), both toolsets are available in the default AppStream repository. We can install their default module profiles to get complete toolchains for Rust and Clang:
# yum module install rust-toolset llvm-toolset
On Red Hat Enterprise Linux 7, the toolsets are available as software collections. After getting access to the devtools repository, we can install the current version of each toolset:
# yum install rust-toolset-1.39 llvm-toolset-8.0
Then, we can add them to our working PATH by starting a new shell with Rust enabled, which implicitly enables LLVM as a dependency:
$ scl enable rust-toolset-1.39 bash
Example: Inlined summation
Rust documents a special option for LTO, -C linker-plugin-lto, with example commands for C calling Rust and vice versa. Let's apply this option to a simple example summing from one to n. This example has a closed form n(n+1)/2 that most optimizers know, and with constant inputs, it may also compile to a constant.
First, there is our C code in main.c:
#include <stdio.h>
int sum(int, int);
int main() {
printf("sum(%d, %d) = %d\n", 1, 100, sum(1, 100));
return 0;
}
Then, we have our Rust code in sum.rs:
#[no_mangle]
pub extern "C" fn sum(start: i32, end: i32) -> i32 {
(start..=end).sum()
}If we compile this combination without LTO:
$ rustc --crate-type=staticlib -Copt-level=2 sum.rs $ clang -c -O2 main.c $ clang ./main.o ./libsum.a -o main $ ./main sum(1, 100) = 5050
We can see in the disassembly (objdump -d main) that main makes a normal function call to sum:
00000000004005d0 <main>: 4005d0: 50 push %rax 4005d1: bf 01 00 00 00 mov $0x1,%edi 4005d6: be 64 00 00 00 mov $0x64,%esi 4005db: e8 20 00 00 00 callq 400600 <sum> 4005e0: bf e8 06 40 00 mov $0x4006e8,%edi 4005e5: be 01 00 00 00 mov $0x1,%esi 4005ea: ba 64 00 00 00 mov $0x64,%edx 4005ef: 89 c1 mov %eax,%ecx 4005f1: 31 c0 xor %eax,%eax 4005f3: e8 d8 fe ff ff callq 4004d0 <printf@plt> 4005f8: 31 c0 xor %eax,%eax 4005fa: 59 pop %rcx 4005fb: c3 retq 4005fc: 0f 1f 40 00 nopl 0x0(%rax) 0000000000400600 <sum>: ...
Now, let's try it with cross-language LTO:
$ rustc --crate-type=staticlib -Clinker-plugin-lto -Copt-level=2 sum.rs $ clang -c -flto=thin -O2 main.c $ clang -flto=thin -fuse-ld=lld -O2 ./main.o ./libsum.a -pthread -ldl -o main $ ./main sum(1, 100) = 5050
This time in the disassembly, we can see that the call is gone, replaced with a simple constant 0x13ba, which is 5050:
00000000002a6250 <main>: 2a6250: 50 push %rax 2a6251: bf b4 66 20 00 mov $0x2066b4,%edi 2a6256: be 01 00 00 00 mov $0x1,%esi 2a625b: ba 64 00 00 00 mov $0x64,%edx 2a6260: b9 ba 13 00 00 mov $0x13ba,%ecx 2a6265: 31 c0 xor %eax,%eax 2a6267: e8 04 03 00 00 callq 2a6570 <printf@plt> 2a626c: 31 c0 xor %eax,%eax 2a626e: 59 pop %rcx 2a626f: c3 retq
This example isn't big enough for us to show any measurable performance difference, but I hope it's clear that there are real opportunities for greater optimization in larger programs—even across such different languages.
Example: Firefox
As mentioned in the LLVM blog post, Firefox was a major motivating case for cross-language LTO as a project mostly written in C++, but increasingly using Rust as well. Both languages can make FFI calls to the other, but there's overhead to those calls if they can never be inlined, even in trivial cases.
I'm no expert on testing browser performance, but I was able to try a few builds of Firefox 73.0 using the Rust and LLVM Toolsets on Red Hat Enterprise Linux 8. Here's my mozconfig file:
export CC=clang export CXX=clang++ export AR=llvm-ar export NM=llvm-nm export RANLIB=llvm-ranlib ac_add_options --disable-elf-hack ac_add_options --enable-release ac_add_options --enable-linker=lld
I then compared builds by adding --enable-lto=thin and --enable-lto=cross (also implies "thin"). The bulk of the code is built into libxul.so, and here's a rough comparison of that resulting binary:
| LTO | .text size |
.data size |
nm symbols |
|---|---|---|---|
| - | 117,770,174 | 4,259,609 | 303,677 |
| thin | 115,602,351 | 4,249,968 | 268,377 |
| cross | 115,627,483 | 4,232,929 | 267,925 |
By these metrics, the bulk of the benefit comes from just enabling thin LTO at all, and in the cross build the code size got slightly larger. Still, the symbol count did drop a little further in the cross build, which indicates in a crude way that there was probably more inlining, at least.
What does that result actually achieve for performance? I'll leave that to Firefox developers to measure themselves.
Conclusion
Link-time optimization is a great way to squeeze more performance into a large build, but if you're using multiple programming languages, the calls between them might have been a barrier. Now, the combination of Clang, Rust, and their common LLVM backend make cross-language LTO possible, and we can see some real benefits!
Last updated: June 29, 2020