Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Multipath TCP on RHEL 8: From one to many subflows

October 20, 2021
Andrea Claudi
Related topics:
DevOpsLinuxOpen source
Related products:
Red Hat Enterprise Linux

Share:

    Many years ago, Hercules was traveling back to Athens when he lost his way in a wood. Arriving at a crossroad, he found two women. One of them was Aletheia, the goddess of truth, and the other was Apate, the goddess of lies. But he didn't know who was who. Hercules needed all his intelligence to find the only question capable of revealing the way to Athens.

    If you are lost in a computer network and don't know which path to take, don't worry—you don't need Hercules to find your way: You can use Multipath TCP.

    Multipath TCP in Red Hat Enterprise Linux 8

    Multipath TCP (MPTCP) is an extension to the Transmission Control Protocol (TCP) to use multiple paths between peers simultaneously. Linux's MPTCP implementation is fairly recent and was included in version 5.6. Red Hat Enterprise Linux includes MPTCP starting with version 8.3.

    In the first part of this series, Davide Caratti showed how to enable MPTCP, open an MPTPC socket in an application, and verify that MPTCP is working as expected. You can follow his tutorial to start experimenting with MPTCP.

    In this article, you will learn how to:

    • Use iproute2 to add multiple subflows to an MPTCP connection.
    • Verify that MPTCP uses multiple subflows.

    Opening an MPTCP socket

    To see MPTCP in action, you need to open an MPTCP socket from a user-space application. Let's follow the instructions from the first article of this series to set up a multiple subflow testbed.

    First of all, because MPTCP is disabled in the default Red Hat Enterprise Linux configuration, you need to enable it using sysctl so you can create MPTCP sockets:

    # sysctl -w net.mptcp.enabled=1
    # sysctl net.mptcp.enabled
    net.mptcp.enabled = 1

    MPTCP sockets are the same as the regular TCP sockets and use the same semantics. An application can natively add support to MPTCP using a socket with IPPROTO_MPTCP like this:

    fd = socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP);

    Implementing a full-fledged MPTCP application is not difficult, but what if you want to use a normal user-space application without any knowledge of IPPROTO_MPTCP?

    Fear not: There are multiple paths to avoid patching and rebuilding all networking applications. The easiest path is probably to use systemtap to replace IPPROTO_TCP with IPPROTO_MPTCP in all the calls to __sys_socket() in the kernel. To make the replacement, you need to install a few packages:

    $  dnf -y install \
      kernel-headers \
      kernel-devel \
      kernel-debuginfo \
      kernel-debuginfo-common-x86_64 \
      systemtap-client \
      systemtap-devel

    Now you can download the systemtap script from this guide and use the following command to start it:

    # stap -vg mpctp.stap

    Verify that the systemtap script is working by looking at the output from dmesg:

    # dmesg
    ...
    [1114254.601040] Kprobes globally unoptimized
    [1114254.611265] stap_d60b4bc4e0332aa247ebd9b2ffa597_213373: module_layout: kernel tainted.
    [1114254.620296] Disabling lock debugging due to kernel taint
    [1114254.626423] stap_d60b4bc4e0332aa247ebd9b2ffa597_213373: loading out-of-tree module taints kernel.
    [1114254.636597] stap_d60b4bc4e0332aa247ebd9b2ffa597_213373: module verification failed: signature and/or required key missing - tainting kernel
    [1114254.680368] stap_d60b4bc4e0332aa247ebd9b2ffa597_213373 (mptcp.stap): systemtap: 4.4/0.182, base: ffffffffc1082000, memory: 224data/32text/15ctx/24678net/202alloc kb, probes: 1

    Setting up the testbed

    As a simple test setup, we will use two network namespaces, mptcp-client and mptcp-server, to simulate the client and the server in an MPTCP connection. The two namespaces will be connected by two different virtual Ethernet (veth) paths: From 10.0.0.1 to 10.0.0.2 and from 192.168.0.1 to 192.168.0.2 (Figure 1).

    Network topology for MPTCP two-stream test. In this setup, 10.0.0.1 on the server communicates with 10.0.0.2 on the client, and 192.168.0.1 on the server communicates with 192.168.0.2 on the client.
    Figure 1: Network topology for two-path MPTCP test.

    You can use the following script to set up the testbed:

    #!/bin/sh
    
    ip netns add mptcp-client
    ip netns add mptcp-server
    
    sysctl -w net.ipv4.conf.all.rp_filter=0
    ip netns exec mptcp-client sysctl -w net.mptcp.enabled=1
    ip netns exec mptcp-server sysctl -w net.mptcp.enabled=1
    
    ip link add red-client netns mptcp-client type veth peer red-server netns mptcp-server
    ip link add blue-client netns mptcp-client type veth peer blue-server netns mptcp-server
    
    ip -n mptcp-server address add 10.0.0.1/24 dev red-server
    ip -n mptcp-server address add 192.168.0.1/24 dev blue-server
    ip -n mptcp-client address add 10.0.0.2/24 dev red-client
    ip -n mptcp-client address add 192.168.0.2/24 dev blue-client
    
    ip -n mptcp-server link set red-server up
    ip -n mptcp-server link set blue-server up
    ip -n mptcp-client link set red-client up
    ip -n mptcp-client link set blue-client up

    You can verify that your setup is working as expected by following the instructions in the first part of this series.

    Working with multiple paths

    Now that you have MPTCP working on a single path, it's time to take multiple paths.

    First, instruct the kernel to set up multiple MPTCP subflows. iproute2 provides a handy mptcp command that can help you:

    # ip -n mptcp-server mptcp endpoint flush
    # ip -n mptcp-server mptcp limits set subflow 2 add_addr_accepted 2
    # ip -n mptcp-client mptcp endpoint flush
    # ip -n mptcp-client mptcp limits set subflow 2 add_addr_accepted 2
    # ip -n mptcp-client mptcp endpoint add 192.168.0.2 dev blue-client id 1 subflow

    These commands configure the MPTCP server to accept up to two different subflows and then add a second subflow to the client. To verify that everything is configured as expected, use iproute2 again:

    # ip -n mptcp-server mptcp limit show
    add_addr_accepted 2 subflows 2
    # ip -n mptcp-client mptcp limit show
    add_addr_accepted 2 subflows 2
    # ip -n mptcp-client mptcp endpoint show
    192.168.0.2 id 1 subflow dev blue-client

    Now you are ready to use subflows. To test them, you can use ncat. The following command starts an ncat server instance on the mptcp-server:

    # ip netns exec mptcp-server ncat -k -4 -i 30 -c "sleep 60" -C -o /tmp/server -l 0.0.0.0 4321 &

    Next, the following command sends a "hello world!" message to the server from the mptcp-client namespace:

    $ ip netns exec mptcp-client ncat -c "echo hello world!" 10.0.0.1 4321

    Using tcpdump, you can verify that there are two different three-way handshakes from different interfaces:

    # tcpdump --number -tnnr /tmp/mptcp.pcap
    reading from file /tmp/mptcp.pcap, link-type LINUX_SLL (Linux cooked v1)
    dropped privs to tcpdump
    1  IP 10.0.0.2.43474 > 10.0.0.1.4321: Flags [S], seq 908898843, win 29200, options [mss 1460,sackOK,TS val 3701631927 ecr 0,nop,wscale 7,mptcp capable[bad opt]>
    2  IP 10.0.0.1.4321 > 10.0.0.2.43474: Flags [S.], seq 3314791626, ack 908898844, win 28960, options [mss 1460,sackOK,TS val 3198006599 ecr 3701631927,nop,wscale 7,mptcp capable Unknown Version (1)], length 0
    3  IP 10.0.0.2.43474 > 10.0.0.1.4321: Flags [.], ack 1, win 229, options [nop,nop,TS val 3701631927 ecr 3198006599,mptcp capable Unknown Version (1)], length 0
    4  IP 10.0.0.2.43474 > 10.0.0.1.4321: Flags [P.], seq 1:14, ack 1, win 229, options [nop,nop,TS val 3701631928 ecr 3198006599,mptcp capable[bad opt]>
    5  IP 10.0.0.1.4321 > 10.0.0.2.43474: Flags [.], ack 14, win 227, options [nop,nop,TS val 3198006600 ecr 3701631928,mptcp dss ack 3158259848540329265], length 0
    6  IP 192.168.0.2.36423 > 10.0.0.1.4321: Flags [S], seq 2791202022, win 29200, options [mss 1460,sackOK,TS val 1604001975 ecr 0,nop,wscale 7,mptcp join id 1 token 0xc0715389 nonce 0xcae83dcb], length 0
    7  IP 10.0.0.1.4321 > 192.168.0.2.36423: Flags [S.], seq 637604674, ack 2791202023, win 28960, options [mss 1460,sackOK,TS val 511057212 ecr 1604001975,nop,wscale 7,mptcp join id 0 hmac 0x465e4bf08492bb0c nonce 0x47d18eca], length 0
    8  IP 10.0.0.2.43474 > 10.0.0.1.4321: Flags [.], ack 1, win 229, options [nop,nop,TS val 3701631928 ecr 3198006600,mptcp dss fin ack 641236127 seq 3158259848540329265 subseq 0 len 1,nop,nop], length 0
    9  IP 10.0.0.1.4321 > 10.0.0.2.43474: Flags [.], ack 14, win 227, options [nop,nop,TS val 3198006600 ecr 3701631928,mptcp dss ack 3158259848540329266], length 0
    10  IP 10.0.0.2.43474 > 10.0.0.1.4321: Flags [F.], seq 14, ack 1, win 229, options [nop,nop,TS val 3701631928 ecr 3198006600,mptcp dss ack 641236127], length 0
    11  IP 10.0.0.1.4321 > 10.0.0.2.43474: Flags [.], ack 15, win 227, options [nop,nop,TS val 3198006641 ecr 3701631928,mptcp dss ack 3158259848540329266], length 0

    Indeed, a first three-way handshake takes place with packets 1-3 on the 10.0.0.* path, while a second handshake kicks in with packets 6-8 on the 192.168.0.* path (note the S flag for SYN and the S. for SYN/ACK).

    Tired of manual subflows? mptcpd to the rescue

    By now you are probably tired of having to manually specify endpoints and subflows for each new test you want to do. Luckily, this is not the only way to use MPTCP endpoints.

    Linux kernel version 5.11 is able to send netlink notifications to user space when the kernel receives an add address netlink command. A small user-space application, mptpcd, can tap into these notifications to provide a path management mechanism to user space and control MPTCP behavior. mptpcd will be available with RHEL 9.

    Conclusion

    In the end, Hercules managed to find his way to Athens by simply asking one of the goddesses which path the other goddess would tell him to use. He chose the other path and quickly returned to Athens.

    With MPTCP, you don't need to solve path riddles anymore: You can ask for multiple roads and take them all together to head where you need to go.

    Last updated: November 7, 2024

    Related Posts

    • Multipath TCP on Red Hat Enterprise Linux 8.3: From 0 to 1 subflows

    • Automating the testing process for SystemTap, Part 1: Test automation with libvirt and Buildbot

    Recent Posts

    • How Kafka improves agentic AI

    • How to use service mesh to improve AI model security

    • How to run AI models in cloud development environments

    • How Trilio secures OpenShift virtual machines and containers

    • How to implement observability with Node.js and Llama Stack

    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue