Skip to main content
Redhat Developers  Logo
  • Products

    Featured

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat OpenShift AI
      Red Hat OpenShift AI
    • Red Hat Enterprise Linux AI
      Linux icon inside of a brain
    • Image mode for Red Hat Enterprise Linux
      RHEL image mode
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • Red Hat Developer Hub
      Developer Hub
    • View All Red Hat Products
    • Linux

      • Red Hat Enterprise Linux
      • Image mode for Red Hat Enterprise Linux
      • Red Hat Universal Base Images (UBI)
    • Java runtimes & frameworks

      • JBoss Enterprise Application Platform
      • Red Hat build of OpenJDK
    • Kubernetes

      • Red Hat OpenShift
      • Microsoft Azure Red Hat OpenShift
      • Red Hat OpenShift Virtualization
      • Red Hat OpenShift Lightspeed
    • Integration & App Connectivity

      • Red Hat Build of Apache Camel
      • Red Hat Service Interconnect
      • Red Hat Connectivity Link
    • AI/ML

      • Red Hat OpenShift AI
      • Red Hat Enterprise Linux AI
    • Automation

      • Red Hat Ansible Automation Platform
      • Red Hat Ansible Lightspeed
    • Developer tools

      • Red Hat Trusted Software Supply Chain
      • Podman Desktop
      • Red Hat OpenShift Dev Spaces
    • Developer Sandbox

      Developer Sandbox
      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Secure Development & Architectures

      • Security
      • Secure coding
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
      • View All Technologies
    • Start exploring in the Developer Sandbox for free

      sandbox graphic
      Try Red Hat's products and technologies without setup or configuration.
    • Try at no cost
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • Java
      Java icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • API Catalog
    • Product Documentation
    • Legacy Documentation
    • Red Hat Learning

      Learning image
      Boost your technical skills to expert-level with the help of interactive lessons offered by various Red Hat Learning programs.
    • Explore Red Hat Learning
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Beyond Loom: Weaving new concurrency patterns

Virtual threads, structured concurrency, and scoped values in Java 21

October 3, 2023
Ben Evans
Related topics:
Java
Related products:
Red Hat build of OpenJDK

Share:

    In this article we're going to discuss some new patterns for concurrent systems that are enabled by the new virtual threads feature from Java 21 and some related new features that "follow on" from virtual threads—​specifically Structured Concurrency (JEP 453) and Scoped Values (JEP 446).

    Note that as of JDK 21, both Structured Concurrency and Scoped Values are in Preview state and so cannot really be used in production applications.

    With this limitation in mind, let's start by looking at the patterns that are enabled by virtual threads alone—​which can, therefore, be used without restriction in Java 21 applications.

    Virtual thread concurrency patterns

    Virtual threads were introduced in Java 21 as one of the major outputs from Project Loom and are, perhaps, most similar to goroutines from the Go language.

    The aim of the project was to solve a scalability problem with Java's concurrency model, which was formerly based solely on operating system (a.k.a. platform) threads. This is achieved by introducing a new type of thread, known as a virtual thread (or vthreads for short).

    Vthreads are managed by the JVM and are not permanently associated with a platform thread. They only use a platform thread (called the carrier) when they are actually running, and they are much cheaper to create and destroy than platform threads.

    To operate, vthreads automatically unbind from their carrier whenever they encounter a blocking I/O call.

    One side effect of this, and our first new pattern, is that vthreads should completely remove the need for developers to use the non-blocking form of the NIO APIs directly. Instead, for each task, programs can create a dedicated virtual thread that uses the blocking API and let the runtime sort it out.

    The intent is that this is essentially the same thing as using non-blocking I/O in terms of performance while providing a simpler programming model.

    In fact, avoiding more complex programming models that display asynchronous contagion (such as async-await or colored functions) was a major design goal for Project Loom. (Colored functions were named as such in the blog post What Color is Your Function?, although the author incorrectly assumes that Java is planning to support async-await.)

    At the same time, bringing explicit "reactive approaches" into the JDK was a definite non-goal. The end result was the form of virtual threads that we see in Java 21.

    This great new language feature opens the door to some interesting new patterns. We've already discussed the possible sunsetting of the direct use of non-blocking I/O, but let's look at some others, starting with one of the most obvious:

    Just replace some of your program's threads with vthreads.

    The carrier threads for vthreads come from a ForkJoinPool executor and will yield on (most) blocking operations. This means that for threads that do at least some I/O, then there is a potential performance benefit by switching them to virtual.

    You should remember that performance depends on a lot of things, and vthread's main advantage is programmer convenience and maintainability. So, if we are able to obtain a performance boost by converting some threads to virtual then we have to test the change—​in a real-world complete system—​to ensure that we actually realize the expected benefit.

    Also, remember that Java 21 does not automatically make existing code use virtual threads—unless you explicitly construct a virtual thread, you will always get a platform thread.

    As well as manual creation of virtual threads, there is also a new executor type, which we can get from Executors.newVirtualThreadPerTaskExecutor(). As the name suggests, rather than relying upon a traditional threadpool that is reused for multiple tasks, this executor creates a new virtual thread for each task that is submitted.

    To accommodate this new executor type, the ExecutorService interface is now AutoCloseable—so it can be used in try-with-resources blocks.

    This is a great example of a new pattern that is designed specifically for virtual threads. Executors for platform threads are typically long-lived objects—​because they create threads at startup, which is an expensive operation. Therefore, it doesn't make sense to create them as local objects within a method—​they're much more likely to be seen as (possibly static) fields.

    Virtual threads, on the other hand, are very cheap to create—​they're just Java objects without any permanently allocated native resources. Creating an executor for virtual threads is similarly cheap, and so creating a locally scoped executor avoids a possible performance penalty.

    This leads to code like this example showing the bare bones of a web server, which makes use of the block-scoped virtual thread executor:

        private volatile boolean isShutdown = false;
    
        void handle(Socket socket) {
            // Handle incoming request
        }
    
        void serveVT(ServerSocket serverSocket) throws IOException, InterruptedException {
            try (var executor = Executors.newVirtualThreadPerTaskExecutor()) {
                try {
                    while (!isShutdown) {
                        var socket = serverSocket.accept();
                        executor.submit(() -> handle(socket));
                    }
                } finally {
                    // If there's been an error, or we're interrupted, we stop accepting
                    executor.shutdown();
                }
            }
        }
    
        public shutdown() {
            isShutdown = true;
        }

    The server socket is passed into the main serveVT() method and handles each incoming request by starting a new virtual thread.

    Every request is isolated from every other request, so there is no need to share data or context, and the requests will all complete in bounded time (and require network I/O). We can call this type of operation fire-and-forget, and this pattern is very suitable for implementing a simple web server using vthreads.

    Also note that this code uses the Volatile Shutdown pattern to provide a controlled shutdown of the server—​another thread calls the shutdown() method, and the volatile nature of the boolean ensures that the no more incoming requests will be processed.

    Let's move on to meet the new Preview APIs that arrived in Java 21 and complement virtual threads.

    Structured Concurrency

    The first of the two new APIs is known as Structured Concurrency. This is an API for thread handling, which provides an approach for cooperating tasks (often virtual threads) to be considered and managed collectively as a collection of subtasks.

    One class of problem in concurrent programming is known as data-parallel problems. These are problems where the same operation is applied to a large amount of data, and the operations are (more or less) independent of each other. This is the domain of applicability of such things as Amdahl's Law, a well-known constraint on the ability of parallel computing to speed up computation.

    By contrast, Structured Concurrency is designed for task-parallel problems, which involves different but related subtasks that need to proceed in parallel. The API's affinity to virtual threads means it is primarily useful for tasks involving some amount of I/O (especially calls to remote services). However, the approach is much less useful for operations that act solely (or mostly) on in-memory data because virtual threads will contend with each other for CPU time.

    The general flow for a structured concurrency task looks something like this:

    1. Create a task scope—​the creating thread owns the scope.

    2. Fork concurrent subtasks in the scope (each is a virtual thread).

    3. Scope owner joins the scope (all subtasks) as a unit.

    4. Scope’s join() method blocks until all subtasks have completed.

    5. After joining, owner handles any errors in forks and processes results.

    6. Owner closes the scope.

    It's worth pointing out that the version of Structured Concurrency that shipped in Java 21 included some minor API changes over Java 20. The main one is that fork() now returns a Subtask (which implements Supplier) instead of a bare Future (as it was in Java 20).

    The reason for this new interface, rather than just using Future, is that results are only queried after a join(), because Structured Concurrency treats multiple subtasks as single unit of work. As a result, neither blocking calls to get() nor checked exceptions from subtasks are useful, and so Future is something of an awkward interface (Subtask is a checked-exception-free interface).

    Developers who have previously used the older previews of Structured Concurrency should be aware of this change when upgrading.

    Let's see this in action in an example using the calculation of a stock tip, a record class that we'll define like this:

    record StockTip(String symbol, double sentiment, double delta24) {}

    We'll assume that the strength of the market's attitude to the stock (the sentiment) and the possible change in price over the next 24 hours (the delta24) are to be calculated by some external process. These elements might take some time to compute, and this is likely to involve network traffic.

    We can therefore use structured subtasks to compute them, like this:

    try (var scope = new StructuredTaskScope.ShutdownOnFailure()) {
        Callable<Double> getSentiment = () -> getSentiment(s);
        Subtask<Double> fSentiment = scope.fork(getSentiment);
    
        Callable<Double> getDelta = () -> getDelta24(s);
        Subtask<Double> fDelta = scope.fork(getDelta);
    
        scope.join();
        scope.throwIfFailed();
    
        return new StockTip(s, fSentiment.get(), fDelta.get());
    } catch (ExecutionException | InterruptedException e) {
        throw new RuntimeException(e);
    }

    This follows the general flow for structured concurrency that we established above. Note that closing the scope is handled implicitly via the try-with-resources block—​this shuts down the scope and waits for any straggling subtasks to complete.

    We should also point out a couple of other points.

    Firstly, joining the subtasks can also be canceled by calling a shutdown() method. Secondly, there is also a timed variant of join(), called joinUntil(), which accepts a deadline (as an Instant parameter).

    There are two built-in shutdown policies for the scope (and custom shutdown policies are also supported):

    • Cancel all subtasks if one of them fails (ShutdownOnFailure)

    • Cancel all subtasks if one of them succeeds (ShutdownOnSuccess)

    We met the first of these inbuilt options in our first example, and so let's move on to introduce the other.

    Consider a library method where multiple subtasks are launched (possibly multiple copies of the same subtask), and the first result (from any of the subtasks) will do. The tasks are racing each other to complete, and the rest of the virtual threads should be shut down as soon as the first success occurs, so we should use the ShutdownOnSuccess policy, like this:

    <T> T race(List<Callable<T>> tasks, Instant deadline)
            throws InterruptedException, ExecutionException, TimeoutException {
    
        try (var scope = new StructuredTaskScope.ShutdownOnSuccess<T>()) {
            for (var task : tasks) {
                scope.fork(task);
            }
            return scope.joinUntil(deadline)
                        .result();  // Throw if none of the subtasks completed successfully
        }
    }

    This has an obvious dual, i.e., when all tasks must run to completion (and where a failure of any subtask should cancel the entire task, so we'll use ShutdownOnFailure again):

    <T> List<T> runAll(List<Callable<T>> tasks)
            throws InterruptedException, ExecutionException {
    
        try (var scope = new StructuredTaskScope.ShutdownOnFailure()) {
            List<? extends Subtask<T>> handles =
                tasks.stream().map(scope::fork).toList();
    
            scope.join()
                 .throwIfFailed();  // Propagate exception if any subtask fails
    
            // Here, all tasks have succeeded, so compose their results
            return handles.stream().map(Subtask::get).toList();
        }
    }

    Note that this version of the code rematerializes the results into a List, but it's also possible to imagine a version that had a different terminal operation that reduced the results and returned a single value.

    We can build more complex structures as well—​the subtasks that we created using forks can themselves create scopes (subscopes). This naturally induces a tree structure of scopes and subtasks, which is useful when we want to condense a final value out of a tree of subtasks.

    If, however, the main point of our code is to operate via side effects, then it is possible to use a StructuredTaskScope<Void>—i.e., use a task scope that returns void, such as in this example:

    void serveScope(ServerSocket serverSocket) throws IOException, InterruptedException {
        try (var scope = new StructuredTaskScope<Void>()) {
            try {
                while (true) {
                    final var socket = serverSocket.accept();
                    Callable<Void> task = () -> {
                        handle(socket);
                        return null;
                    };
                    scope.fork(task);
                }
            } finally {
                // If there's been an error or we're interrupted, we stop accepting
                scope.shutdown();  // Close all active connections
                scope.join();
            }
        }
    }

    However, this is arguably often better handled using a fire-and-forget pattern, such as newVirtualThreadPerTaskExecutor(). There are also some small wrinkles with the generics here, ​such as needing to explicitly return null.

    One recurring theme in all the patterns that we have met so far is that using these techniques requires applying design thinking and knowledge of the domain and context of the problem being solved. There is no software tool that can tell with 100% accuracy whether a thread is a good candidate for being converted to a vthread—​that is a task for a human software engineer.

    Likewise, the restructuring of a task into subtasks and the definition of the relevant scopes requires the programmer to have a good understanding of the domain and any data dependencies between the subtasks.

    Let's move on to look at the second of the new APIs that we want to discuss.

    Scoped Values

    In addition to Structured Concurrency, the new Scoped Values API arrived in Java 21 as Preview. Red Hat contributes to many projects within OpenJDK, including Project Loom, and we are the main contributors to the Scoped Values API. Andrew Haley, one of our Distinguished Engineers, is the overall lead for the Scoped Values JEP.

    The Scoped Values API is based on a new class ScopedValue<T> in java.lang, and it represents a binding of a value to a variable within a specific scope. This value is written once and is then immutable on a per-scope basis.

    The bound value can be retrieved at any point down any call chain within the scope, but only within the scope in which it was set—​this provides robustness and a form of encapsulation. In particular, there is no need to explicitly pass the scoped value down the call chain.

    Scoped values can be thought of as invisible parameters that go into every method and are, in turn, passed to every method they invoke. We sometimes say they are implicitly available, but this is a much more controlled (and more Java-ish) form than, say, Scala's implicit method parameters.

    The Scoped Values API can also be thought of as a modern alternative to thread-local variables, but with a number of enhancements, such as immutability. This means that there is no set() method to let faraway code change a scoped value. This also enables possible future runtime optimizations, as the runtime can be certain that a scoped value cannot change.

    Some goals of the API are:

    • To share data within a thread and with child threads

    • Controlled and bounded lifetime of values

    • Lifetimes visible from the structure of code

    • Immutability allows sharing by lots of threads

    • Immutability and explicit lifetime is often a better fit

    It is not necessary for programmers to move away from ThreadLocal, but scoped values combine well with virtual thread patterns, such as fire-and-forget. It therefore seems quite likely that as scoped values are adopted, then ThreadLocal will be gradually replaced for almost all use cases.

    Let's rewrite the virtual thread web server to use scoped values:

    public class ServerSV {
        public final static ScopedValue<Socket> SOCKETSV = ScopedValue.newInstance();
    
        void serve(ServerSocket serverSocket) throws IOException, InterruptedException {
            while (true) {
                var socket = serverSocket.accept();
                ScopedValue.where(SOCKETSV, socket)
                            .run(() -> handle());
            }
        }
    
        private void handle() {
            var socket = SOCKETSV.get();
            // handle incoming traffic
        }
    }

    Note that the handle() method now no longer takes a parameter, and instead, the socket is accessed via the scoped value—​this is the implicit availability we discussed above.

    This example is very simple, as all we're really doing is to replace the parameter passing with a scoped value—​an almost trivial application. The real power of scoped values is that the call chains and the scoping and subscoping can be arbitrarily complex, and the scoped value will still be available.

    Overall, the intent of scoped values is to provide a dynamic scope, a concept that has not been seen in Java before. This approach to scopes is similar to that found in some other languages—​such as shells, Lisp dialects, and Perl.

    We can contrast it with the traditional Java form of scoping—​usually called lexical scoping. This is where the scope of a variable is determined by the structure of the code, usually defined by a matching pair of curly braces.

    Our dynamic scoping example shows a key pattern in action:

    • Using a static final field as a holder for a scoped value

    • Declaring the ScopedValue instance in class scope

    • Creating the dynamic scope (e.g., runWhere()) within a method

    • Using a lambda to define the scope body (where the call chains will live)

    Scoped values are intended to be very useful for passing values like transaction contexts and other examples of ambient context data.

    Scoped values interact well with structure concurrency, as they can be constructed for scope and then rebound by subscopes. Any values that are not rebound will be inherited by the subscope. This technique allows for upgrading access control and similar patterns, such as in this example, where we will consider two security access levels:

    enum SecurityLevel { USER, ADMIN }

    We'll use a scoped value to hold the current security level and another to hold the current request number:

        private static final ScopedValue<SecurityLevel> securitySV = ScopedValue.newInstance();
        private static final ScopedValue<Integer> requestSV = ScopedValue.newInstance();
    
        private final AtomicInteger req = new AtomicInteger();
    
        public void run() {
            ScopedValue.where(securitySV, level())
                    .where(requestSV, req.getAndIncrement())
                    .run(() -> process());
        }

    To demonstrate rebinding, let us assume that admin privileges are not available, and so any attempt to use them will result in a fallback to user privileges:

        private void process() {
            var level = securitySV.get();
            if (level == SecurityLevel.USER) {
                System.out.println("User privileges granted for "+ requestSV.get() +" on: "+ Thread.currentThread());
            } else {
                System.out.println("Admin privileges requested for "+ requestSV.get() +" on: "+ Thread.currentThread());
                System.out.println("System is in lockdown. Falling back to user privileges");
                ScopedValue.where(securitySV, SecurityLevel.USER)
                        .run(() -> process());
            }
        }

    To conclude this article, we should also point out that classes that represent continuations and other low-level building blocks for virtual threads and other components do exist in Java 21. However, they are in the package jdk.internal.vm and so are not intended for direct use by Java programmers as of this release.

    Related Posts

    • What's new for developers in JDK 21

    • Best practices for Java in single-core containers

    • How to run the correct Java version after an update

    • Write operators in Java with JOSDK, Part 4: Upgrading strategies

    • How to migrate your Java applications to Red Hat OpenShift

    • Stages and levels of Java garbage collection

    Recent Posts

    • Meet the Red Hat Node.js team at PowerUP 2025

    • How to use pipelines for AI/ML automation at the edge

    • What's new in network observability 1.8

    • LLM Compressor: Optimize LLMs for low-latency deployments

    • How to set up NVIDIA NIM on Red Hat OpenShift AI

    What’s up next?

    java-nutshell-cover_Share

    Download Java in a Nutshell, the reference guide every Java developer needs at their fingertips. This book helps you get the most out of versions through Java 17, with examples that show how to take advantage of modern Java APIs and development best practices.

    Get the e-book
    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue