Skip to main content
Redhat Developers  Logo
  • Products

    Platforms

    • Red Hat Enterprise Linux
      Red Hat Enterprise Linux Icon
    • Red Hat AI
      Red Hat AI
    • Red Hat OpenShift
      Openshift icon
    • Red Hat Ansible Automation Platform
      Ansible icon
    • View All Red Hat Products

    Featured

    • Red Hat build of OpenJDK
    • Red Hat Developer Hub
    • Red Hat JBoss Enterprise Application Platform
    • Red Hat OpenShift Dev Spaces
    • Red Hat OpenShift Local
    • Red Hat Developer Sandbox

      Try Red Hat products and technologies without setup or configuration fees for 30 days with this shared Openshift and Kubernetes cluster.
    • Try at no cost
  • Technologies

    Featured

    • AI/ML
      AI/ML Icon
    • Linux
      Linux Icon
    • Kubernetes
      Cloud icon
    • Automation
      Automation Icon showing arrows moving in a circle around a gear
    • View All Technologies
    • Programming Languages & Frameworks

      • Java
      • Python
      • JavaScript
    • System Design & Architecture

      • Red Hat architecture and design patterns
      • Microservices
      • Event-Driven Architecture
      • Databases
    • Developer Productivity

      • Developer productivity
      • Developer Tools
      • GitOps
    • Automated Data Processing

      • AI/ML
      • Data Science
      • Apache Kafka on Kubernetes
    • Platform Engineering

      • DevOps
      • DevSecOps
      • Ansible automation for applications and services
    • Secure Development & Architectures

      • Security
      • Secure coding
  • Learn

    Featured

    • Kubernetes & Cloud Native
      Openshift icon
    • Linux
      Rhel icon
    • Automation
      Ansible cloud icon
    • AI/ML
      AI/ML Icon
    • View All Learning Resources

    E-Books

    • GitOps Cookbook
    • Podman in Action
    • Kubernetes Operators
    • The Path to GitOps
    • View All E-books

    Cheat Sheets

    • Linux Commands
    • Bash Commands
    • Git
    • systemd Commands
    • View All Cheat Sheets

    Documentation

    • Product Documentation
    • API Catalog
    • Legacy Documentation
  • Developer Sandbox

    Developer Sandbox

    • Access Red Hat’s products and technologies without setup or configuration, and start developing quicker than ever before with our new, no-cost sandbox environments.
    • Explore Developer Sandbox

    Featured Developer Sandbox activities

    • Get started with your Developer Sandbox
    • OpenShift virtualization and application modernization using the Developer Sandbox
    • Explore all Developer Sandbox activities

    Ready to start developing apps?

    • Try at no cost
  • Blog
  • Events
  • Videos

Tips for handling localized ranges in regular expressions

April 6, 2023
Carlos O'Donell
Related topics:
C, C#, C++
Related products:
Red Hat Enterprise Linux

Share:

    Developers as well as casual grep users are accustomed to using ranges in regular expressions, such as [a-zA-Z] or [0-9]. However, they often don't realize that these regular expressions harbor problems that can lead to unexpected behavior.

    This article delves into the issues with using ranges in different locales and the solutions sought by developers of various libraries, including the GNU C Library (glibc).

    The problem with regular expression ranges

    Under the POSIX standard, a regular expression using a range expression has unspecified behavior in any locale other than the POSIX locale. This locale applies only to programs or environments whose environment variables for the locale (such as LANG or LC_ALL) specify either POSIX or C, or who don't have those environment variables set at all.

    Of course, this hardly ever happens. Most people specify their country and language when setting up their system and get a locale such as en_US.UTF-8, in this case, indicating U.S. English with UTF-8 characters.

    For most programs and users, therefore, a popular regular expression range such as [a-zA-Z] or [0-9] has undefined and ultimately unreliable behavior. In theory, users should employ bracket expressions such as [[:alpha:]] and [[:digit:]]. In practice, it works as expected in many, but not all, locales.

    What should a library do to support developers and make developing applications easier? We will explore current and upcoming solutions in the next sections.

    Possible solutions and their relationships to POSIX and Unicode

    There are a number of possible solutions. The support for ranges such as [a-zA-Z] in the C (POSIX) locale is a clue that support for the ranges was implemented by early C libraries when ASCII was the norm. Although there are many conflicting solutions, each generally maps to one of the following implementations:

    • Native character order (NCO): This means that a developer looking at a code chart for the character set can logically identify all characters in the range by reviewing, in order, those characters in the code chart from the start of the range to the end of the range.
    • Collation element order (CEO): This means that a developer looking at the locale sources for the current locale can logically identify all characters in the range by reviewing, in order, those characters in the LC_COLLATE definition in the POSIX locale sources (later compiled into the binary locale on your system, e.g., en_US.UTF-8) from the start of the range to the end of the range.
    • Collation sequence order (CSO): This means that a developer looking at, for one definition of a natural language order, a dictionary with said natural language order can logically identify all characters in the range by reviewing, in order, those characters in the dictionary from the start of the range to the end of the range.

    For example, in the article Boost C++ POSIX regular extended expression APIs, the authors implemented CSO with an option to fall back to NCO. As another example, the GNU Awk (Gawk) implementation has two modes: a "traditional" mode that emulates NCO within certain ASCII ranges and a POSIX-based mode that emulates CSO. The Boost and Gawk implementations offer a very similar degree of choice between NCO and CSO.

    In glibc, the implementation is based on the early POSIX specifications that required CEO. In the built-in C and POSIX locale, the NCO and CEO are equivalent because the ASCII character set order can be ordered the same as the collation elements in the locale source specification. The glibc locale for en_US.UTF-8 makes the NCO and CEO equivalent for lowercase Latin characters, uppercase Latin characters, and numbers in order to preserve developer expectations for sorting these ranges; e.g., lowercase Latin characters are not interleaved with uppercase Latin characters.

    CEO and CSO require large element lists and thus add a lot more overhead to implementations than NCO.

    The published ISO 14651 (2020) standard, most recently derived from Unicode 13.0.0 (2020), defines the international string ordering and comparison, and glibc uses this standard as the basis for string collation. The collation element ordering in the ISO standard interleaves lowercase and uppercase characters in such a way that CEO is more aligned to logical groups of letters e.g. A and a, instead of NCO. Direct usage of ISO 14651 in glibc caused regressions due to this grouping; e.g., [a-z] would match A unexpectedly.

    Current and upcoming solutions

    Boost's interface allows one to choose between a logical NCO or CSO (as defined for a single natural language ordering), thus offering two of three solutions listed in the previous section. A user who desires a distinct CEO can create a completely new locale source definition and distribute that to users that want a distinct ordering. Thread-safe locale APIs can be used to set and use the locale on a per-thread basis.

    The APIs implemented by the ICU project support many possible CSOs for a given language, including dictionary sort, address book sort, calendar sort, etc. No single CSO will solve the needs of all users.

    The glibc implementation of CEO does not meet the needs of developers who are either looking at a code chart or applying common-sense logic to natural language ordering. Migrating glibc from CEO to CSO seems like a logical way forward, but the internal implementation will need to be significantly improved to support this transition. The most straightforward first step is a C.UTF-8 that uses NCO in glibc and avoids the overhead of CEO or CSO.

    With the release of glibc 2.35 in February 2022, the project now has an official harmonized and C.UTF-8 that will use NCO for ASCII regular expression ranges and NCO for collation (code-point collation order).

    You can already use this new C.UTF-8 locale in Fedora (starting with Fedora 35).  In the future, C.UTF-8 will be extended to allow rational ranges that cover all code points in NCO.

    Last updated: August 14, 2023

    Related Posts

    • A beginner’s guide to regular expressions with grep

    • Filter content in HTML using regular expressions in grep

    • Regex how-to: Quantifiers, pattern collections, and word boundaries

    • Advanced regex: Capture groups, lookaheads, and lookbehinds

    • Why we added restartable sequences support to glibc in RHEL 9

    Recent Posts

    • A deep dive into Apache Kafka's KRaft protocol

    • Staying ahead of artificial intelligence threats

    • Strengthen privacy and security with encrypted DNS in RHEL

    • How to enable Ansible Lightspeed intelligent assistant

    • Why some agentic AI developers are moving code from Python to Rust

    What’s up next?

    systemd Commands cheat sheet card image

    Users and administrators query and control systemd behavior through the systemctl command. The systemd Commands Cheat Sheet presents the most common uses of systemctl, along with journalctl for displaying information about systemd activities from its logs.

    Get the cheat sheet
    Red Hat Developers logo LinkedIn YouTube Twitter Facebook

    Products

    • Red Hat Enterprise Linux
    • Red Hat OpenShift
    • Red Hat Ansible Automation Platform

    Build

    • Developer Sandbox
    • Developer Tools
    • Interactive Tutorials
    • API Catalog

    Quicklinks

    • Learning Resources
    • E-books
    • Cheat Sheets
    • Blog
    • Events
    • Newsletter

    Communicate

    • About us
    • Contact sales
    • Find a partner
    • Report a website issue
    • Site Status Dashboard
    • Report a security problem

    RED HAT DEVELOPER

    Build here. Go anywhere.

    We serve the builders. The problem solvers who create careers with code.

    Join us if you’re a developer, software engineer, web designer, front-end designer, UX designer, computer scientist, architect, tester, product manager, project manager or team lead.

    Sign me up

    Red Hat legal and privacy links

    • About Red Hat
    • Jobs
    • Events
    • Locations
    • Contact Red Hat
    • Red Hat Blog
    • Inclusion at Red Hat
    • Cool Stuff Store
    • Red Hat Summit
    © 2025 Red Hat

    Red Hat legal and privacy links

    • Privacy statement
    • Terms of use
    • All policies and guidelines
    • Digital accessibility

    Report a website issue