Node.js circuit breakers for serverless functions

September 15, 2021
Lucas Holmquist
Related topics:
Node.jsOperators
Related products:
Red Hat OpenShift

Using circuit breakers in Node.js applications helps mitigate failures when an application calls external services. For example, if the application calls a service to get a list of movies, and that service is not running, the circuit breaker helps the application fall back and find a different way to satisfy the request—or at least inform the user that the movie list is unavailable.

In a previous article, I showed how to use circuit breakers with Node.js. Using circuit breakers with Red Hat OpenShift Serverless Functions introduces a new challenge, which we can solve with a slightly different implementation. I'll explain the issue first, then show you how to solve it.

Note: I introduced the circuit breaker pattern for Node.js in my article Fail fast with Opossum circuit breaker in Node.js.

Circuit breakers in serverless functions

When an application uses OpenShift Serverless Functions, the number of running instances can scale to zero. This ensures you don’t pay for resources you aren’t using. The problem is that when a serverless function uses a circuit breaker, such as Opossum for Node.js, the circuit's state and statistics are lost every time the function scales to zero. As a result, you don’t get the benefits of the circuit breaker.

The latest release of Opossum, version 6.2.0, addresses this problem by allowing a circuit to be re-initialized based on the previous state and statistics. For this to work, we need a persistent database.

Failing fast with Opossum 6.2.0

Here is a very simple example of what a circuit might look like when using the Opossum module in Node.js:

const CircuitBreaker = require('opossum');


function asyncFunctionThatCouldFail(x, y) {

  return new Promise((resolve, reject) => {

    // Do something, maybe on the network or a disk

  });

}


const options = {

  timeout: 3000, // If our function takes longer than 3 seconds, trigger a failure

  errorThresholdPercentage: 50, // When 50% of requests fail, trip the circuit

  resetTimeout: 30000 // After 30 seconds, try again.

};

const breaker = new CircuitBreaker(asyncFunctionThatCouldFail, options);


breaker.fire(x, y)

  .then(console.log)

  .catch(console.error);

This example illustrates a function call that could fail. The call could be anything, but it's usually a call to a network service. We'll use this function and a few basic options to create a new circuit breaker instance. Then, using that new circuit breaker instance, we'll call the fire method to execute the protected function.

To learn more about Opossum and circuit breaking in general, check out Opossum’s Readme file.

Using Opossum with Node.js serverless functions

The example that we are going to use can be found at the Opossum with Functions example site. The example is created with the kn command along with the func plugin:

$ kn func create

The example is a very basic function application that returns a greeting when called. You can run it locally or in a Red Hat OpenShift cluster with the serverless operator installed. For this article, I'll show how to run the application both ways. For an introduction to serverless functions on OpenShift, see Create your first serverless function with Red Hat OpenShift Serverless Functions.

Install the Opossum circuit breaker

You can add modules to the example application the same way you would with any Node.js application. Here's the npm install command to install the Opossum circuit breaker:

$ npm install opossum

As stated earlier, we want to be able to initialize our circuit to a previous state that we have saved. For this example, we are going to make a call to the database when the function starts up to find the latest record. If one is there, we can use it when creating the circuit:

const result = await cruds.find(circuitName);


    if (result.rowCount !== 0) {

      circuitExport = JSON.parse(result.rows[0].circuit);

      console.log('init export state', circuitExport.state);

    }



 // Set some circuit breaker options

  const circuitOptions = {

    name: circuitName,

    timeout: 3000, // If name service takes longer than .3 seconds, trigger a failure

    errorThresholdPercentage: 50, // When 50% of requests fail, trip the circuit

    resetTimeout: 10000, // After 10 seconds, try again.

    ...circuitExport // the saved off status/state if any

  };


  // Use a circuit breaker for the external service and define fallback function

  circuit = new Opossum(externalService, circuitOptions);

  circuit.fallback(_ => 'Fallback');

Opossum emits various status messages, so we can either write to the database during these events to save the state of the circuit breaker, or write to the database each time the function is called. The choice depends on your application. For this application, I’ve chosen to save the circuit's state after a call to the function:

async function handle({ name }) {

  // log out the current circuit

  outputCircuitOptions('invoke circuit state before', circuit);


  const result = await circuit.fire();


   // Write to the DB async, so we don't need to hold up the return of the function

  cruds.create(JSON.stringify(circuit.toJSON()), circuitName);



  return `Hello ${result}!`;

}

Set up the database

Whether the application runs locally or in an OpenShift cluster, you need a database where the circuit stores its state. The example uses a PostgreSQL database. We can start an instance of the database relatively easily by using Docker. The command might look something like this:

docker run --name opossum-postgres-db -e POSTGRES_USER=luke -e POSTGRES_PASSWORD=secret -e POSTGRES_DB=opossum -d -p 5432:5432 postgres

Let’s break down some of the options we are passing to the docker command:

  • POSTGRES_USER is the user we use to access the database, in this case luke.
  • POSTGRES_PASSWORD is the password, which is set to secret.
  • POSTGRES_DB is set to opossum.
  • The -p 5432 option looks for the database on port 5432.
  • postgres is the Docker image that is used.

You can set up the database on an OpenShift cluster using just a few clicks in the developer console. First, click on the +Add link in the left sidebar shown in Figure 1.

Pressing the +Add button allows you to add a new component to your cluster.
Figure 1: Adding a new component in the OpenShift developer console.

Then, select Database from the Developer Catalog and choose PostgreSQL (Ephemeral) for the database, as shown in Figure 2.

The Openshift Developer Catalog offers a large choice of databases.
Figure 2: Choosing a database in the OpenShift developer console.

Click on Instantiate Template and use the same values that were used in the local installation for user, password, and database. Keep the defaults for the other values (Figure 3).

The Instantiate Template button opens a dialog where information about the component can be entered.
Figure 3: Instantiating a new database based on an OpenShift template.

Run the function

Once the database has been created, you can run the application locally by issuing an npm run local command. This will start a Node.js process that can be accessed at http://localhost:8080. Each visit to that URL displays the current state of the circuit. The example also listens for various events emitted by Opossum and logs the state of the circuit each time. The output might look something like this:

{

  name: 'funtimes',

  enabled: true,

  closed: false,

  open: true,

  halfOpen: false,

  warmUp: false,

  shutdown: false

}

The "external" function called here is a simple function that returns either a resolved or a rejected promise. You can change the state of this external function by navigating to the http://localhost:8080?service=true endpoint (which reports a running circuit) or the http://localhost:8080?service=false endpoint (which reports a failed circuit).

To see how Opossum re-initializes itself, visit http://localhost:8080?service=false to fail the service and then visit http://localhost:8080 a few times to invoke it. You will see that the function is failing and falling back.

The next step is to restart the Node.js process. When it comes back up, the initial state in the logs should show that the circuit is open:

{

...

Open: true

...

}

Deploy and test the function

You can deploy the function to OpenShift with the kn func deploy command, and test it by interacting with the application just as with the local version. When you enter the URLs, instead of the hostname you used on your system (http://localhost), you have to use the hostname assigned within your cluster.

Conclusion

This article showed how to use a circuit breaker such as Opossum inside a Node.js serverless function and persist and reload its state. See the following resources to learn more about serverless functions and Node.js:

Last updated: September 20, 2023