Article

JSON logging updates in Open Liberty 20.0.0.8

August 20, 2020
DevOpsJavaKubernetesMicroservices
Jakub Pomykala
Open Liberty

With Open Liberty 20.0.0.8, you can now customize HTTP access log fields in JSON logs. This feature allows you to include fields from the accessLogging logFormat attribute in your JSON logs. You also can write a JSON log file directly to system.out, without wrapping it in a liberty_message event.

I'll introduce these new features and get you started with using them in Open Liberty 20.0.0.8. To see the list of fixed bugs, visit the GitHub repository for Open Liberty 20.0.0.8.

How to run your apps with Open Liberty 20.0.0.8

If you're using Maven, update the following coordinates to run your apps with Open Liberty 20.0.0.8:

    <dependency>
    <groupId>io.openliberty</groupId>
    <artifactId>openliberty-runtime</artifactId>
    <version>20.0.0.8</version>
    <type>zip </type>
    </dependency>

If you're using Gradle, enter:

dependencies {
    libertyRuntime group: 'io.openliberty', name: 'openliberty-runtime', version: '[20.0.0.8,)'
}

If you're using docker, enter:

FROM open-liberty

See the Open Liberty downloads page for a downloadable archive of Open Liberty 20.0.0.8.

Customize HTTP access log fields in your JSON logs

In Open Liberty, you have the option to format your server logs in either basic or JSON format. When logs are in JSON format, you must specify the sources (message, trace, accessLog, ffdc, or audit) that you want to send to messages.log or console.log and standard-out.

In Open Liberty 20.0.0.8, we've added the option to include fields from the accessLogging logFormat attribute in your JSON logs. Previously, only selected fields were printed in these logs. Now, you can include other NCSA (National Center for Supercomputing Applications) access log fields in your JSON logs. This new feature lets you receive more informative logs that suit your needs.

Customizing JSON access log fields

When logs are in JSON format, you can use the new jsonAccessLogFields logging attribute to specify whether you want your access logs to have the default set of fields or a custom set based on the HTTP accessLogging logFormat attribute. You can use the accessLogging logFormat attribute to define the log fields that you want. You can then send the logs to a log analysis tool, such as the ELK (Elasticsearch, Logstash, Kibana) stack.

As an example, you might specify that you wanted the user ID and request-time fields in your JSON access logs. You could then filter these fields by user ID in Kibana and track performance on a user-by-user basis.

To receive access logs, you must set the property accessLogging or httpAccessLogging. For example, you might set the following attributes in your server.xml:

    <httpEndpoint id="defaultHttpEndpoint" httpPort="9080" httpsPort="9443" host="*">
    <accessLogging logFormat='%R{W} %u %{my_cookie}C %s'/>
    </httpEndpoint>
    <logging messageFormat="json" messageSource="message,accessLog" jsonAccessLogFields="logFormat"/>

In the messages.log file, your access logs would now contain the four fields specified in the accessLogging logFormat attribute (elapsed time, user ID, cookie, and response code):

{
    "type": "liberty_accesslog",
    "host": "192.168.1.15",
    "ibm_userDir": "/you/jennifer.zhen.chengibm.com/libertyGit/open-liberty/dev/build.image/wlp/usr/",
    "ibm_serverName": "defaultServer",
    "ibm_cookie_my_cookie": "example_cookie",
    "ibm_responseCode": 200,
    "ibm_datetime": "2020-06-18T09:30:47.693-0400",
    "ibm_sequence": "1592487047653_0000000000001"
}

You can also integrate this new functionality with Open Liberty's logstashCollector-1.0 feature by adding the following to your server.xml:

    <featureManager>
    <feature>logstashCollector-1.0</feature>
    </featureManager>

    <logstashCollector
    jsonAccessLogFields="logFormat">
    </logstashCollector>

Complete list of the new access log fields

This table describes the new fields available with the corresponding logFormat token:

Field Description logFormat token
ibm_remoteIP Remote IP address; e.g., 127.0.0.1. %a
ibm_bytesSent Response size in bytes excluding headers. %b
ibm_cookie_{cookiename} Cookie value from the request. %{cookieName}C or %C
ibm_requestElapsedTime The elapsed time of the request: millisecond accuracy, microsecond precision. %D
ibm_requestHeader_{headername} Header value from the request. %{headerName}i
ibm_responseHeader_{headername} Header value from the response. %{headerName}o
ibm_requestFirstLine The first line of the request. %r
ibm_requestStartTime The start time of the request, in NCSA format.

%t
ibm_accessLogDatetime The time when the message to the access log is queued to be logged, in normal NCSA format. %{t}W
ibm_remoteUserID Remote user according to the WebSphere Application Server specific $WSRU header. %u

See the documentation for Open Liberty logging, and Open Liberty log and trace configuration for more information.

Write pre-formatted JSON application logs directly to System.out or System.err

Prior to this release, Open Liberty embedded any pre-formatted JSON application logs written to System.out or System.err into the message field of a liberty_message event. Now, you can write these logs directly to System.out or System.err without having them wrapped in a liberty_message event. You can then send the logs to a log analysis tool, such as the ELK (Elasticsearch, Logstash, Kibana) stack.

Here's an example of a pre-formatted JSON log prior to this release:

{
    "type":"liberty_message",
    "host":"192.168.0.119",
    "ibm_userDir":"\/you\/yushan.lin@ibm.com\/Documents\/archived-guide-log4j\/finish\/target\/liberty\/wlp\/usr\
    ",
    "ibm_serverName":"log4j.sampleServer",
    "message":"{\n   \"timeMillis\" : 1587666082123,\n
            \"thread\" : \"Default Executor-thread-8\",\n
            \"level\" : \"WARN\",\n
            \"loggerName\" : \"application.servlet.LibertyServlet\",\n
            \"message\" : \"hello liberty servlet warning message!\",\n
            \"endOfBatch\" : false,\n
            \"loggerFqcn\" : \"org.apache.logging.log4j.spi.AbstractLogger\",\n
            \"threadId\" : 53,\n
            \"threadPriority\" : 5\n}\r",
    "ibm_threadId":"00000035",
    "ibm_datetime":"2020-04-23T14:21:22.124-0400",
    "module":"SystemOut",
    "loglevel":"SystemOut",
    "ibm_methodName":"",
    "ibm_className":"",
    "ibm_sequence":"1587666082124_000000000001B",
    "ext_thread":"Default Executor-thread-8”
}

You can now output the JSON application logs so that they are not wrapped in liberty_message events. Enable this functionality by setting appsWriteJson="true" in the logging element of the server.xml. Another option is to have it set from the moment the server starts by setting the bootstrap.properties to: com.ibm.ws.logging.apps.write.json=true.

For more information about this new logging feature, see Open Liberty logging.

Try Open Liberty 20.0.0.8 in Red Hat Runtimes now

Open Liberty is part of the Red Hat Runtimes offering and is available to Red Hat Runtimes subscribers. To learn more about deploying Open Liberty applications to OpenShift, take a look at our Open Liberty guide: Deploying microservices to OpenShift.

Last updated: January 12, 2024