.NET container troubleshooting in OpenShift 4

The .NET framework is extremely useful for the deployment of applications given its large amount of libraries and resources associated with the .NET Foundation. That said, Red Hat provides the containerized version of .NET for Red Hat OpenShift. One of its main features is a cross-platform application for web and console applications for Windows, Linux, and macOS, as well as its long-term support and scalable performance.

Microsoft's .NET fundamentals is an excellent reference providing excellent, compiled information about its core usage with extensive examples. 

It provides automatic memory management via a garbage collector (GC), and it can be objective to tuning and investigations. I will list the steps for collecting and analyzing data, including a few troubleshooting suggestions and recommendations that can be useful for some issues.

Deployment example

There are several options to create a .NET container image, as author Tom Deseyn explains in his article. Let's start with an example build for .NET in local development:

s2i --loglevel 5 build https://github.com/redhat-developer/s2i-dotnetcore --context-dir=6.0/build/test/asp-net-hello-world registry.access.redhat.com/ubi8/dotnet-60:6.0-51.20240627154839 dotnet-sample-app
I0704 20:36:41.117474 1172557 build.go:52] Running S2I version "unknown"
I0704 20:36:41.117634 1172557 util.go:70] Getting docker credentials for registry.access.redhat.com/ubi8/dotnet-60:6.0-51.20240627154839
...
I0704 20:37:06.173553 1172557 cleanup.go:33] Removing temporary directory /tmp/s2i3632451278
I0704 20:37:06.173561 1172557 fs.go:307] Removing directory '/tmp/s2i3632451278'
I0704 20:37:06.173723 1172557 build.go:182] Build completed successfully
...
...
$ docker image list
REPOSITORY                                                       TAG                     IMAGE ID       CREATED          SIZE
dotnet-sample-app                                                latest                  dda3eaacd694   31 seconds ago   856MB
...
###Run it:
$ docker run  dotnet-sample-app:latest
---> Running application ...
Hosting environment: Production
Content root path: /opt/app-root/app
Now listening on: http://0.0.0.0:8080
Application started. Press Ctrl+C to shut down.

Otherwise, another alternative is to build an image with the application as follows:

FROM registry.access.redhat.com/ubi8/dotnet-80
WORKDIR /opt/app-root/src 
RUN dotnet new web  && dotnet publish -c Release -o /opt/app-root/publish
WORKDIR /opt/app-root/publish
CMD ["dotnet", "src.dll" ]

From this image, there are two alternative approaches:

1. Send the image to Quay.
2. Load the image directly into Red Hat OpenShift Container Platform.

Push to Quay (or internal registry) as follows:

podman push localhost/image:tag quay.io/user/image <-- default should go as latest
...
Writing manifest to image destination

Deploy the image straight from the local build as follows: 

1. Build image and tag it: $ sudo podman build --authfile=${HOME}/.docker/config.json -f Dockerfile . --tag springboot.
2. Expose registry:
   1. $ oc patch configs.imageregistry.operator.openshift.io/cluster --patch '{"spec":{"defaultRoute":true}}' --type=merge.
   2. $ HOST=$(oc get route default-route -n openshift-image-registry --template='{{ .spec.host }}').
3. Log in via Podman: $ podman login -u kubeadmin -p $(oc whoami -t) --tls-verify=false $HOST.
4. Create an empty image stream: $ oc apply -f is.yaml.
5. Get the image stream: oc get is .
6. Get the local IMAGE ID: $ sudo podman image list.
7. Send the image $ sudo podman push IMAGE_ID $HOST/NAMESPACE/IS_NAME --tls-verify=false.
8. Start the deployment using the IS: $ oc new-app --image-stream=imagestream-name.

Finally, there is another method, using the built-in SDK feature container support for the .NET SDK. Here is an example: 

# install the build-image
$ dotnet tool install -g dotnet-build-image
# build image as below
$ dotnet build-image -t example-app:latest -b ubi

In terms of OpenShift 4, it is possible to use a BuildConfig as an argument the .NET image and the GitHub (and branch) for the build as follows:

$ cat github-buildconfig.yaml
kind: BuildConfig
apiVersion: build.openshift.io/v1
metadata:
  name: quarkus-test-build
  namespace: quarkus
spec:
  nodeSelector: null
  output:
    to:
      kind: ImageStreamTag
      name: 'output-dotnet:latest'
  resources: {}
  successfulBuildsHistoryLimit: 5
  failedBuildsHistoryLimit: 5
  strategy:
    type: Source
    sourceStrategy:
      from:
        kind: ImageStreamTag
        namespace: openshift
        name: 'dotnet-60:6.0-51.20240627154839'
  postCommit: {}
  source:
    type: Git
    git:
      uri: 'https://github.com/redhat-developer/s2i-dotnetcore.git'
      ref: main
    contextDir: 6.0/build/test/asp-net-hello-world
  runPolicy: Serial
...
...
$ cat is.yaml 
apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
  annotations:
    openshift.io/generated-by: OpenShiftNewApp  
  generation: 1
  labels:
    app: dotnet
  name: output-dotnet
  namespace: dotnet
spec:
  lookupPolicy:
    local: false

Then apply the BuildConfig and ImageStream YAMLs so the image can be built and outputted for an ImageStream:

$ oc apply -f is.yaml
imagestream.image.openshift.io/output-dotnet created
...
...
$ oc apply -f github-buildconfig.yaml
buildconfig.build.openshift.io/dotnet-build created
...
...
$ oc get bc
oc NAME           TYPE     FROM       LATEST
dotnet-build   Source   Git@main   1
$ oc get build
NAME             TYPE     FROM          STATUS                       STARTED              DURATION
dotnet-build-2   Source   Git@a0c7ee1   Complete                     About a minute ago   38s
$ oc get pod
NAME                   READY   STATUS       RESTARTS   AGE
dotnet-build-1-build   0/1     Completed    0          93s

Issues and methods

From the previous deployment, in case of memory issues, see the following steps. Note, this will be an explanation of the main solution .NET Core image troubleshooting.

Given a scenario where there is a memory issue, collect the data and for them to be analyzed. Let's break down two steps in the analysis as follows:

• First, the collection of the data inside the container.
• Second, the analysis of this data.

Data collection

For the data collection part, one can use the createdump tool using the .NET application process ID (PID) as the argument. For .NET images built using the OpenShift s2i flow, the PID is 1 (as used in the previous example). If the .NET application does not run as PID 1, you can find it using tools like ps and pidof:

$ oc exec -ti $POD /bin/sh
sh-4.4$ dotnet --list-runtimes
Microsoft.AspNetCore.App 6.0.12
[/usr/lib64/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 6.0.12 
[/usr/lib64/dotnet/shared/Microsoft.NETCore.App]
sh-4.4$ /usr/lib64/dotnet/shared/Microsoft.NETCore.App/6.0.12/createdump
-f /tmp/dump --full 1
[createdump] Gathering state for process 1 dotnet
[createdump] Writing full dump to file /tmp/dump
[createdump] Written 871575552 bytes (212787 pages) to core file
[createdump] Target process is alive
[createdump] Dump successfully written
sh-4.4$ exit
exit
$ oc cp $POD:/tmp/dump /tmp/dump

Data analysis

With the core dump in hand, opening a debug container, you can analyze the core dump file using the dotnet-dump global tool.

Install the global tool:

$ dotnet tool install -g dotnet-dump

Add the core dump and load it on the dotnet-dump diagnostic tool:

$ dotnet dump analyze /tmp/dump
Loading core dump: /tmp/dump ...
    Ready to process analysis commands. Type 'help' to list available commands or 'help [command]' to get detailed help on a command.
    Type 'quit' or 'exit' to exit the session.
    >

To expand the original article to include analysis examples:

1. Get the image:

$ podman pull registry.access.redhat.com/ubi8/dotnet-60:6.0-39

    2. Create a dotnet container with the dump—passing it as an argument:

$ podman run -u 0 --entrypoint /bin/sh -ti -v /path_host/dump.dmp:/tmp/dump.dmp:Z registry.access.redhat.com/ubi8/dotnet-60:6.0-39 

3. Install dotnet tools: $ dotnet tool install -g dotnet-dump
4. Load the dump: $ dotnet-dump analyze /tmp/dump.dmp
5. Select the command (e.g., print stack):

> clrstack                                                                                                           
OS Thread Id: 0x12 (0)
        Child SP               IP Call Site
00007FFF710C1670 00007f68c70de45c [HelperMethodFrame_1OBJ: 00007fff710c1670] System.Threading.Monitor.ObjWait(Int32, System.Object)
00007FFF710C17A0 00007F68523A6BAE System.Threading.Monitor.Wait(System.Object, Int32)
00007FFF710C17F0 00007F68523A588B System.Threading.ManualResetEventSlim.Wait(Int32, System.Threading.CancellationToken)

7. For native investigations, lldb can be useful: $ dnf install lldb; $ lldb -c ./path/to/file+ bt:

sh-4.4# lldb -c ./dump.dmp 
(lldb) target create --core "./dump.dmp"
Core file '/tmp/dump.dmp' (x86_64) was loaded.
(lldb) bt
* thread #1, name = 'dotnet', stop reason = signal SIGSTOP
  * frame #0: 0x00007f68c70de45c libpthread.so.0`pthread_cond_wait@@GLIBC_2.3.2 + 508
    frame #1: 0x00007f68c58feaa3 libcoreclr.so`CorUnix::CPalSynchronizationManager::ThreadNativeWait(CorUnix::_ThreadNativeWaitData*, unsigned int, CorUnix::ThreadWakeupReason*, unsigned int*) + 355    

Final thoughts

In summary, .NET in OpenShift allows for an easy deployment of .NET applications in OpenShift Container Platform. The steps showed a straightforward approach for memory analysis inside a container. This article provided example analyses and steps to collect the data. The process to collect the data is streamlined via Microsoft's tool dotnet dump, including the analyze option. 

Although Red Hat can help generate and collect data in the .NET container, the memory usage won't be covered by the support investigation given the memory is either from the .NET platform or from the application itself; therefore, very unlikely a container issue.

To learn more about .NET, refer to the Microsoft's  Learn platform. In the matter of C#, Tom Deseyn has done a great series about it in the C#11 series, Some more C# 11.

Special thanks to Omair and Tom Deseyn for their collaboration throughout the years on issues with .NET. For any other specific inquiries, please open a case with Red Hat support. Our global team of experts can help you with any issues.