Well, it finally happened. Despite the added assurances of working with containers and Kubernetes, the old "It works on my machine" scenario reared its ugly head in my .NET Core (C#) code. The image that I created worked fine on my local PC—a Fedora 32 machine—but it crashed when I tried running it in my Red Hat OpenShift cluster.
The error was "Unable to obtain lock file access on /tmp/NuGetScratch." Let's take a quick look at what happened, and then I'll explain how I fixed it.
After a lot of web searching and a discussion with a Red Hat .NET Core engineer, I discovered the underlying problem. It turns out that within a container, the identity used to initially run the program (using the
dotnet run command) must be the same for subsequent users.
The problem might be easy to understand, but what's the solution?
A temporary problem
The solution was not only simple, but it was the right way to do things. Consider the initial Dockerfile that I used to build the image:
FROM registry.access.redhat.com/ubi8/dotnet-31:3.1 USER 1001 RUN mkdir qotd-csharp WORKDIR qotd-csharp ADD . . RUN dotnet publish -c Release EXPOSE 10000 CMD ["dotnet", "run", "/bin/Release/netcoreapp3.1/publish/qotd-csharp.dll"]
Notice the last line, which is the command that is called when you run the image:
CMD ["dotnet", "run", "./bin/Release/netcoreapp3.0/publish/qotd-csharp.dll"]
Because I was using
dotnet run, the .NET Core framework was attempting to access the temporary directory
/tmp/NuGetScratch. Because the user that built the image and the user attempting to run it were not the same, it failed inside of the Kubernetes cluster. The .NET runtime did not have permission to access this directory.
Update the Dockerfile (hint: don't run)
The solution was simple: I just used the following Dockerfile. Once again, notice the final line:
FROM registry.access.redhat.com/ubi8/dotnet-31:3.1 USER 1001 RUN mkdir qotd-csharp WORKDIR qotd-csharp ADD . . RUN dotnet publish -c Release EXPOSE 10000 CMD ["dotnet", "./bin/Release/netcoreapp3.0/publish/qotd-csharp.dll"]
The updated file not only worked, but it worked better.
Because the library (
qotd-csharp.dll) is already built, there's no need to use the
dotnet run command when a simple
dotnet <path-to-dll> is correct. As an added benefit, it starts hundreds of times faster.
Even working with containerized applications on Kubernetes, the "it works on my machine" problem can still arise from time to time, especially in scenarios involving permissions. In this instance, not only did a workaround exist, but it was the right way to run the image. Chalk this up to a PEBKAC error—that is, a problem that exists between the keyboard and the chair. I learned my lesson.
You can read more in the Red Hat Knowledgebase article for this error.Last updated: August 6, 2020