By David Krammer in Tech — 26 Jan 2025

Chiseled Ubuntu & trimmed .net - How I reduced my Docker image size by 74%

I recently came across this talk by Scott Hanselman where he explains the problem with oversized Docker images that can lead to security issues, longer download times & unneeded container sizes. Quite an important and alluring topic.
In this entry I'm going to showcase how I adopted his solution and cut the Docker image size of a full production .net WebApp down by 74%, which problems can occur and how to solve them (or fail to do so).

After the introduction, Hanselman goes on and starts cutting it down step by step (I'll go into detail on some of these later on). Using the example of a .net Hello-World app, he at first cuts away the standalone installation of the .net runtime and compiles the app self-contained, only to let the publish process afterward trim all the functionality (of the app and the runtime) that he doesn't need. It's like removing the 35 programs from your washing machine that you never use. Yes, they're there, but why not get rid of them when you don't even know what they are and - worse - make the machine heavier (this is where the analogy starts to struggle, but you get the idea).
Now with only the bare bone .net code & runtime at hand, he goes on and removes everything from the Docker image's base distro (Ubuntu) that he doesn't need and ends with dismissing the runtime & any remaining OS components entirely by compiling directly to native code. This results in a 9.15MB Docker image.

Sounds impressive. And it is. I watched this on a Saturday evening and was so motivated to do the same that I couldn't wait for Sunday morning to arrive. But it quickly turned out to not be as easy as he makes it look. In the following I'll present to you the results of 3 full weekends of diving through a rabbit hole.

Chiseling Ubuntu

Let's start with the OS inside the Docker image. Hanselman speaks of "chiseled" Ubuntu where (like using a chisel) you remove everything from the operating system that you don't need, ending up with a "distroless" image. Checking out Microsoft's artifact registry, they provide pre-chiseled aspnet-images for Ubuntu that are significantly smaller (174MB, 9.0-noble-chiseled) in comparison to their "standard" version (358MB, 9.0-noble). Although one has to say that the standard alpine image has the same size as the chiseled Ubuntu one (174MB, 9.0-alpine).

To be honest, I didn't care about image sizes before and just used the standard aspnet one, bringing my normal image size to 381MB. This will be the reference point for the future.

Back to the base images. 174MB sounds a lot better than the 358MB, but still quite a chunk. There's got to be room for more chiseling.

At first, we have to acknowledge that there's not the perfectly pre-chiseled base image out there for you because depending on your application's need you'll require a different set of libraries. That's where Hanselman's presentation looks a bit too easy. He's showing the cut-down of a Hello-World app (which is absolutely fine for such a presentation purpose), but doing that for a >17k lines of code app in my case is a bit of a different story with the main issues arising later on when we talk about trimming.

Because I wanted to get rid of all the stuff that I don't even need from Microsoft's chiseled Ubuntu image, I needed to (quite literally) start from scratch.

"Do I need the shell?"

It took me a bit of time, but it's possible to chisel your own distroless image. This blog post by Canonical gives a nice first introduction to how you might do it. They provide a base image chisel that contains Ubuntu & the chiseling tools with which you can cut certain pre-defined packages out of Ubuntu and copy them into an entirely empty Docker image (FROM scratch)

FROM chisel:22.04 as installer
WORKDIR /staging
# Use chisel to cut out the necessary package slices from the
# chisel:22.04 image and store them in the /staging directory
RUN ["chisel", "cut", "--root", "/staging", \
    "base-files_base", \
    "base-files_release-info", \
    "ca-certificates_data", \
    "libc6_libs" ]
FROM scratch
Copy the package slices from the installer image
to the / directory of our chiselled Ubuntu base image
COPY --from=installer [ "/staging/", "/" ]

But I didn't want to rely on them maintaining this pre-built Docker image. I wanted to build that myself. The only thing I needed was the chisel, but where would I get one?

Quite conveniently, the tooling (written in Go) is openly available on GitHub with installation and execution instructions. From there on, I was able to build my own chiseling stage

################## CHISEL ################
FROM golang:latest AS chisel
RUN mkdir /extract
RUN mkdir /extract/tmp
RUN go install github.com/canonical/chisel/cmd/chisel@latest
RUN chisel cut --release ubuntu-22.04 --root /extract/ ca-certificates_data libstdc++6_libs libssl3_libs
################## BASE ###################
FROM scratch AS base
COPY --from=chisel /extract /

The chisel stage installs the chisel tool into a go-container and extracts the packages ca-certificates_data libstdc++6_libs libssl3_libs that I needed to run my ASP.net Core WebApp. Apart from that, there's nothing left of the distro. Not even a shell to enter the container.

How I found out which packages I needed? A lot of trial & error and reading stack traces that called for missing libraries. If you want to search through all packages that you can chisel, run a go container, install the chisel tool and then execute a search with chisel find (e.g. for libssl*). Here's also an overview over all available chisel slices.

docker run --rm -it golang:latest
root@61ca6c532f6b:/go# go install github.com/canonical/chisel/cmd/chisel@latest
root@61ca6c532f6b:/go# chisel find --release ubuntu-22.04 libssl*

One important note: to reduce the image size further, I went without the libicu70_libs package. Omitting it renders your application incapable of using globalization. Mine didn't need it, so I deactivated it in a later step in the Dockerfile (alternative options) through

ENV DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1

Otherwise, the application will not start with an error about the missing icu package.

But that's it. I now had a base image that only contained the bare minimum that's required to run the app. Speaking of app, that's next.

Trimming .net

Before we start trimming .net, we have to understand what a self-contained .net app is.

Normally when you publish a .net app, the target machine has to have the .net runtime installed in order to run the app. A self-contained app, though, brings the runtime with it, so that it doesn't have to be pre-installed on the target machine. The flag --sc true makes this possible when calling dotnet publish (you can also provide a target architecture with -a).

As our above image does not install the .net runtime anymore, this is what we need. But just bringing the .net runtime with the application does not change much compared to pre-installing it. We need to change the chisel for a trimmer and start cutting.

What trimming does and where it fails

As with the distros, the .net runtime (and other packages you reference) brings a lot of functionality that you don't use. Regardless of whether you do a Console.WriteLine in your code, the runtime still keeps this functionality in stock, even though it will never be utilized.

The idea of trimming is now to analyze the application you're publishing, remove all the code that's never called and by that reduce the size of the runtime & application by dozens of MB. In my case 116MB, 397 files (untrimmed) vs 57MB, 288 files (trimmed). This already includes the runtime.

In order to activate the trimming, you need to set PublishTrimmed to true in the root project's .csproj that you publish. To determine which of the projects to trim, you can define a TrimMode partial or full. With the latter all projects will be trimmed, with the former only the projects that are defined as TrimmableAssembly will be trimmed.

<PropertyGroup>
    <PublishTrimmed>true</PublishTrimmed>
    <TrimMode>partial</TrimMode>
</PropertyGroup>
<ItemGroup>
    <TrimmableAssembly Include="Xipona.Api.WebApp" />
    <TrimmableAssembly Include="Xipona.Api.Core" />
    <TrimmableAssembly Include="Xipona.Api.Secrets" />
</ItemGroup>

But you might already have a suspicion where this is going to go wrong: Reflection.

Reflection runs an analysis during runtime over the available members - if they're not there anymore because they were trimmed ... ¯\_(ツ)_/¯

Event when you don't implement reflection yourself, there's a ton of it in a (more or less) standard ASP.net Core WebApp. Be it the (de)serialization of endpoint inputs & outputs, the config binding from IConfiguration or EFCore in general. Let's have a look at how to combat each of these 4 reflection scenarios

Combating failing reflection

1. Custom reflection

This was the one where I was banging my head against the most and in the end gave up. The trimming yielded only a couple 100s of KB in size reduction and this wasn't worth it for me to spend hours on it. You got to do something with the DynamicallyAccessedMembers attribute, but whatever I tried, it still trimmed away the classed that I only accessed via reflection.

2. (De)serialization with System.Text.Json

Endpoint input/output (de)serialization is by default done through reflection in System.Text.Json. In order to eliminate this kind of reflection, you have to implement json source generators. These source generators run a compile-time analyze over the given types and generate serialization logic / data access models.
Let's say you have the following structure (doesn't matter if it's input or output)

public class Parent
{
    public Child Child { get; set; }
}
public class Child
{
    public int Age { get; set; }
}

You create your json source generator by defining all types that need to be (de)serialized through a JsonSerializable attribute (don't forget the int from the Child's Age property.

[JsonSerializable(typeof(Parent))]
[JsonSerializable(typeof(Child))]
[JsonSerializable(typeof(int))]
public partial class MyCustomContext : JsonSerializerContext
{
}

Now, the sources are generated during compile-time. In order to use them, you have to add the generated serializer context to the TypeInfoResolverChain while building the web application.

var builder = WebApplication.CreateBuilder();
builder.Services
    .AddControllers()
    .AddJsonOptions(opt => opt.JsonSerializerOptions.TypeInfoResolverChain
      .Add(MyCustomContext.Default));

But this only goes for (de)serialization of endpoint input/output by the framework. If you're using (de)serialization manually, you have to register the source generators in the respective JsonSerializerOptions.

var options = new JsonSerializerOptions();
options.TypeInfoResolverChain.Add(MyCustomContext.Default);
var serialized = JsonSerializer.Serialize(myData, options);

3. Config binding

Configuration-binding code like this also uses reflection under the hood to map the configuration into the provided object.

var authOptions = new AuthenticationOptions();
configuration.GetSection("Auth").Bind(authOptions);

Resolving this is even easier than fixing the (de)serialization. You just activate the binding generator in the .csproj of every assembly in which you use config binding. Done.

<PropertyGroup>
    <EnableConfigurationBindingGenerator>true</EnableConfigurationBindingGenerator>
</PropertyGroup>

4. EFCore

EFCore is a tricky one. There are pre-compiled queries as experimental features, but all in all EFCore is not fully trimmable (or even AOT compatible). Thus, I also left the EFCore assembly un-trimmed.

Other hardships: package references

The hardest thing that you can encounter, though, is when one of the libraries that you use does not support trimming because they e.g. use serialization without source generators. There's not a lot you can do. I ran into this problem and ended up re-writing & integrating portions of that library into my own code and eventually removed the library reference.

Executing the app

When finally publishing the application, you'll get a ton of files including one that carries the name of your project, without any file extension. That's the one you want to make executable (chmod +x) and run it on container start.

Full Dockerfile:

################## CHISEL ################
FROM golang:latest AS chisel
RUN mkdir /extract
RUN mkdir /extract/tmp
RUN go install github.com/canonical/chisel/cmd/chisel@latest
RUN chisel cut --release ubuntu-22.04 --root /extract/ ca-certificates_data libstdc++6_libs libssl3_libs
################## BASE ###################
FROM scratch AS base
ARG APP_VERSION
COPY --from=chisel /extract /
################## PUBLISH ################
FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:8.0-bookworm-slim AS publish
ARG TARGETARCH
WORKDIR /src
COPY . .
RUN dotnet publish "Xipona.Api.WebApp/Xipona.Api.WebApp.csproj" -c Release -o /app/publish --sc true -a $TARGETARCH
RUN chmod +x ./Xipona.Api.WebApp
################## FINAL ##################
FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENV APP_VERSION=${APP_VERSION}
ENV DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1
ENTRYPOINT ["./Xipona.Api.WebApp"]

Next step AOT

As there's especially no full AOT support for EFCore, I left that on the side for now, but having source generators in place in the first good step in AOT direction. Meaning, I won't achieve the mind-bogglingly low image sizes. For now ;)

Final look

So, where did I end up? After chiseling Ubuntu and trimming .net I went from 381MB to 98.6MB. A big 74% reduction. The image (and, thus, the container) is not only significantly smaller, but also more secure than before. It only contains what's absolutely necessary.
And on top of that I learned in the process a lot about reflection usage in .net and what a possible way towards AOT compilation could look like.