State of the art on using .NET + Docker + Apple M1

[richlander]

A friend of mine asked for an update on what the story was on using .NET container images on Apple M1, assuming your goal was to deploy to x64 services in Azure (or pick your cloud).

There are various challenges, some specific to .NET and others more general. Windows Arm64 (with x64 emulation) is likely identical to this.

Here's what you want:

• High-fidelity dev experience with prod.
• High performance dev experience.
• Intuitive docker gestures.

Today, it is very easy to fall into an Arm64-centric experience on M1 (for any app plat, not just .NET) and then push an Arm64 image to your registry, and then realize it won't work on your x64 cloud hardware. And then you are not sure what happened. Ughh.

Docker desktop defaults to Arm64

Docker on Apple M1 defaults to native architecture, which is Arm64. If the images you are pulling are multi-arch, you'll pull Arm64. If they are single-arch (x64 or Arm64), you'll pull whatever that arch is. That's the right design choice, but also confusing if your deployment target in x64.

All .NET images are multi-arch. We want to enable using the same tags in a variety of environments. That's all good. It however means that you'll always get Arm64 images by default, on Apple M1 machines. Other app platforms will be the same.

I'll show you, using dotnetapp:

% docker build --pull -t dotnetapp .
% docker run --rm dotnetapp
         42
         42              ,d                             ,d
         42              42                             42
 ,adPPYb,42  ,adPPYba, MM42MMM 8b,dPPYba,   ,adPPYba, MM42MMM
a8"    `Y42 a8"     "8a  42    42P'   `"8a a8P_____42   42
8b       42 8b       d8  42    42       42 8PP"""""""   42
"8a,   ,d42 "8a,   ,a8"  42,   42       42 "8b,   ,aa   42,
 `"8bbdP"Y8  `"YbbdP"'   "Y428 42       42  `"Ybbd8"'   "Y428

.NET 6.0.5
Debian GNU/Linux 11 (bullseye)

OSArchitecture: Arm64
ProcessorCount: 4
TotalAvailableMemoryBytes: 3.84 GiB

That's clearly Arm64. We can force building as x64.

% docker build --pull --platform linux/amd64 -t dotnetapp .
% docker run --rm dotnetapp
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
         42
         42              ,d                             ,d
         42              42                             42
 ,adPPYb,42  ,adPPYba, MM42MMM 8b,dPPYba,   ,adPPYba, MM42MMM
a8"    `Y42 a8"     "8a  42    42P'   `"8a a8P_____42   42
8b       42 8b       d8  42    42       42 8PP"""""""   42
"8a,   ,d42 "8a,   ,a8"  42,   42       42 "8b,   ,aa   42,
 `"8bbdP"Y8  `"YbbdP"'   "Y428 42       42  `"Ybbd8"'   "Y428

.NET 6.0.5
Debian GNU/Linux 11 (bullseye)

OSArchitecture: X64
ProcessorCount: 4
TotalAvailableMemoryBytes: 3.84 GiB

That's x64. That's what we wanted for this scenario.

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

That error is coming from Docker. It's there to tell that you're using emulation.

If you want to live the x64 lifestyle on your Apple M1 machine, you need to use the --platform linux/amd64 flag. Perhaps there is a way to enable amd64 as the default. I don't know.

.NET isn't supported in QEMU

The bigger issue is that .NET isn't supported in QEMU. QEMU is the emulator that Docker Desktop uses for emulation, on both x64 (to emulate Arm64) and Arm64 (to emulate x64) machines.

I'll show you the problem, using aspnetapp (building and running as x64):

% docker build --pull --platform linux/amd64 -t aspnetapp .
% docker run --rm -p 8000:80 aspnetapp
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Unhandled exception. System.IO.IOException: Function not implemented
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
   at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value)
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
   at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__1_0()
   at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func`1 changeTokenProducer, Action changeTokenConsumer)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(FileConfigurationSource source)
   at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder)
   at Microsoft.Extensions.Configuration.ConfigurationManager.AddSource(IConfigurationSource source)
   at Microsoft.Extensions.Configuration.ConfigurationManager.Microsoft.Extensions.Configuration.IConfigurationBuilder.Add(IConfigurationSource source)
   at Microsoft.Extensions.Configuration.ConfigurationExtensions.Add[TSource](IConfigurationBuilder builder, Action`1 configureSource)
   at Microsoft.Extensions.Configuration.JsonConfigurationExtensions.AddJsonFile(IConfigurationBuilder builder, IFileProvider provider, String path, Boolean optional, Boolean reloadOnChange)
   at Microsoft.Extensions.Configuration.JsonConfigurationExtensions.AddJsonFile(IConfigurationBuilder builder, String path, Boolean optional, Boolean reloadOnChange)
   at Microsoft.Extensions.Hosting.HostingHostBuilderExtensions.<>c__DisplayClass11_0.<ConfigureDefaults>b__1(HostBuilderContext hostingContext, IConfigurationBuilder config)
   at Microsoft.AspNetCore.Hosting.BootstrapHostBuilder.RunDefaultCallbacks(ConfigurationManager configuration, HostBuilder innerBuilder)
   at Microsoft.AspNetCore.Builder.WebApplicationBuilder..ctor(WebApplicationOptions options, Action`1 configureDefaults)
   at Microsoft.AspNetCore.Builder.WebApplication.CreateBuilder(String[] args)
   at Program.<Main>$(String[] args) in /source/aspnetapp/Program.cs:line 1
qemu: uncaught target signal 6 (Aborted) - core dumped

That's not pretty. We're hoping this QEMU issue gets resolved.

Workaround 1

Use the multi-arch tags as intended, just like in the aspnetapp Dockerfile. Run .NET in containers as Arm64 in Apple M1. .NET has excellent fidelity across architectures so you will get an experience. If you want to deploy images to a registry, either build with --platform linux/amd64 or build in a CI service like GitHub Actions. They will naturally build as x64.

This approach 100% works. You get the best performance on Apple M1 and straightforward gestures. It's the no-compromises option, since emulated QEMU will always be a lot slower than native architecture.

Workaround 2

This workaround forces your Dockerfile to produce x64 assets by default. The SDK runs in whatever arch is chosen for docker build. It enables you to pivot the final image pretty easily with a --build-arg. The intent of this is that docker build will produce an asset that runs in your x64 cloud by default.

I modified the aspnetapp Dockerfile a bit:

ARG ARCH=amd64
ARG TAG=6.0-bullseye-slim-$ARCH
FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build
WORKDIR /source

# copy csproj and restore as distinct layers
COPY *.sln .
COPY aspnetapp/*.csproj ./aspnetapp/
RUN dotnet restore

# copy everything else and build app
COPY aspnetapp/. ./aspnetapp/
WORKDIR /source/aspnetapp
RUN dotnet publish -c release -o /app --no-restore

# final stage/image
FROM mcr.microsoft.com/dotnet/aspnet:$TAG
WORKDIR /app
COPY --from=build /app ./
ENTRYPOINT ["dotnet", "aspnetapp.dll"]

I added the two ARG lines at the start and added $TAG in the last FROM statement. By default, this Dockerfile will produce x64 images by default, even when built on Apple M1 machines.

However, if you build that image and docker run it, it will still fail, as I demonstrated earlier. If you want a good dev experience, you can opt into building and running it as Arm64.

Like this:

% docker build --pull --build-arg ARCH=arm64v8 -t aspnetapp .
% docker run --rm -p 8000:80 aspnetapp

That approach allows you to build an image on your M1 machine -- x64 by default -- and then push to a registry. For some folks, that's might be what they are looking for.

[woachk]

.NET isn't supported in QEMU

Note: On the Rosetta for Linux implementation present on macOS Ventura, .NET is functional.

$ sudo docker run --rm --platform linux/amd64  -it mcr.microsoft.com/dotnet/samples:aspnetapp
Unable to find image 'mcr.microsoft.com/dotnet/samples:aspnetapp' locally
aspnetapp: Pulling from dotnet/samples
42c077c10790: Already exists 
87ed72daf751: Already exists 
5873b5a7554b: Already exists 
016563de4cce: Already exists 
3ce4f4cca166: Pull complete 
f9a0382f79fd: Pull complete 
d98dcc714b37: Pull complete 
Digest: sha256:25c74bf653c681ab74d72d5d712405cf26778e079f66744037d212c83d2863e9
Status: Downloaded newer image for mcr.microsoft.com/dotnet/samples:aspnetapp
warn: Microsoft.AspNetCore.DataProtection.Repositories.FileSystemXmlRepository[60]
      Storing keys in a directory '/root/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed.
warn: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35]
      No XML encryptor configured. Key {0d4fcd8a-bc32-4a7b-b857-bc03304e7d48} may be persisted to storage in unencrypted form.
info: Microsoft.Hosting.Lifetime[14]
      Now listening on: http://[::]:80
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: /app/

I'd vote for approach 1. Servers on the cloud are not always x86 either. :)

And then override via --platform when required, that's a task to be done by the user.

[richlander]

Very nice! This is great to see. I specifically posted this here in the hope that I'd learn something. This is better than I was expecting.

100% agree on all your points.

I guess I need to upgrade to Ventura!

[richlander]

Got a cheatsheet on that @woachk? I installed Ventura but don't see a difference in behavior. I'm sure there is more to it. I looked around and didn't see any instructions anywhere.

Also, why do you use sudo?

[woachk]

@richlander the cheat sheet is at https://developer.apple.com/documentation/virtualization/running_intel_binaries_in_linux_vms_with_rosetta?language=objc and https://developer.apple.com/documentation/virtualization/running_gui_linux_in_a_virtual_machine_on_a_mac?language=objc for the overall VM test project. (Adding Rosetta tidbits to it is quite trivial, as explained on the Rosetta page)

Docker Desktop didn’t pick that up yet (and i’m currently writing an alternative to it anyway, because of licensing reasons notably, but not only that.)

Why I use sudo is because the account that I’m calling Docker from isn’t part of the docker group, and as such can’t communicate with the dockerd daemon otherwise.

[richlander]

OK. I saw all that. I was wondering if there was something more productized from Docker. Good to know that I wasn't missing anything.

OK, the usual reason for using sudo. Just checking.

Do you just install Docker on Linux (in the VM) and then re-direct your docker daemon to it?

[woachk]

Yeah, I just ssh or execute directly from the VM.

[richlander]

Oh. I see. I was fixated on docker desktop (for macOS). Thanks!