ec2: NatProvider.instanceV2 UserData fails due to dnf race condition

[tmokmss]

Describe the bug

When using NatProvider.instanceV2 to create a NAT instance, the UserData script sometimes fails to install iptables-services on Amazon Linux 2023. This is caused by a race condition where another process runs dnf clean all during the yum install transaction, deleting the downloaded package cache before the installation completes.

The NAT instance becomes non-functional because iptables is not installed and configured.

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Library Version

No response

Expected Behavior

The UserData script generated by NatProvider.instanceV2 should successfully install iptables-services and configure the NAT instance, even when other dnf/yum processes are running during cloud-init.

Current Behavior

The yum install iptables-services -y command fails with the following error:

[Errno 2] No such file or directory: '/var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/packages/iptables-libs-1.8.8-3.amzn2023.0.2.aarch64.rpm'

Subsequent commands also fail:

Failed to enable unit: Unit file iptables.service does not exist.
Failed to start iptables.service: Unit iptables.service not found.
sudo: /sbin/iptables: command not found

Reproduction Steps

import { NatProvider, InstanceType, InstanceClass, InstanceSize, Vpc } from 'aws-cdk-lib/aws-ec2';

const vpc = new Vpc(this, 'Vpc', {
  natGatewayProvider: NatProvider.instanceV2({
    instanceType: InstanceType.of(InstanceClass.T4G, InstanceSize.NANO),
  }),
  natGateways: 1,
  maxAzs: 2,
});

1. Deploy the stack. The race condition sometimes happens (with <10% probability I suppose)
2. SSH into the NAT instance
3. Check /var/log/cloud-init-output.log for iptables installation errors
4. Check /var/log/dnf.log for the race condition evidence

Possible Solution

Add retry logic to the UserData script to handle transient dnf/yum failures.

Additional Information/Context

Root cause analysis from /var/log/dnf.log:

The timeline shows that during the yum install iptables-services transaction, another process executed dnf clean all which deleted the cached RPM files:

01:04:04 - yum install iptables-services -y started
01:04:20 - yum -y remove audit started (separate process, likely Amazon Linux system initialization)
01:04:21 - iptables-services packages downloaded to cache
01:04:22 - dnf clean all executed by another process, removing cached packages
01:04:22 - iptables-services transaction check fails - FileNotFoundError

The log shows the exact moment the cache was deleted:

2026-01-19T01:04:22+0000 DDEBUG Command: dnf clean all
2026-01-19T01:04:22+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/packages/iptables-libs-1.8.8-3.amzn2023.0.2.aarch64.rpm

This issue occurs on Amazon Linux 2023 where multiple system initialization scripts run in parallel during cloud-init. The race condition is timing-dependent and may not occur on every deployment, making it difficult to reproduce consistently.

AWS CDK Library version (aws-cdk-lib)

2.232.2

AWS CDK CLI version

2.1034.0

Node.js Version

v24.11.1

OS

macOS

Language

TypeScript

Language Version

No response

Other information

Complete execution logs

cloud-init-output.log

[root@ip-10-0-51-107 bin]# cat /var/log/cloud-init-output.log
Cloud-init v. 22.2.2 running 'init' at Mon, 19 Jan 2026 01:04:01 +0000. Up 5.03 seconds.
ci-info: ++++++++++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++++++++++
ci-info: +--------+------+-----------------------------+---------------+--------+-------------------+
ci-info: | Device |  Up  |           Address           |      Mask     | Scope  |     Hw-Address    |
ci-info: +--------+------+-----------------------------+---------------+--------+-------------------+
ci-info: |  ens5  | True |         10.0.51.107         | 255.255.192.0 | global | 06:6e:7f:26:c9:b1 |
ci-info: |  ens5  | True | fe80::46e:7fff:fe26:c9b1/64 |       .       |  link  | 06:6e:7f:26:c9:b1 |
ci-info: |   lo   | True |          127.0.0.1          |   255.0.0.0   |  host  |         .         |
ci-info: |   lo   | True |           ::1/128           |       .       |  host  |         .         |
ci-info: +--------+------+-----------------------------+---------------+--------+-------------------+
ci-info: ++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++
ci-info: +-------+-------------+----------+-----------------+-----------+-------+
ci-info: | Route | Destination | Gateway  |     Genmask     | Interface | Flags |
ci-info: +-------+-------------+----------+-----------------+-----------+-------+
ci-info: |   0   |   0.0.0.0   | 10.0.0.1 |     0.0.0.0     |    ens5   |   UG  |
ci-info: |   1   |   10.0.0.0  | 0.0.0.0  |  255.255.192.0  |    ens5   |   U   |
ci-info: |   2   |   10.0.0.1  | 0.0.0.0  | 255.255.255.255 |    ens5   |   UH  |
ci-info: |   3   |   10.0.0.2  | 0.0.0.0  | 255.255.255.255 |    ens5   |   UH  |
ci-info: +-------+-------------+----------+-----------------+-----------+-------+
ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
ci-info: +-------+-------------+---------+-----------+-------+
ci-info: | Route | Destination | Gateway | Interface | Flags |
ci-info: +-------+-------------+---------+-----------+-------+
ci-info: |   0   |  fe80::/64  |    ::   |    ens5   |   U   |
ci-info: |   2   |    local    |    ::   |    ens5   |   U   |
ci-info: |   3   |  multicast  |    ::   |    ens5   |   U   |
ci-info: +-------+-------------+---------+-----------+-------+
Generating public/private ed25519 key pair.
Your identification has been saved in /etc/ssh/ssh_host_ed25519_key
Your public key has been saved in /etc/ssh/ssh_host_ed25519_key.pub
The key fingerprint is:
SHA256:rHFKZ62EYMkXa7Ay/ZdEp9XaxCheuXI/WWekvNm+ABI root@ip-10-0-51-107.ap-northeast-1.compute.internal
The key's randomart image is:
+--[ED25519 256]--+
|    . . . o=     |
|   o + +.++ +   .|
|  o B +.oE = . o |
|   + = +oo= . + o|
|      = S+.o o * |
|     . X .. = o .|
|      o .    o . |
|              . .|
|               ..|
+----[SHA256]-----+
Generating public/private ecdsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_ecdsa_key
Your public key has been saved in /etc/ssh/ssh_host_ecdsa_key.pub
The key fingerprint is:
SHA256:sHwGLhczfG4sOmAc+zXX8Zrlbn0EP99K2Ia6+zy6F80 root@ip-10-0-51-107.ap-northeast-1.compute.internal
The key's randomart image is:
+---[ECDSA 256]---+
|                 |
|     .           |
|  .   B . .      |
| . o o @ . o  .  |
|  = . O S . oo o |
| . o = B   =.+E.o|
|    +     o +o+.+|
|     .     ++o. +|
|          *O+..o |
+----[SHA256]-----+
Cloud-init v. 22.2.2 running 'modules:config' at Mon, 19 Jan 2026 01:04:02 +0000. Up 6.42 seconds.
Cloud-init v. 22.2.2 running 'modules:final' at Mon, 19 Jan 2026 01:04:03 +0000. Up 6.90 seconds.
Amazon Linux 2023 repository                     81 MB/s |  52 MB     00:00
Amazon Linux 2023 Kernel Livepatch repository   223 kB/s |  28 kB     00:00
Dependencies resolved.
================================================================================
 Package                  Arch      Version                 Repository     Size
================================================================================
Installing:
 iptables-services        noarch    1.8.8-3.amzn2023.0.2    amazonlinux    18 k
Installing dependencies:
 iptables-libs            aarch64   1.8.8-3.amzn2023.0.2    amazonlinux   424 k
 iptables-nft             aarch64   1.8.8-3.amzn2023.0.2    amazonlinux   185 k
 iptables-utils           aarch64   1.8.8-3.amzn2023.0.2    amazonlinux    43 k
 libnetfilter_conntrack   aarch64   1.0.8-2.amzn2023.0.2    amazonlinux    58 k
 libnfnetlink             aarch64   1.0.1-19.amzn2023.0.2   amazonlinux    30 k
 libnftnl                 aarch64   1.2.2-2.amzn2023.0.2    amazonlinux    83 k

Transaction Summary
================================================================================
Install  7 Packages

Total download size: 841 k
Installed size: 25 M
Downloading Packages:
(1/7): iptables-services-1.8.8-3.amzn2023.0.2.n 352 kB/s |  18 kB     00:00
(2/7): iptables-libs-1.8.8-3.amzn2023.0.2.aarch 7.0 MB/s | 424 kB     00:00
(3/7): iptables-nft-1.8.8-3.amzn2023.0.2.aarch6 2.8 MB/s | 185 kB     00:00
(4/7): libnetfilter_conntrack-1.0.8-2.amzn2023. 2.3 MB/s |  58 kB     00:00
(5/7): iptables-utils-1.8.8-3.amzn2023.0.2.aarc 1.2 MB/s |  43 kB     00:00
(6/7): libnfnetlink-1.0.1-19.amzn2023.0.2.aarch 935 kB/s |  30 kB     00:00
(7/7): libnftnl-1.2.2-2.amzn2023.0.2.aarch64.rp 2.6 MB/s |  83 kB     00:00
--------------------------------------------------------------------------------
Total                                           4.6 MB/s | 841 kB     00:00
Running transaction check
[Errno 2] No such file or directory: '/var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/packages/iptables-libs-1.8.8-3.amzn2023.0.2.aarch64.rpm'
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'yum clean packages'.
Failed to enable unit: Unit file iptables.service does not exist.
Failed to start iptables.service: Unit iptables.service not found.
net.ipv4.ip_forward = 1
sudo: /sbin/iptables: command not found
sudo: /sbin/iptables: command not found
The service command supports only basic LSB actions (start, stop, restart, try-restart, reload, reload-or-restart, try-reload-or-restart, force-reload, status, condrestart). For other actions, please try to use systemctl.
2026-01-19 01:04:23,074 - cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
2026-01-19 01:04:23,076 - util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python3.9/site-packages/cloudinit/config/cc_scripts_user.py'>) failed
Cloud-init v. 22.2.2 finished at Mon, 19 Jan 2026 01:04:23 +0000. Datasource DataSourceEc2.  Up 26.76 seconds

[root@ip-10-0-51-107 bin]# cat /var/lib/cloud/instances/i-0dbc85ad8c34169d0/user-data.txt
#!/bin/bash
yum install iptables-services -y
systemctl enable iptables
systemctl start iptables
echo "net.ipv4.ip_forward=1" > /etc/sysctl.d/custom-ip-forwarding.conf
sudo sysctl -p /etc/sysctl.d/custom-ip-forwarding.conf
sudo /sbin/iptables -t nat -A POSTROUTING -o $(route | awk '/^default/{print $NF}') -j MASQUERADE
sudo /sbin/iptables -F FORWARD

dnf.log

[root@ip-10-0-51-107 bin]# cat /var/log/dnf.log
2026-01-19T01:04:03+0000 INFO --- logging initialized ---
2026-01-19T01:04:03+0000 DDEBUG timer: config: 51 ms
2026-01-19T01:04:04+0000 DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, release-notification, repoclosure, repodiff, repograph, repomanage, reposync, supportinfo
2026-01-19T01:04:04+0000 DEBUG YUM version: 4.14.0
2026-01-19T01:04:04+0000 DDEBUG Command: yum install iptables-services -y
2026-01-19T01:04:04+0000 DDEBUG Installroot: /
2026-01-19T01:04:04+0000 DDEBUG Releasever: 2023.10.20260105
2026-01-19T01:04:04+0000 DEBUG cachedir: /var/cache/dnf
2026-01-19T01:04:04+0000 DDEBUG Base command: install
2026-01-19T01:04:04+0000 DDEBUG Extra commands: ['install', 'iptables-services', '-y']
2026-01-19T01:04:04+0000 DEBUG User-Agent: constructed: 'libdnf (Amazon Linux 2023; generic; Linux.aarch64)'
2026-01-19T01:04:04+0000 DEBUG repo: downloading from remote: amazonlinux
2026-01-19T01:04:20+0000 INFO --- logging initialized ---
2026-01-19T01:04:20+0000 DDEBUG timer: config: 26 ms
2026-01-19T01:04:20+0000 DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, release-notification, repoclosure, repodiff, repograph, repomanage, reposync, supportinfo
2026-01-19T01:04:20+0000 DEBUG YUM version: 4.14.0
2026-01-19T01:04:20+0000 DDEBUG Command: yum -y remove audit
2026-01-19T01:04:20+0000 DDEBUG Installroot: /
2026-01-19T01:04:20+0000 DDEBUG Releasever: 2023.10.20260105
2026-01-19T01:04:20+0000 DEBUG cachedir: /var/cache/dnf
2026-01-19T01:04:20+0000 DDEBUG Base command: remove
2026-01-19T01:04:20+0000 DDEBUG Extra commands: ['-y', 'remove', 'audit']
2026-01-19T01:04:20+0000 INFO Waiting for process with pid 1617 to finish.
2026-01-19T01:04:20+0000 DEBUG amazonlinux: using metadata from Wed Dec 31 18:33:08 2025.
2026-01-19T01:04:20+0000 DEBUG repo: downloading from remote: kernel-livepatch
2026-01-19T01:04:20+0000 DEBUG kernel-livepatch: using metadata from Mon Dec 29 22:09:30 2025.
2026-01-19T01:04:20+0000 DDEBUG timer: sack setup: 16482 ms
2026-01-19T01:04:20+0000 DEBUG Completion plugin: Generating completion cache...
2026-01-19T01:04:20+0000 INFO --- logging initialized ---
2026-01-19T01:04:20+0000 DDEBUG timer: config: 2 ms
2026-01-19T01:04:20+0000 DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, release-notification, repoclosure, repodiff, repograph, repomanage, reposync, supportinfo
2026-01-19T01:04:20+0000 DEBUG YUM version: 4.14.0
2026-01-19T01:04:20+0000 DDEBUG Command: yum list installed kernel-livepatch-*
2026-01-19T01:04:20+0000 DDEBUG Installroot: /
2026-01-19T01:04:20+0000 DDEBUG Releasever: 2023.10.20260105
2026-01-19T01:04:20+0000 DEBUG cachedir: /var/cache/dnf
2026-01-19T01:04:20+0000 DDEBUG Base command: list
2026-01-19T01:04:20+0000 DDEBUG Extra commands: ['list', 'installed', 'kernel-livepatch-*']
2026-01-19T01:04:20+0000 DEBUG User-Agent: constructed: 'libdnf (Amazon Linux 2023; generic; Linux.aarch64)'
2026-01-19T01:04:20+0000 DDEBUG timer: sack setup: 137 ms
2026-01-19T01:04:20+0000 DDEBUG Cleaning up.
2026-01-19T01:04:21+0000 DEBUG User-Agent: constructed: 'libdnf (Amazon Linux 2023; generic; Linux.aarch64)'
2026-01-19T01:04:21+0000 DDEBUG timer: sack setup: 1069 ms
2026-01-19T01:04:21+0000 DEBUG --> Starting dependency resolution
2026-01-19T01:04:21+0000 DEBUG --> Finding unneeded leftover dependencies
2026-01-19T01:04:21+0000 DEBUG ---> Package audit.aarch64 3.1.5-1.amzn2023.0.2 will be erased
2026-01-19T01:04:21+0000 DEBUG --> Finished dependency resolution
2026-01-19T01:04:21+0000 DDEBUG timer: depsolve: 41 ms
2026-01-19T01:04:21+0000 INFO Dependencies resolved.
2026-01-19T01:04:21+0000 INFO ================================================================================
 Package      Architecture   Version                       Repository      Size
================================================================================
Removing:
 audit        aarch64        3.1.5-1.amzn2023.0.2          @System        2.0 M

Transaction Summary
================================================================================
Remove  1 Package

2026-01-19T01:04:21+0000 INFO Freed space: 2.0 M
2026-01-19T01:04:21+0000 INFO Running transaction check
2026-01-19T01:04:21+0000 INFO Transaction check succeeded.
2026-01-19T01:04:21+0000 INFO Running transaction test
2026-01-19T01:04:21+0000 INFO Transaction test succeeded.
2026-01-19T01:04:21+0000 DDEBUG timer: transaction test: 104 ms
2026-01-19T01:04:21+0000 INFO Running transaction
2026-01-19T01:04:21+0000 DEBUG RPMDB altered outside of YUM.
2026-01-19T01:04:21+0000 DDEBUG RPM transaction start.
2026-01-19T01:04:21+0000 DEBUG --> Starting dependency resolution
2026-01-19T01:04:21+0000 DEBUG ---> Package iptables-libs.aarch64 1.8.8-3.amzn2023.0.2 will be installed
2026-01-19T01:04:21+0000 DEBUG ---> Package iptables-nft.aarch64 1.8.8-3.amzn2023.0.2 will be installed
2026-01-19T01:04:21+0000 DEBUG ---> Package iptables-services.noarch 1.8.8-3.amzn2023.0.2 will be installed
2026-01-19T01:04:21+0000 DEBUG ---> Package iptables-utils.aarch64 1.8.8-3.amzn2023.0.2 will be installed
2026-01-19T01:04:21+0000 DEBUG ---> Package libnetfilter_conntrack.aarch64 1.0.8-2.amzn2023.0.2 will be installed
2026-01-19T01:04:21+0000 DEBUG ---> Package libnfnetlink.aarch64 1.0.1-19.amzn2023.0.2 will be installed
2026-01-19T01:04:21+0000 DEBUG ---> Package libnftnl.aarch64 1.2.2-2.amzn2023.0.2 will be installed
2026-01-19T01:04:21+0000 DEBUG --> Finished dependency resolution
2026-01-19T01:04:21+0000 DDEBUG timer: depsolve: 69 ms
2026-01-19T01:04:21+0000 INFO Dependencies resolved.
2026-01-19T01:04:21+0000 INFO ================================================================================
 Package                  Arch      Version                 Repository     Size
================================================================================
Installing:
 iptables-services        noarch    1.8.8-3.amzn2023.0.2    amazonlinux    18 k
Installing dependencies:
 iptables-libs            aarch64   1.8.8-3.amzn2023.0.2    amazonlinux   424 k
 iptables-nft             aarch64   1.8.8-3.amzn2023.0.2    amazonlinux   185 k
 iptables-utils           aarch64   1.8.8-3.amzn2023.0.2    amazonlinux    43 k
 libnetfilter_conntrack   aarch64   1.0.8-2.amzn2023.0.2    amazonlinux    58 k
 libnfnetlink             aarch64   1.0.1-19.amzn2023.0.2   amazonlinux    30 k
 libnftnl                 aarch64   1.2.2-2.amzn2023.0.2    amazonlinux    83 k

Transaction Summary
================================================================================
Install  7 Packages

2026-01-19T01:04:21+0000 INFO Total download size: 841 k
2026-01-19T01:04:21+0000 INFO Installed size: 25 M
2026-01-19T01:04:21+0000 INFO Downloading Packages:
2026-01-19T01:04:21+0000 DDEBUG RPM transaction over.
2026-01-19T01:04:21+0000 INFO --------------------------------------------------------------------------------
2026-01-19T01:04:21+0000 INFO Total                                           4.6 MB/s | 841 kB     00:00
2026-01-19T01:04:21+0000 DEBUG Using rpmkeys executable at /usr/bin/rpmkeys to verify signatures
2026-01-19T01:04:21+0000 INFO --- logging initialized ---
2026-01-19T01:04:21+0000 DDEBUG timer: config: 28 ms
2026-01-19T01:04:22+0000 DDEBUG timer: verify transaction: 189 ms
2026-01-19T01:04:22+0000 DDEBUG timer: transaction: 638 ms
2026-01-19T01:04:22+0000 DEBUG Completion plugin: Generating completion cache...
2026-01-19T01:04:22+0000 DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, release-notification, repoclosure, repodiff, repograph, repomanage, reposync, supportinfo
2026-01-19T01:04:22+0000 DEBUG DNF version: 4.14.0
2026-01-19T01:04:22+0000 DDEBUG Command: dnf clean all
2026-01-19T01:04:22+0000 DDEBUG Installroot: /
2026-01-19T01:04:22+0000 DDEBUG Releasever: 2023.10.20260105
2026-01-19T01:04:22+0000 DEBUG cachedir: /var/cache/dnf
2026-01-19T01:04:22+0000 DDEBUG Base command: clean
2026-01-19T01:04:22+0000 DDEBUG Extra commands: ['clean', 'all']
2026-01-19T01:04:22+0000 DEBUG Cleaning data: packages metadata dbcache
2026-01-19T01:04:22+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux.solv
2026-01-19T01:04:22+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-filenames.solvx
2026-01-19T01:04:22+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-updateinfo.solvx
2026-01-19T01:04:22+0000 DDEBUG Removing file /var/cache/dnf/kernel-livepatch.solv
2026-01-19T01:04:22+0000 DDEBUG Removing file /var/cache/dnf/kernel-livepatch-filenames.solvx
2026-01-19T01:04:22+0000 DDEBUG Removing file /var/cache/dnf/kernel-livepatch-updateinfo.solvx
2026-01-19T01:04:22+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/mirrorlist
2026-01-19T01:04:22+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/repodata/repomd.xml
2026-01-19T01:04:22+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/repodata/primary.xml.gz
2026-01-19T01:04:22+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/repodata/filelists.xml.gz
2026-01-19T01:04:22+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/repodata/updateinfo.xml.gz
2026-01-19T01:04:22+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/repodata/comps.xml.gz
2026-01-19T01:04:22+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/packages/iptables-libs-1.8.8-3.amzn2023.0.2.aarch64.rpm
2026-01-19T01:04:22+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/packages/iptables-nft-1.8.8-3.amzn2023.0.2.aarch64.rpm
2026-01-19T01:04:22+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/packages/iptables-services-1.8.8-3.amzn2023.0.2.noarch.rpm
2026-01-19T01:04:22+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/packages/iptables-utils-1.8.8-3.amzn2023.0.2.aarch64.rpm
2026-01-19T01:04:22+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/packages/libnetfilter_conntrack-1.0.8-2.amzn2023.0.2.aarch64.rpm
2026-01-19T01:04:22+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/packages/libnfnetlink-1.0.1-19.amzn2023.0.2.aarch64.rpm
2026-01-19T01:04:22+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/packages/libnftnl-1.2.2-2.amzn2023.0.2.aarch64.rpm
2026-01-19T01:04:22+0000 DDEBUG Removing file /var/cache/dnf/kernel-livepatch-0880f278187beb37/mirrorlist
2026-01-19T01:04:22+0000 DDEBUG Removing file /var/cache/dnf/kernel-livepatch-0880f278187beb37/repodata/repomd.xml
2026-01-19T01:04:22+0000 DDEBUG Removing file /var/cache/dnf/kernel-livepatch-0880f278187beb37/repodata/primary.xml.gz
2026-01-19T01:04:22+0000 DDEBUG Removing file /var/cache/dnf/kernel-livepatch-0880f278187beb37/repodata/filelists.xml.gz
2026-01-19T01:04:22+0000 DDEBUG Removing file /var/cache/dnf/kernel-livepatch-0880f278187beb37/repodata/updateinfo.xml.gz
2026-01-19T01:04:22+0000 INFO 24 files removed
2026-01-19T01:04:22+0000 DDEBUG Cleaning up.
2026-01-19T01:04:22+0000 INFO Running transaction check
2026-01-19T01:04:22+0000 SUBDEBUG
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/dnf/cli/main.py", line 176, in resolving
    base.do_transaction(display=displays)
  File "/usr/lib/python3.9/site-packages/dnf/cli/cli.py", line 246, in do_transaction
    tid = super(BaseCli, self).do_transaction(display)
  File "/usr/lib/python3.9/site-packages/dnf/base.py", line 977, in do_transaction
    self.transaction._populate_rpm_ts(self._ts)
  File "/usr/lib/python3.9/site-packages/dnf/db/group.py", line 328, in _populate_rpm_ts
    hdr = tsi.pkg._header
  File "/usr/lib/python3.9/site-packages/dnf/package.py", line 110, in _header
    return dnf.rpm._header(self.localPkg())
  File "/usr/lib/python3.9/site-packages/dnf/rpm/__init__.py", line 76, in _header
    with open(path) as package:
FileNotFoundError: [Errno 2] No such file or directory: '/var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/packages/iptables-libs-1.8.8-3.amzn2023.0.2.aarch64.rpm'
2026-01-19T01:04:22+0000 CRITICAL [Errno 2] No such file or directory: '/var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/packages/iptables-libs-1.8.8-3.amzn2023.0.2.aarch64.rpm'
2026-01-19T01:04:22+0000 DDEBUG Cleaning up.
2026-01-19T01:04:22+0000 INFO The downloaded packages were saved in cache until the next successful transaction.
2026-01-19T01:04:22+0000 INFO You can remove cached packages by executing 'yum clean packages'.
2026-01-19T01:04:22+0000 DEBUG Removed: audit-3.1.5-1.amzn2023.0.2.aarch64
2026-01-19T01:04:22+0000 INFO Complete!
2026-01-19T01:04:22+0000 DDEBUG Cleaning up.
2026-01-19T01:04:22+0000 INFO --- logging initialized ---
2026-01-19T01:04:22+0000 DDEBUG timer: config: 15 ms
2026-01-19T01:04:22+0000 INFO --- logging initialized ---
2026-01-19T01:04:22+0000 DDEBUG timer: config: 5 ms
2026-01-19T01:04:22+0000 DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, release-notification, repoclosure, repodiff, repograph, repomanage, reposync, supportinfo
2026-01-19T01:04:22+0000 DEBUG YUM version: 4.14.0
2026-01-19T01:04:22+0000 DDEBUG Command: yum -y install /usr/local/chronicle/chronicled.rpm
2026-01-19T01:04:22+0000 DDEBUG Installroot: /
2026-01-19T01:04:22+0000 DDEBUG Releasever: 2023.10.20260105
2026-01-19T01:04:22+0000 DEBUG cachedir: /var/cache/dnf
2026-01-19T01:04:22+0000 DDEBUG Base command: install
2026-01-19T01:04:22+0000 DDEBUG Extra commands: ['-y', 'install', '/usr/local/chronicle/chronicled.rpm']
2026-01-19T01:04:22+0000 DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, release-notification, repoclosure, repodiff, repograph, repomanage, reposync, supportinfo
2026-01-19T01:04:22+0000 DEBUG DNF version: 4.14.0
2026-01-19T01:04:22+0000 DDEBUG Command: dnf check-release-update
2026-01-19T01:04:22+0000 DDEBUG Installroot: /
2026-01-19T01:04:22+0000 DDEBUG Releasever: 2023.10.20260105
2026-01-19T01:04:22+0000 DEBUG cachedir: /var/cache/dnf
2026-01-19T01:04:22+0000 DDEBUG Base command: check-release-update
2026-01-19T01:04:22+0000 DDEBUG Extra commands: ['check-release-update']
2026-01-19T01:04:23+0000 DEBUG User-Agent: constructed: 'libdnf (Amazon Linux 2023; generic; Linux.aarch64)'
2026-01-19T01:04:23+0000 DEBUG repo: downloading from remote: amazonlinux
2026-01-19T01:04:23+0000 DDEBUG Cleaning up.
2026-01-19T01:04:23+0000 INFO --- logging initialized ---
2026-01-19T01:04:23+0000 DDEBUG timer: config: 3 ms
2026-01-19T01:04:23+0000 DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, release-notification, repoclosure, repodiff, repograph, repomanage, reposync, supportinfo
2026-01-19T01:04:23+0000 DEBUG DNF version: 4.14.0
2026-01-19T01:04:23+0000 DDEBUG Command: dnf clean all
2026-01-19T01:04:23+0000 DDEBUG Installroot: /
2026-01-19T01:04:23+0000 DDEBUG Releasever: 2023.10.20260105
2026-01-19T01:04:23+0000 DEBUG cachedir: /var/cache/dnf
2026-01-19T01:04:23+0000 DDEBUG Base command: clean
2026-01-19T01:04:23+0000 DDEBUG Extra commands: ['clean', 'all']
2026-01-19T01:04:23+0000 INFO Waiting for process with pid 2404 to finish.
2026-01-19T01:04:24+0000 INFO --- logging initialized ---
2026-01-19T01:04:24+0000 DDEBUG timer: config: 3 ms
2026-01-19T01:04:24+0000 DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, release-notification, repoclosure, repodiff, repograph, repomanage, reposync, supportinfo
2026-01-19T01:04:24+0000 DEBUG DNF version: 4.14.0
2026-01-19T01:04:24+0000 DDEBUG Command: dnf --debuglevel 2 updateinfo
2026-01-19T01:04:24+0000 DDEBUG Installroot: /
2026-01-19T01:04:24+0000 DDEBUG Releasever: 2023.10.20260105
2026-01-19T01:04:24+0000 DEBUG cachedir: /var/cache/dnf
2026-01-19T01:04:24+0000 DDEBUG Base command: updateinfo
2026-01-19T01:04:24+0000 DDEBUG Extra commands: ['--debuglevel', '2', 'updateinfo']
2026-01-19T01:04:24+0000 INFO Waiting for process with pid 2404 to finish.
2026-01-19T01:04:26+0000 INFO Waiting for process with pid 2404 to finish.
2026-01-19T01:04:29+0000 INFO Waiting for process with pid 2404 to finish.
2026-01-19T01:04:32+0000 INFO Waiting for process with pid 2404 to finish.
2026-01-19T01:04:35+0000 INFO Waiting for process with pid 2404 to finish.
2026-01-19T01:04:38+0000 INFO Waiting for process with pid 2404 to finish.
2026-01-19T01:04:41+0000 DEBUG amazonlinux: using metadata from Wed Dec 31 18:33:08 2025.
2026-01-19T01:04:41+0000 DEBUG repo: downloading from remote: kernel-livepatch
2026-01-19T01:04:41+0000 DEBUG kernel-livepatch: using metadata from Mon Dec 29 22:09:30 2025.
2026-01-19T01:04:41+0000 DDEBUG timer: sack setup: 18623 ms
2026-01-19T01:04:41+0000 DEBUG Completion plugin: Generating completion cache...
2026-01-19T01:04:41+0000 DEBUG User-Agent: constructed: 'libdnf (Amazon Linux 2023; generic; Linux.aarch64)'
2026-01-19T01:04:41+0000 INFO Waiting for process with pid 3309 to finish.
2026-01-19T01:04:41+0000 DEBUG repo: using cache for: amazonlinux
2026-01-19T01:04:42+0000 DEBUG amazonlinux: using metadata from Wed Dec 31 18:33:08 2025.
2026-01-19T01:04:42+0000 DEBUG repo: using cache for: kernel-livepatch
2026-01-19T01:04:42+0000 DEBUG kernel-livepatch: using metadata from Mon Dec 29 22:09:30 2025.
2026-01-19T01:04:42+0000 INFO Last metadata expiration check: 0:00:01 ago on Mon Jan 19 01:04:41 2026.
2026-01-19T01:04:42+0000 DDEBUG timer: sack setup: 18076 ms
2026-01-19T01:04:42+0000 DEBUG --> Starting dependency resolution
2026-01-19T01:04:42+0000 DEBUG ---> Package chronicled.aarch64 2.0.10766.0-1_naws will be installed
2026-01-19T01:04:42+0000 DEBUG --> Finished dependency resolution
2026-01-19T01:04:42+0000 DDEBUG timer: depsolve: 80 ms
2026-01-19T01:04:42+0000 INFO Dependencies resolved.
2026-01-19T01:04:42+0000 INFO ================================================================================
 Package         Architecture Version                   Repository         Size
================================================================================
Installing:
 chronicled      aarch64      2.0.10766.0-1_naws        @commandline       19 M

Transaction Summary
================================================================================
Install  1 Package

2026-01-19T01:04:42+0000 INFO Total size: 19 M
2026-01-19T01:04:42+0000 INFO Installed size: 53 M
2026-01-19T01:04:42+0000 INFO Downloading Packages:
2026-01-19T01:04:42+0000 INFO Running transaction check
2026-01-19T01:04:43+0000 INFO Transaction check succeeded.
2026-01-19T01:04:43+0000 INFO Running transaction test
2026-01-19T01:04:43+0000 INFO Transaction test succeeded.
2026-01-19T01:04:43+0000 DDEBUG timer: transaction test: 176 ms
2026-01-19T01:04:43+0000 INFO Running transaction
2026-01-19T01:04:43+0000 DDEBUG RPM transaction start.
2026-01-19T01:04:44+0000 INFO Waiting for process with pid 2404 to finish.
2026-01-19T01:04:45+0000 DDEBUG Cleaning up.
2026-01-19T01:04:47+0000 INFO --- logging initialized ---
2026-01-19T01:04:47+0000 DDEBUG timer: config: 28 ms
2026-01-19T01:04:47+0000 DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, release-notification, repoclosure, repodiff, repograph, repomanage, reposync, supportinfo
2026-01-19T01:04:47+0000 DEBUG DNF version: 4.14.0
2026-01-19T01:04:47+0000 DDEBUG Command: dnf check-release-update
2026-01-19T01:04:47+0000 DDEBUG Installroot: /
2026-01-19T01:04:47+0000 DDEBUG Releasever: 2023.10.20260105
2026-01-19T01:04:47+0000 DEBUG cachedir: /var/cache/dnf
2026-01-19T01:04:47+0000 DDEBUG Base command: check-release-update
2026-01-19T01:04:47+0000 DDEBUG Extra commands: ['check-release-update']
2026-01-19T01:04:47+0000 DDEBUG Cleaning up.
2026-01-19T01:04:47+0000 INFO Waiting for process with pid 2404 to finish.
2026-01-19T01:04:49+0000 DDEBUG RPM transaction over.
2026-01-19T01:04:49+0000 DDEBUG timer: verify transaction: 80 ms
2026-01-19T01:04:49+0000 DDEBUG timer: transaction: 6603 ms
2026-01-19T01:04:49+0000 DEBUG Completion plugin: Generating completion cache...
2026-01-19T01:04:50+0000 DEBUG Installed: chronicled-2.0.10766.0-1_naws.aarch64
2026-01-19T01:04:50+0000 INFO Complete!
2026-01-19T01:04:50+0000 DDEBUG Cleaning up.
2026-01-19T01:04:50+0000 DEBUG Cleaning data: packages dbcache metadata
2026-01-19T01:04:50+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux.solv
2026-01-19T01:04:50+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-filenames.solvx
2026-01-19T01:04:50+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-updateinfo.solvx
2026-01-19T01:04:50+0000 DDEBUG Removing file /var/cache/dnf/kernel-livepatch.solv
2026-01-19T01:04:50+0000 DDEBUG Removing file /var/cache/dnf/kernel-livepatch-filenames.solvx
2026-01-19T01:04:50+0000 DDEBUG Removing file /var/cache/dnf/kernel-livepatch-updateinfo.solvx
2026-01-19T01:04:50+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/mirrorlist
2026-01-19T01:04:50+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/repodata/repomd.xml
2026-01-19T01:04:50+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/repodata/primary.xml.gz
2026-01-19T01:04:50+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/repodata/filelists.xml.gz
2026-01-19T01:04:50+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/repodata/updateinfo.xml.gz
2026-01-19T01:04:50+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/repodata/comps.xml.gz
2026-01-19T01:04:50+0000 DDEBUG Removing file /var/cache/dnf/kernel-livepatch-0880f278187beb37/mirrorlist
2026-01-19T01:04:50+0000 DDEBUG Removing file /var/cache/dnf/kernel-livepatch-0880f278187beb37/repodata/repomd.xml
2026-01-19T01:04:50+0000 DDEBUG Removing file /var/cache/dnf/kernel-livepatch-0880f278187beb37/repodata/primary.xml.gz
2026-01-19T01:04:50+0000 DDEBUG Removing file /var/cache/dnf/kernel-livepatch-0880f278187beb37/repodata/filelists.xml.gz
2026-01-19T01:04:50+0000 DDEBUG Removing file /var/cache/dnf/kernel-livepatch-0880f278187beb37/repodata/updateinfo.xml.gz
2026-01-19T01:04:50+0000 INFO 17 files removed
2026-01-19T01:04:50+0000 DDEBUG Cleaning up.
2026-01-19T01:04:51+0000 INFO --- logging initialized ---
2026-01-19T01:04:51+0000 DDEBUG timer: config: 6 ms
2026-01-19T01:04:51+0000 DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, release-notification, repoclosure, repodiff, repograph, repomanage, reposync, supportinfo
2026-01-19T01:04:51+0000 DEBUG DNF version: 4.14.0
2026-01-19T01:04:51+0000 DDEBUG Command: dnf makecache --setopt skip_if_unavailable=False
2026-01-19T01:04:51+0000 DDEBUG Installroot: /
2026-01-19T01:04:51+0000 DDEBUG Releasever: 2023.10.20260105
2026-01-19T01:04:51+0000 DEBUG cachedir: /var/cache/dnf
2026-01-19T01:04:51+0000 DDEBUG Base command: makecache
2026-01-19T01:04:51+0000 DDEBUG Extra commands: ['makecache', '--setopt', 'skip_if_unavailable=False']
2026-01-19T01:04:51+0000 DEBUG Making cache files for all metadata files.
2026-01-19T01:04:51+0000 DEBUG amazonlinux: has expired and will be refreshed.
2026-01-19T01:04:51+0000 DEBUG kernel-livepatch: has expired and will be refreshed.
2026-01-19T01:04:51+0000 DEBUG repo: downloading from remote: amazonlinux
2026-01-19T01:05:06+0000 DEBUG amazonlinux: using metadata from Wed Dec 31 18:33:08 2025.
2026-01-19T01:05:06+0000 DEBUG repo: downloading from remote: kernel-livepatch
2026-01-19T01:05:07+0000 DEBUG kernel-livepatch: using metadata from Mon Dec 29 22:09:30 2025.
2026-01-19T01:05:07+0000 DDEBUG timer: sack setup: 15953 ms
2026-01-19T01:05:07+0000 DEBUG User-Agent: constructed: 'libdnf (Amazon Linux 2023; generic; Linux.aarch64)'
2026-01-19T01:05:07+0000 DEBUG Completion plugin: Generating completion cache...
2026-01-19T01:05:08+0000 INFO Metadata cache created.
2026-01-19T01:05:08+0000 DDEBUG Cleaning up.
2026-01-19T01:05:09+0000 INFO --- logging initialized ---
2026-01-19T01:05:09+0000 DDEBUG timer: config: 18 ms
2026-01-19T01:05:09+0000 DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, release-notification, repoclosure, repodiff, repograph, repomanage, reposync, supportinfo
2026-01-19T01:05:09+0000 DEBUG DNF version: 4.14.0
2026-01-19T01:05:09+0000 DDEBUG Command: dnf list installed
2026-01-19T01:05:09+0000 DDEBUG Installroot: /
2026-01-19T01:05:09+0000 DDEBUG Releasever: 2023.10.20260105
2026-01-19T01:05:09+0000 DEBUG cachedir: /var/cache/dnf
2026-01-19T01:05:09+0000 DDEBUG Base command: list
2026-01-19T01:05:09+0000 DDEBUG Extra commands: ['list', 'installed']
2026-01-19T01:05:09+0000 DEBUG User-Agent: constructed: 'libdnf (Amazon Linux 2023; generic; Linux.aarch64)'
2026-01-19T01:05:09+0000 DDEBUG timer: sack setup: 115 ms
2026-01-19T01:05:09+0000 DDEBUG Cleaning up.
2026-01-19T01:05:09+0000 INFO --- logging initialized ---
2026-01-19T01:05:09+0000 DDEBUG timer: config: 2 ms
2026-01-19T01:05:09+0000 DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, release-notification, repoclosure, repodiff, repograph, repomanage, reposync, supportinfo
2026-01-19T01:05:09+0000 DEBUG DNF version: 4.14.0
2026-01-19T01:05:09+0000 DDEBUG Command: dnf module list --enabled
2026-01-19T01:05:09+0000 DDEBUG Installroot: /
2026-01-19T01:05:09+0000 DDEBUG Releasever: 2023.10.20260105
2026-01-19T01:05:09+0000 DEBUG cachedir: /var/cache/dnf
2026-01-19T01:05:09+0000 DDEBUG Base command: module
2026-01-19T01:05:09+0000 DDEBUG Extra commands: ['module', 'list', '--enabled']
2026-01-19T01:05:10+0000 DEBUG User-Agent: constructed: 'libdnf (Amazon Linux 2023; generic; Linux.aarch64)'
2026-01-19T01:05:10+0000 DEBUG repo: using cache for: amazonlinux
2026-01-19T01:05:10+0000 DEBUG amazonlinux: using metadata from Wed Dec 31 18:33:08 2025.
2026-01-19T01:05:10+0000 DEBUG repo: using cache for: kernel-livepatch
2026-01-19T01:05:10+0000 DEBUG kernel-livepatch: using metadata from Mon Dec 29 22:09:30 2025.
2026-01-19T01:05:10+0000 INFO Last metadata expiration check: 0:00:03 ago on Mon Jan 19 01:05:07 2026.
2026-01-19T01:05:10+0000 DDEBUG timer: sack setup: 201 ms
2026-01-19T01:05:10+0000 DDEBUG Cleaning up.
2026-01-19T01:05:10+0000 INFO --- logging initialized ---
2026-01-19T01:05:10+0000 DDEBUG timer: config: 2 ms
2026-01-19T01:05:10+0000 DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, release-notification, repoclosure, repodiff, repograph, repomanage, reposync, supportinfo
2026-01-19T01:05:10+0000 DEBUG DNF version: 4.14.0
2026-01-19T01:05:10+0000 DDEBUG Command: dnf --available repoquery --queryformat %{NAME}.%{ARCH} %{EVR} %{REPONAME} %{BUILDTIME}
2026-01-19T01:05:10+0000 DDEBUG Installroot: /
2026-01-19T01:05:10+0000 DDEBUG Releasever: 2023.10.20260105
2026-01-19T01:05:10+0000 DEBUG cachedir: /var/cache/dnf
2026-01-19T01:05:10+0000 DDEBUG Base command: repoquery
2026-01-19T01:05:10+0000 DDEBUG Extra commands: ['--available', 'repoquery', '--queryformat', '%{NAME}.%{ARCH} %{EVR} %{REPONAME} %{BUILDTIME}']
2026-01-19T01:05:10+0000 DEBUG User-Agent: constructed: 'libdnf (Amazon Linux 2023; generic; Linux.aarch64)'
2026-01-19T01:05:10+0000 DEBUG repo: using cache for: amazonlinux
2026-01-19T01:05:10+0000 DEBUG amazonlinux: using metadata from Wed Dec 31 18:33:08 2025.
2026-01-19T01:05:10+0000 DEBUG repo: using cache for: kernel-livepatch
2026-01-19T01:05:10+0000 DEBUG kernel-livepatch: using metadata from Mon Dec 29 22:09:30 2025.
2026-01-19T01:05:10+0000 INFO Last metadata expiration check: 0:00:03 ago on Mon Jan 19 01:05:07 2026.
2026-01-19T01:05:10+0000 DDEBUG timer: sack setup: 188 ms
2026-01-19T01:05:11+0000 DDEBUG Cleaning up.
2026-01-19T01:05:16+0000 INFO --- logging initialized ---
2026-01-19T01:05:16+0000 DDEBUG timer: config: 2 ms
2026-01-19T01:05:16+0000 DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, release-notification, repoclosure, repodiff, repograph, repomanage, reposync, supportinfo
2026-01-19T01:05:16+0000 DEBUG DNF version: 4.14.0
2026-01-19T01:05:16+0000 DDEBUG Command: dnf updateinfo --all --info --json --quiet
2026-01-19T01:05:16+0000 DDEBUG Installroot: /
2026-01-19T01:05:16+0000 DDEBUG Releasever: 2023.10.20260105
2026-01-19T01:05:16+0000 DEBUG cachedir: /var/cache/dnf
2026-01-19T01:05:16+0000 DDEBUG Base command: updateinfo
2026-01-19T01:05:16+0000 DDEBUG Extra commands: ['updateinfo', '--all', '--info', '--json', '--quiet']
2026-01-19T01:05:16+0000 DEBUG User-Agent: constructed: 'libdnf (Amazon Linux 2023; generic; Linux.aarch64)'
2026-01-19T01:05:16+0000 DEBUG repo: using cache for: amazonlinux
2026-01-19T01:05:16+0000 DEBUG amazonlinux: using metadata from Wed Dec 31 18:33:08 2025.
2026-01-19T01:05:16+0000 DEBUG repo: using cache for: kernel-livepatch
2026-01-19T01:05:16+0000 DEBUG kernel-livepatch: using metadata from Mon Dec 29 22:09:30 2025.
2026-01-19T01:05:16+0000 INFO Last metadata expiration check: 0:00:09 ago on Mon Jan 19 01:05:07 2026.
2026-01-19T01:05:16+0000 DDEBUG timer: sack setup: 194 ms
2026-01-19T01:05:16+0000 DDEBUG Cleaning up.
2026-01-19T01:05:17+0000 INFO --- logging initialized ---
2026-01-19T01:05:17+0000 DDEBUG timer: config: 2 ms
2026-01-19T01:05:17+0000 DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, release-notification, repoclosure, repodiff, repograph, repomanage, reposync, supportinfo
2026-01-19T01:05:17+0000 DEBUG DNF version: 4.14.0
2026-01-19T01:05:17+0000 DDEBUG Command: dnf updateinfo --all --list --json --quiet
2026-01-19T01:05:17+0000 DDEBUG Installroot: /
2026-01-19T01:05:17+0000 DDEBUG Releasever: 2023.10.20260105
2026-01-19T01:05:17+0000 DEBUG cachedir: /var/cache/dnf
2026-01-19T01:05:17+0000 DDEBUG Base command: updateinfo
2026-01-19T01:05:17+0000 DDEBUG Extra commands: ['updateinfo', '--all', '--list', '--json', '--quiet']
2026-01-19T01:05:17+0000 DEBUG User-Agent: constructed: 'libdnf (Amazon Linux 2023; generic; Linux.aarch64)'
2026-01-19T01:05:17+0000 DEBUG repo: using cache for: amazonlinux
2026-01-19T01:05:17+0000 DEBUG amazonlinux: using metadata from Wed Dec 31 18:33:08 2025.
2026-01-19T01:05:17+0000 DEBUG repo: using cache for: kernel-livepatch
2026-01-19T01:05:17+0000 DEBUG kernel-livepatch: using metadata from Mon Dec 29 22:09:30 2025.
2026-01-19T01:05:17+0000 INFO Last metadata expiration check: 0:00:10 ago on Mon Jan 19 01:05:07 2026.
2026-01-19T01:05:17+0000 DDEBUG timer: sack setup: 186 ms
2026-01-19T01:05:17+0000 DDEBUG Cleaning up.
2026-01-19T01:05:20+0000 INFO --- logging initialized ---
2026-01-19T01:05:20+0000 DDEBUG timer: config: 2 ms
2026-01-19T01:05:20+0000 DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, release-notification, repoclosure, repodiff, repograph, repomanage, reposync, supportinfo
2026-01-19T01:05:20+0000 DEBUG DNF version: 4.14.0
2026-01-19T01:05:20+0000 DDEBUG Command: dnf clean all
2026-01-19T01:05:20+0000 DDEBUG Installroot: /
2026-01-19T01:05:20+0000 DDEBUG Releasever: 2023.10.20260105
2026-01-19T01:05:20+0000 DEBUG cachedir: /var/cache/dnf
2026-01-19T01:05:20+0000 DDEBUG Base command: clean
2026-01-19T01:05:20+0000 DDEBUG Extra commands: ['clean', 'all']
2026-01-19T01:05:20+0000 DEBUG Cleaning data: packages metadata dbcache
2026-01-19T01:05:20+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux.solv
2026-01-19T01:05:20+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-filenames.solvx
2026-01-19T01:05:20+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-updateinfo.solvx
2026-01-19T01:05:20+0000 DDEBUG Removing file /var/cache/dnf/kernel-livepatch.solv
2026-01-19T01:05:20+0000 DDEBUG Removing file /var/cache/dnf/kernel-livepatch-filenames.solvx
2026-01-19T01:05:20+0000 DDEBUG Removing file /var/cache/dnf/kernel-livepatch-updateinfo.solvx
2026-01-19T01:05:20+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/mirrorlist
2026-01-19T01:05:20+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/repodata/repomd.xml
2026-01-19T01:05:20+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/repodata/primary.xml.gz
2026-01-19T01:05:20+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/repodata/filelists.xml.gz
2026-01-19T01:05:20+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/repodata/updateinfo.xml.gz
2026-01-19T01:05:20+0000 DDEBUG Removing file /var/cache/dnf/amazonlinux-98c6f4d1ac0baa66/repodata/comps.xml.gz
2026-01-19T01:05:20+0000 DDEBUG Removing file /var/cache/dnf/kernel-livepatch-0880f278187beb37/mirrorlist
2026-01-19T01:05:20+0000 DDEBUG Removing file /var/cache/dnf/kernel-livepatch-0880f278187beb37/repodata/repomd.xml
2026-01-19T01:05:20+0000 DDEBUG Removing file /var/cache/dnf/kernel-livepatch-0880f278187beb37/repodata/primary.xml.gz
2026-01-19T01:05:20+0000 DDEBUG Removing file /var/cache/dnf/kernel-livepatch-0880f278187beb37/repodata/filelists.xml.gz
2026-01-19T01:05:20+0000 DDEBUG Removing file /var/cache/dnf/kernel-livepatch-0880f278187beb37/repodata/updateinfo.xml.gz
2026-01-19T01:05:20+0000 INFO 17 files removed
2026-01-19T01:05:20+0000 DDEBUG Cleaning up.
2026-01-19T01:05:21+0000 INFO --- logging initialized ---
2026-01-19T01:05:21+0000 DDEBUG timer: config: 2 ms
2026-01-19T01:05:21+0000 DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, release-notification, repoclosure, repodiff, repograph, repomanage, reposync, supportinfo
2026-01-19T01:05:21+0000 DEBUG YUM version: 4.14.0
2026-01-19T01:05:21+0000 DDEBUG Command: yum info amazon-ssm-agent
2026-01-19T01:05:21+0000 DDEBUG Installroot: /
2026-01-19T01:05:21+0000 DDEBUG Releasever: 2023.10.20260105
2026-01-19T01:05:21+0000 DEBUG cachedir: /var/cache/dnf
2026-01-19T01:05:21+0000 DDEBUG Base command: info
2026-01-19T01:05:21+0000 DDEBUG Extra commands: ['info', 'amazon-ssm-agent']
2026-01-19T01:05:21+0000 DEBUG User-Agent: constructed: 'libdnf (Amazon Linux 2023; generic; Linux.aarch64)'
2026-01-19T01:05:21+0000 DEBUG repo: downloading from remote: amazonlinux
2026-01-19T01:05:37+0000 DEBUG amazonlinux: using metadata from Wed Dec 31 18:33:08 2025.
2026-01-19T01:05:37+0000 DEBUG repo: downloading from remote: kernel-livepatch
2026-01-19T01:05:37+0000 DEBUG kernel-livepatch: using metadata from Mon Dec 29 22:09:30 2025.
2026-01-19T01:05:37+0000 DDEBUG timer: sack setup: 16305 ms
2026-01-19T01:05:37+0000 DEBUG Completion plugin: Generating completion cache...
2026-01-19T01:05:38+0000 DDEBUG Cleaning up.
2026-01-19T01:05:39+0000 INFO --- logging initialized ---
2026-01-19T01:05:39+0000 DDEBUG timer: config: 11 ms
2026-01-19T01:05:40+0000 DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, release-notification, repoclosure, repodiff, repograph, repomanage, reposync, supportinfo
2026-01-19T01:05:40+0000 DEBUG YUM version: 4.14.0
2026-01-19T01:05:40+0000 DDEBUG Command: yum info amazon-ssm-agent
2026-01-19T01:05:40+0000 DDEBUG Installroot: /
2026-01-19T01:05:40+0000 DDEBUG Releasever: 2023.10.20260105
2026-01-19T01:05:40+0000 DEBUG cachedir: /var/cache/dnf
2026-01-19T01:05:40+0000 DDEBUG Base command: info
2026-01-19T01:05:40+0000 DDEBUG Extra commands: ['info', 'amazon-ssm-agent']
2026-01-19T01:05:40+0000 DEBUG User-Agent: constructed: 'libdnf (Amazon Linux 2023; generic; Linux.aarch64)'
2026-01-19T01:05:40+0000 DEBUG repo: using cache for: amazonlinux
2026-01-19T01:05:40+0000 DEBUG amazonlinux: using metadata from Wed Dec 31 18:33:08 2025.
2026-01-19T01:05:40+0000 DEBUG repo: using cache for: kernel-livepatch
2026-01-19T01:05:40+0000 DEBUG kernel-livepatch: using metadata from Mon Dec 29 22:09:30 2025.
2026-01-19T01:05:40+0000 INFO Last metadata expiration check: 0:00:03 ago on Mon Jan 19 01:05:37 2026.
2026-01-19T01:05:40+0000 DDEBUG timer: sack setup: 258 ms
2026-01-19T01:05:40+0000 DDEBUG Cleaning up.

[pahud]

Hi @tmokmss,

Thank you for the detailed bug report, excellent root cause analysis, and for already submitting a fix in #36718! 🎉

Your analysis of the race condition is spot-on - the yum install iptables-services -y command in DEFAULT_USER_DATA_COMMANDS is vulnerable to concurrent dnf operations during Amazon Linux 2023 system initialization.

The fix in your PR using an infinite retry loop addresses the issue:

while ! yum install iptables-services -y; do echo "yum install failed, retrying in 5 seconds..."; sleep 5; done

I'm not sure if this is the best option though - an infinite loop could delay boot indefinitely if there's a real failure (network down, repo unavailable, etc.). Some alternatives to consider:

Bounded retry with failure handling:

for i in {1..5}; do yum install iptables-services -y && break || { echo "Attempt $i failed"; sleep 10; }; done

Use dnf's built-in retry mechanism (cleaner for AL2023):

dnf install -y iptables-services --setopt=retries=5 --setopt=timeout=30

Wait for package manager lock first:

while fuser /var/lib/rpm/.rpm.lock >/dev/null 2>&1; do sleep 2; done
yum install iptables-services -y

That said, for a NAT instance, a non-functional NAT is arguably worse than a slow boot, so the infinite loop approach may be acceptable here.

As this could technically happen but I can't easily reproduce, I am making it a p2 for now and we welcome 👍 to help us prioritize.

[tmokmss]

Hi thanks @pahud! Yes, I agree that the infinite loop approach is acceptable here. The most ideal way can be to rollback CFn deployment if the EC2 userdata failed, but it requires more complex setup using CFn custom resources. I believe all we can reasonably do is to keep retrying until it succeeds.