After enabling the Longhorn V2 Data Engine in Harvester, some nodes may report kernel workqueue lockups and become unstable. In severe cases, rke2-server may terminate, Longhorn may stop making progress, and the node may require recovery.

This issue is most likely to occur when SPDK is pinned to dedicated CPUs, but the Linux kernel is still allowed to run IRQ handlers or unbound workqueue workers on those same CPUs.

Affected Scenario​

This issue can affect Harvester clusters that meet all of the following conditions:

• The Longhorn V2 Data Engine is enabled.
• The Longhorn V2 instance-manager pod runs SPDK (spdk_tgt), which busy-polls on the CPUs selected by the Longhorn V2 data engine CPU mask.
• Host IRQ handling or unbound kernel workqueues are still allowed to run on the same busy CPUs used by spdk_tgt.

The default Longhorn V2 CPU mask is 0x3, which maps to CPUs 0 and 1. However, the issue is not limited to CPUs 0 and 1; it can occur with any CPU mask if kernel IRQs or unbound workqueues still run on the CPUs occupied by spdk_tgt.

The issue was observed on bare-metal Harvester nodes with Longhorn V2 enabled. It is not always reproducible in every environment.

Symptoms​

Affected nodes may show one or more of the following symptoms:

• Kernel logs repeatedly contain BUG: workqueue lockup.
• The lockup is reported on a CPU that is part of the Longhorn V2 data engine CPU mask.
• rke2-server becomes unstable or terminates without a clear user-visible reason.
• Longhorn volumes, engines, replicas, or instance managers stop progressing.
• User switching commands, such as sudo -i, may become very slow before the root shell appears.
• The node becomes unstable or unhealthy.

Example kernel message:

Jul 27 07:27:55 hp-114-tink-system kernel: BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 2117s!
Jul 27 07:27:55 hp-114-tink-system kernel: Showing busy workqueues and worker pools:
Jul 27 07:27:55 hp-114-tink-system kernel: workqueue events: flags=0x0
Jul 27 07:27:55 hp-114-tink-system kernel:   pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=7 refcnt=8
Jul 27 07:27:55 hp-114-tink-system kernel:     in-flight: 2935011:output_poll_execute ,32008:drm_fb_helper_damage_work drm_fb_helper_damage_work
Jul 27 07:27:55 hp-114-tink-system kernel:     pending: vmstat_shepherd, switchdev_deferred_process_work, rht_deferred_worker, key_garbage_collector
Jul 27 07:27:55 hp-114-tink-system kernel: workqueue events_unbound: flags=0x2
Jul 27 07:27:55 hp-114-tink-system kernel:   pwq 98: cpus=0-23 node=1 flags=0x4 nice=0 active=4 refcnt=8
Jul 27 07:27:55 hp-114-tink-system kernel: workqueue events_power_efficient: flags=0x80
Jul 27 07:27:55 hp-114-tink-system kernel:   pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=4 refcnt=5
Jul 27 07:27:55 hp-114-tink-system kernel:     pending: neigh_managed_work, neigh_periodic_work, gc_worker [nf_conntrack], check_lifetime
Jul 27 07:27:55 hp-114-tink-system kernel: pool 2: cpus=0 node=0 flags=0x0 nice=0 hung=2117s workers=5 idle: 1025200 1541794
Jul 27 07:27:55 hp-114-tink-system kernel: task:kworker/0:0     state:R  running task     stack:0     pid:32008 tgid:32008 ppid:2      flags:0x00004000
Jul 27 07:27:55 hp-114-tink-system kernel: Workqueue: events drm_fb_helper_damage_work
Jul 27 07:28:12 hp-114-tink-system rke2[2909320]: time="2026-07-27T07:28:12Z" level=warning msg="Proxy error: write failed: write tcp 127.0.0.1:9345->127.0.0.1:46562: write: connection reset by peer"

Root Cause​

Longhorn V2 uses SPDK. SPDK reactor threads are designed to busy-poll on the CPUs selected by the Longhorn V2 data engine CPU mask. This is expected for high-performance storage I/O.

The problem occurs when normal host kernel work is also allowed to run on those same CPUs. For example:

• NIC or storage IRQs may still target the SPDK CPUs.
• Unbound kernel workqueues may still include the SPDK CPUs in /sys/devices/virtual/workqueue/cpumask.
• Per-workqueue CPU masks may still allow existing workers to run on the SPDK CPUs.

When SPDK fully occupies those CPUs, kernel workers can be delayed long enough for the kernel workqueue lockup detector to fire. Short or occasional workqueue lockup reports do not always mean the node is unrecoverable. The operational problem is when the delayed kernel work keeps accumulating and cascades into networking, RKE2, kubelet, and Longhorn instability.

For example, if the Longhorn V2 CPU mask is the default value 0x3, SPDK uses CPUs 0 and 1. IRQs and kernel workqueues should be moved away from CPUs 0 and 1.

Confirming the Issue​

Run the following checks on each affected Harvester node.

1. Check Kernel Logs​

journalctl -k --since "24 hours ago" | grep -E "BUG: workqueue lockup|soft lockup"

Look for the CPU reported in messages such as pool cpus=0, pool cpus=1, kworker/0, or kworker/1.

2. Check the Longhorn V2 CPU Mask​

kubectl -n longhorn-system get settings.longhorn.io data-engine-cpu-mask -o jsonpath='{.value}{"\n"}'

The value may be data-engine-specific. For example:

{"v2":"0x3"}

Some Longhorn versions support CPU-list input for this setting. If the value is already a CPU list, use that list directly in the checks below.

Convert the mask to CPU IDs:

python3 - <<'PY'
mask = int("0x3", 0)
print(",".join(str(cpu) for cpu in range(mask.bit_length()) if mask & (1 << cpu)))
PY

For 0x3, the output is:

0,1

3. Check SPDK Placement​

SPDK_PIDS=$(pgrep -f '[s]pdk_tgt' || true)

for pid in ${SPDK_PIDS}; do
  ps -T -p "${pid}" -o pid,tid,psr,stat,pcpu,comm,args
done

This command matches the full command line because the SPDK process or its threads may appear as reactor_<cpu> in the kernel process name. In that case, pgrep -x spdk_tgt can return empty even when spdk_tgt is running.

Example output for the default CPU mask 0x3:

    PID     TID PSR STAT %CPU COMMAND         COMMAND
2913161 2913161   0 RLl  99.9 reactor_0       spdk_tgt -L all --mem-size 2048 -m 0x3
2913161 2913166  21 SLl   0.0 dpdk-intr       spdk_tgt -L all --mem-size 2048 -m 0x3
2913161 2913220   1 RLl  99.9 reactor_1       spdk_tgt -L all --mem-size 2048 -m 0x3

The PSR column shows the CPU that each thread is running on. In the example above, reactor_0 is running on CPU 0 and reactor_1 is running on CPU 1, so the SPDK CPU list is 0,1.

To check whether running or blocked kernel workers are currently on the SPDK CPUs, use the following more targeted command. Replace SPDK_CPUS with the comma-separated CPU IDs from the reactor_* rows.

SPDK_CPUS="0,1"

ps -eLo pid,tid,psr,stat,pcpu,wchan:30,comm,args | \
  awk -v cpus="${SPDK_CPUS}" '
    BEGIN {
      printf "%7s %7s %3s %-5s %5s %-30s %-16s %s\n", "PID", "TID", "PSR", "STAT", "%CPU", "WCHAN", "COMMAND", "ARGS"
      split(cpus, cpu_list, ",")
      for (i in cpu_list) {
        spdk_cpu[cpu_list[i]] = 1
      }
    }
    NR > 1 && spdk_cpu[$3] && $4 ~ /[RD]/ && ($7 ~ /^kworker\// || $7 ~ /^ksoftirqd\//) {
      print
    }
  '

On nodes that are not affected, this command normally prints only the header line. If the header line is removed from the command, empty output is expected. This means there are no running or uninterruptible kworker/* or ksoftirqd/* threads on the SPDK CPUs at that moment.

Example output showing kernel work on an SPDK CPU:

    PID     TID PSR STAT   %CPU WCHAN                          COMMAND          ARGS
     17      17   0 R     0.0 -                              ksoftirqd/0     [ksoftirqd/0]
  32008   32008   0 R     0.0 -                              kworker/0:0+eve [kworker/0:0+events]
2935011 2935011   0 D     0.0 mgag200_ddc_algo_bit_data_pre_ kworker/0:2+eve [kworker/0:2+events]
4012518 4012518   0 R     0.0 -                              kworker/0:3+mm_ [kworker/0:3+mm_percpu_wq]

In this example, the PSR value is 0, and the process names are ksoftirqd/0 and kworker/0:*. This means kernel softirq and workqueue threads are running or blocked on CPU 0.

If the SPDK reactor threads are running on the CPUs reported in the workqueue lockup, or if the targeted command shows kworker/* or ksoftirqd/* activity on the SPDK CPUs, continue with the IRQ and workqueue checks.

4. Check IRQ Affinity​

The following example checks CPUs 0 and 1 because the reactor_* rows above are running on those CPUs. Replace SPDK_CPUS with the CPU IDs used in your environment.

SPDK_CPUS="0,1"

cpu_list_overlaps() {
  python3 - "$1" "$2" <<'PY'
import sys

target = {int(cpu) for cpu in sys.argv[1].split(",") if cpu}
seen = set()

for part in sys.argv[2].split(","):
    part = part.strip()
    if not part:
        continue
    if "-" in part:
        start, end = map(int, part.split("-", 1))
        seen.update(range(start, end + 1))
    else:
        seen.add(int(part))

sys.exit(0 if target & seen else 1)
PY
}

for irqdir in /proc/irq/[0-9]*; do
  irq=${irqdir##*/}
  eff=$(cat "${irqdir}/effective_affinity_list" 2>/dev/null || true)
  conf=$(cat "${irqdir}/smp_affinity_list" 2>/dev/null || true)

  if cpu_list_overlaps "${SPDK_CPUS}" "${eff}"; then
    echo "IRQ=${irq} configured=${conf} effective=${eff}"
    grep -w "^ *${irq}:" /proc/interrupts 2>/dev/null || true
  fi
done

Example output:

IRQ=108 configured=0-5,12-17 effective=0
 108:   25762242          0 ... IR-PCI-MSIX-0000:04:00.0    3-edge      netboot-TxRx-3
IRQ=109 configured=0-5,12-17 effective=1
 109:          0   15555565 ... IR-PCI-MSIX-0000:04:00.0    4-edge      netboot-TxRx-4
IRQ=120 configured=0-5,12-17 effective=0
 120:   10898692          0 ... IR-PCI-MSIX-0000:04:00.0   15-edge      netboot-TxRx-15
IRQ=121 configured=0-5,12-17 effective=1
 121:          0    8135324 ... IR-PCI-MSIX-0000:04:00.0   16-edge      netboot-TxRx-16
IRQ=92 configured=0 effective=0
  92:    2242087          0 ... IR-PCI-MSIX-0000:08:00.0   13-edge      nvme0q13
IRQ=93 configured=1 effective=1
  93:          0     448454 ... IR-PCI-MSIX-0000:08:00.0   14-edge      nvme0q14

The effective value shows where the IRQ is actually running. In this example, NIC queues and NVMe queues are effectively landing on CPUs 0 and 1, which are the default SPDK reactor CPUs for mask 0x3. If device IRQs, especially high-traffic NIC or storage IRQs, are effectively landing on SPDK CPUs, the node is at risk.

5. Check Kernel Workqueue Masks​

cat /sys/devices/virtual/workqueue/cpumask

for f in /sys/devices/virtual/workqueue/*/cpumask; do
  echo "${f}: $(cat "${f}")"
done

Example output:

/sys/devices/virtual/workqueue/blkcg_punt_bio/cpumask: ffffff
/sys/devices/virtual/workqueue/ib-comp-unb-wq/cpumask: ffffff
/sys/devices/virtual/workqueue/iscsi_conn_cleanup/cpumask: ffffff
/sys/devices/virtual/workqueue/nvme-auth-wq/cpumask: ffffff
/sys/devices/virtual/workqueue/nvme-delete-wq/cpumask: ffffff
/sys/devices/virtual/workqueue/nvme-reset-wq/cpumask: ffffff
/sys/devices/virtual/workqueue/nvme-wq/cpumask: ffffff
/sys/devices/virtual/workqueue/scsi_tmf_0/cpumask: ffffff
/sys/devices/virtual/workqueue/writeback/cpumask: ffffff

On a 24-CPU node, ffffff means the workqueue can run on CPUs 0-23. If SPDK uses the default CPU mask 0x3, CPUs 0 and 1 are included in this workqueue mask. If the global or per-workqueue masks include the SPDK CPUs, unbound kernel work may still run on the SPDK CPUs.

Preferred Risk-Reduction Setting​

Longhorn added the data-engine-cpu-isolation-enabled setting to reduce the chance of this issue. When enabled for the V2 Data Engine, the Longhorn V2 instance-manager:

• Persists the SPDK CPU mask under /var/lib/longhorn/instance-manager/v2/spdk_cpu_mask on the host.
• Programs /proc/irq/*/smp_affinity to the inverse of the SPDK CPU mask.
• Writes the same inverse mask to /sys/devices/virtual/workqueue/cpumask.
• Updates per-workqueue CPU masks when possible.
• Reconciles stale affinity state on the next instance-manager restart if the setting is later disabled.

This setting steers IRQs and unbound workqueues away from SPDK CPUs. In some environments, this may stop the workqueue lockup messages entirely. In others, it may only reduce how often they occur because CPU-bound or per-CPU kernel workers such as kworker/0:* are tied to a specific CPU and cannot be moved by the unbound workqueue CPU mask.

Use this setting when the Longhorn version bundled with Harvester includes it. If the Longhorn setting exists but Harvester does not expose it in the UI, you can still configure it through the Longhorn setting resource. For Harvester versions earlier than v1.9.0, the bundled Longhorn version does not include the setting, so use the manual workaround in the next section.

If the setting exists in your Longhorn version, you can check it with:

kubectl -n longhorn-system get settings.longhorn.io data-engine-cpu-isolation-enabled

Enable it for the V2 Data Engine:

kubectl -n longhorn-system patch settings.longhorn.io data-engine-cpu-isolation-enabled \
  --type=merge \
  -p '{"value":"{\"v2\":\"true\"}"}'

Then wait for the V2 instance-manager pods to be recreated after all V2 volumes are detached.

Verify the instance-manager log:

kubectl -n longhorn-system logs <v2-instance-manager-pod> | \
  grep -E "Setting IRQ affinity|Setting workqueue cpumask|Applied IRQ affinity|Applied global workqueue cpumask"

Example expected messages:

Setting IRQ affinity to exclude SPDK CPUs
Applied IRQ affinity mask
Setting workqueue cpumask to exclude SPDK CPUs
Applied global workqueue cpumask
Applied per-workqueue cpumask

Manual Workaround for Versions Without the Longhorn Setting​

Use this workaround for Harvester versions earlier than v1.9.0, where the bundled Longhorn version does not include data-engine-cpu-isolation-enabled. For later versions, first check whether the Longhorn setting exists and prefer the setting-based risk-reduction path when possible.

The goal is to move IRQs and unbound workqueues away from the SPDK CPUs. For example, if SPDK uses the default CPUs 0 and 1, IRQs and unbound workqueues should use CPU 2 through the last online CPU.

This workaround is a risk-reduction step, not a guaranteed fix for every workqueue lockup. CPU-bound or per-CPU kernel workers can still run on the SPDK CPUs because they are tied to those CPUs by the kernel. The expected result is that the node remains stable and any remaining workqueue stalls recover quickly instead of hanging for a long time.

Repeat this workaround whenever the Longhorn V2 CPU mask changes. If CPU allocation is managed dynamically, the IRQ and workqueue masks must be recalculated after each placement change.

1. Identify the Non-SPDK CPUs and Mask​

The default Longhorn V2 CPU mask is 0x3, so SPDK uses CPUs 0 and 1.

On the example node, all online CPUs are 0-23:

cat /sys/devices/system/cpu/online

0-23

Use the following values for this default example:

The Linux affinity mask is a CPU bitmap written in hexadecimal. CPU 0 is bit 0, CPU 1 is bit 1, and so on. On a 24-CPU node, all CPUs enabled is ffffff. Excluding CPUs 0 and 1 clears the lowest two bits, so the mask becomes fffffc. Leading zeros are optional, so fffffc and 00fffffc are equivalent.

Do not copy these values blindly if your Longhorn V2 CPU mask or online CPU list is different. The non-SPDK CPU list must be all online CPUs except the CPUs used by the SPDK reactor threads.

2. Apply IRQ Affinity at Runtime​

Set NON_SPDK_AFFINITY_MASK to the Linux affinity mask from the previous step.

NON_SPDK_AFFINITY_MASK=fffffc

echo "${NON_SPDK_AFFINITY_MASK}" > /proc/irq/default_smp_affinity

for f in /proc/irq/[0-9]*/smp_affinity; do
  echo "${NON_SPDK_AFFINITY_MASK}" > "${f}" 2>/dev/null || true
done

Some IRQs may reject affinity updates because they are managed by the kernel. This is expected. Always verify effective_affinity_list after applying the change.

Changing IRQ affinity alone may not make existing workqueue lockup messages disappear. Kernel workqueues can still run on the SPDK CPUs until the workqueue CPU masks are updated, and workers that are already stuck may continue to be reported by the kernel. Apply the workqueue affinity change as well, and then verify whether new lockup messages stop appearing.

These runtime IRQ affinity changes are not persistent across reboot. Use the persistence step later in this section if the runtime change mitigates the issue.

3. Apply Workqueue Affinity at Runtime​

NON_SPDK_AFFINITY_MASK=fffffc

echo "${NON_SPDK_AFFINITY_MASK}" > /sys/devices/virtual/workqueue/cpumask

for f in /sys/devices/virtual/workqueue/*/cpumask; do
  echo "${NON_SPDK_AFFINITY_MASK}" > "${f}" 2>/dev/null || true
done

These runtime workqueue affinity changes are not persistent across reboot. Use the persistence step later in this section if the runtime change mitigates the issue.

4. Restart the V2 Instance-Manager Pod​

After changing both IRQ and workqueue affinity, restart the Longhorn V2 instance-manager pod on the affected node. This is required because the existing spdk_tgt process can keep the SPDK reactor threads on the busy CPUs, and already-stuck CPU-bound workers may continue to be reported by the kernel until spdk_tgt is recreated.

Check the Longhorn V2 volumes first:

kubectl -n longhorn-system get volumes.longhorn.io \
  -o custom-columns=NAME:.metadata.name,DATAENGINE:.spec.dataEngine,STATE:.status.state,NODE:.status.currentNodeID

Only continue after all v2 volumes whose NODE is the affected node are detached.

Find the V2 instance-manager pod on the affected node:

AFFECTED_NODE=hp-114-tink-system

kubectl -n longhorn-system get pods \
  -l longhorn.io/component=instance-manager,longhorn.io/data-engine=v2 \
  --field-selector spec.nodeName="${AFFECTED_NODE}" \
  -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName

Restart the V2 instance-manager pod:

V2_IM_POD=<v2-instance-manager-pod-name>

kubectl -n longhorn-system delete pod "${V2_IM_POD}"

5. Make IRQ and Workqueue Affinity Persistent​

The previous IRQ and workqueue affinity commands only change runtime procfs and sysfs state, so they are lost after reboot. If those runtime changes reduce or stop new lockup messages, use a Harvester CloudInit resource to reapply both IRQ and workqueue affinity after reboot. This follows the same model as the Longhorn CPU isolation setting, which updates host IRQ and workqueue affinity at runtime.

The following manifest is a Harvester CloudInit resource, not a file that you manually place on each node. Save it on a machine with kubectl access to the Harvester cluster and apply it with kubectl apply -f <file-name>.yaml. The Harvester controller writes the content to /oem/99_longhorn_v2_cpu_affinity.yaml on each node matched by matchSelector.

You can add this resource after the cluster is already installed. The file is synchronized to the matched nodes after the resource is applied, but the cloud-init commands take effect only after those nodes are rebooted. After reboot, the commands are applied again on each boot.

Example CloudInit resource for the default SPDK CPUs 0 and 1 on a 24-CPU node:

apiVersion: node.harvesterhci.io/v1beta1
kind: CloudInit
metadata:
  name: longhorn-v2-cpu-affinity
spec:
  matchSelector:
    kubernetes.io/hostname: "hp-114-tink-system"
  filename: 99_longhorn_v2_cpu_affinity.yaml
  contents: |
    stages:
      network:
        - commands:
            - echo fffffc > /proc/irq/default_smp_affinity
            - for f in /proc/irq/[0-9]*/smp_affinity; do echo fffffc > "${f}" 2>/dev/null || true; done
            - echo fffffc > /sys/devices/virtual/workqueue/cpumask
            - for f in /sys/devices/virtual/workqueue/*/cpumask; do echo fffffc > "${f}" 2>/dev/null || true; done

Adjust the affinity mask and matchSelector for your environment. To target a different affected node, replace hp-114-tink-system with that node's kubernetes.io/hostname label value. Do not use matchSelector: {} unless every node should receive this workaround. This workaround is not dynamically reconciled. If the Longhorn V2 CPU mask changes, update the CloudInit resource and reboot the affected nodes.

After configuring the CloudInit resource, reboot the affected nodes for the commands to take effect. To apply the workaround immediately before reboot, run the runtime commands in the previous steps.

If Lockups Still Appear​

The IRQ and workqueue affinity changes reduce the chance of a workqueue lockup, but they may not eliminate it in every environment. The workqueue CPU mask mainly controls unbound workqueues. Bound or per-CPU workqueues can still run on their associated CPU, so kernel workers such as kworker/20:* may still appear on an SPDK CPU even after unbound workqueues are moved away.

If the node still reports occasional workqueue lockups after the IRQ and workqueue affinity mitigation is applied, use the following additional mitigations.

1. Change the Longhorn V2 Disk Driver to auto​

If the affected Longhorn V2 disk is using the aio disk driver, consider changing the requested disk driver to auto. For an NVMe disk, verify after reprovisioning that Longhorn reports the actual disk driver as nvme in the Longhorn Node custom resource. In recent validation, after applying IRQ affinity and confirming that the actual disk driver was nvme, no new workqueue lockup messages were observed during the test window.

Before changing the disk driver, remove the affected disk from Harvester. Follow the Remove Disks guide, and make sure the disk no longer contains active Longhorn replicas or backing images.

Find the BlockDevice resource for the disk:

kubectl -n longhorn-system get blockdevices.harvesterhci.io \
  -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,PROVISION:.spec.provision,PHASE:.status.provisionPhase,STATE:.status.state,ENGINE:.spec.provisioner.longhorn.engineVersion,DRIVER:.spec.provisioner.longhorn.diskDriver

After the disk is removed and the BlockDevice is no longer provisioned, patch the disk driver:

BLOCKDEVICE=<blockdevice-name>

kubectl -n longhorn-system patch blockdevice.harvesterhci.io "${BLOCKDEVICE}" \
  --type=merge \
  -p '{"spec":{"provisioner":{"longhorn":{"engineVersion":"LonghornV2","diskDriver":"auto"}}}}'

Provision the disk again from the Harvester UI. Harvester will add the disk back to Longhorn V2. Then verify the requested driver in the BlockDevice resource and the actual driver in the Longhorn Node resource.

Verify the BlockDevice after provisioning:

kubectl -n longhorn-system get blockdevice.harvesterhci.io "${BLOCKDEVICE}" -o yaml

The matching BlockDevice should show:

spec:
  provision: true
  provisioner:
    longhorn:
      engineVersion: LonghornV2
      diskDriver: auto
status:
  provisionPhase: Provisioned
  state: Active

Also check the Longhorn Node custom resource to confirm the actual disk driver selected by Longhorn:

kubectl -n longhorn-system get nodes.longhorn.io \
  -o go-template='{{printf "NODE\tDISK\tTYPE\tPATH\tSPEC_DRIVER\tSTATUS_DRIVER\n"}}{{range .items}}{{ $node := . }}{{range $diskName, $disk := .spec.disks}}{{ $status := index $node.status.diskStatus $diskName }}{{printf "%s\t%s\t%s\t%s\t%s\t%s\n" $node.metadata.name $diskName $disk.diskType $disk.path $disk.diskDriver $status.diskDriver}}{{end}}{{end}}'

For the affected disk, SPEC_DRIVER should be auto, and STATUS_DRIVER should be nvme:

NODE                  DISK                                  TYPE    PATH          SPEC_DRIVER   STATUS_DRIVER
hp-161-tink-system    32f43222-1eb1-4ab6-9e65-c4f8ddad700d  block   0000:08:00.0  auto          nvme

The spec.disks.<blockdevice-name>.diskDriver value is the requested driver mode from Harvester. The status.diskStatus.<blockdevice-name>.diskDriver value is the driver Longhorn actually uses. For an NVMe disk, the status value should be nvme.

2. Enable Longhorn V2 Interrupt Mode​

If the lockup still occurs after changing the disk driver and applying IRQ affinity, consider enabling Longhorn V2 interrupt mode.

Check the setting:

kubectl -n longhorn-system get settings.longhorn.io data-engine-interrupt-mode-enabled

Enable interrupt mode for the V2 Data Engine:

kubectl -n longhorn-system patch settings.longhorn.io data-engine-interrupt-mode-enabled \
  --type=merge \
  -p '{"value":"{\"v2\":\"true\"}"}'

Wait for the V2 instance-manager pods to be recreated after all V2 volumes are detached.

Verification​

After applying the mitigation, verify the items that directly confirm the node recovered and the settings are still applied.

1. Kernel logs no longer report new workqueue lockups, or any remaining reports are short and do not keep increasing for a long time.

   journalctl -k --since "30 minutes ago" | grep -E "BUG: workqueue lockup|soft lockup" || true
   

   Copy

   If occasional messages still appear, compare the stuck for <seconds>s value over time. The mitigation is still useful if the stuck time stops growing, the node remains responsive, and RKE2, kubelet, and Longhorn keep making progress.

2. The Longhorn V2 instance-manager pod is running on the affected node.

   AFFECTED_NODE=hp-114-tink-system
   
   kubectl -n longhorn-system get pods \
     -l longhorn.io/component=instance-manager,longhorn.io/data-engine=v2 \
     --field-selector spec.nodeName="${AFFECTED_NODE}"
   

   Copy

3. If you configured the persistent workaround and rebooted the node, the IRQ and workqueue masks are still applied.

   cat /proc/irq/default_smp_affinity
   cat /sys/devices/virtual/workqueue/cpumask
   

   Copy

References​

• Harvester bug: harvester/harvester#11030
• Longhorn bug: longhorn/longhorn#13417
• Longhorn setting reference: Data Engine CPU Mask
• Longhorn setting reference: Data Engine Interrupt Mode Enabled
• Longhorn documentation: Interrupt Mode Support
• Linux kernel documentation: IRQs
• Linux kernel documentation: Workqueue

This article provides information and mitigation steps for the following vulnerabilities in Harvester:

• CVE-2026-53359

The mitigation steps involves disabling the nested virtualization feature of the KVM kernel module on your Harvester hosts.

On your Harvester hosts, use the following commands to confirm that the KVM kernel module is loaded with nested virtualization enabled:

lsmod | grep -iE "kvm_amd|kvm_intel"

sudo grep -H '' /sys/module/{kvm_amd|kvm_intel}/parameters/* 2>&1 |grep nested

If the above commands return module and parameter information about the KVM kernel module, then your host is affected by this vulnerability.

Deploy the following CloudInit configuration to disable the nested virtualization feature of the KVM kernel module on all your Harvester hosts:

apiVersion: node.harvesterhci.io/v1beta1
kind: CloudInit
metadata:
  name: disabled-nested-virtualization
spec:
  matchSelector:
    harvesterhci.io/managed: "true"
  filename: 99-disabled-nested-virtualization
  contents: |
    stages:
      initramfs:
      - name: "disable nested virtualization in kvm modules"
        files:
        - path: "/etc/modprobe.d/99-disabled-nested-virtualization.conf"
          content: |
            options kvm_amd nested=0
            options kvm_intel nested=0

Once the configuration is applied, reboot your Harvester hosts for the changes to take effect.

Once you have upgraded to a fixed version of Harvester, you can re-enable the nested virtualization feature by deleting the CloudInit configuration and rebooting your Harvester hosts:

kubectl delete cloudinit disabled-nested-virtualization

References​

• https://support.scc.suse.com/s/kb/Security-Vulnerability-x86-64-kvm-Guest-to-Host-escape-aka-Januscape-CVE-2026-53359
• https://www.suse.com/security/cve/CVE-2026-53359.html

When running production workloads on virtualized infrastructure like Harvester, memory management is critical. In Harvester versions prior to v1.4.0, certain workloads experienced sudden Virtual Machine (VM) terminations due to the host Linux operating system triggering Out-Of-Memory (OOM) kills.

What is KubeVirt? Harvester uses KubeVirt as its core virtualization engine. KubeVirt is an open-source technology that allows Kubernetes to run and manage traditional Virtual Machines inside standard containers, translating VM specifications directly into Pod configurations.

This article explores why these OOM events occur in a Kubernetes-native virtualization environment and how Harvester provides granular tools to eliminate them.

Anatomy of a VM OOM Event​

When a VM is terminated due to insufficient memory at the host level, the Linux kernel logs specific keywords that help pinpoint the fault. In Harvester, these logs generally fall into two distinct categories depending on which process triggered the exhaustion.

Example 1: virt-launcher invoked oom-killer

The virt-launcher process runs inside the dedicated Kubernetes Pod backing the VM. If this component or its direct sub-processes run out of the memory allocated to their cgroup, the kernel triggers a memory cgroup (memcg) OOM event.

Feb 03 19:57:08 ** kernel: virt-launcher invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=986
Feb 03 19:57:08 ** kernel: CPU: 40 PID: 40785 Comm: virt-launcher Tainted: G          I    X    5.14.21-150400.24.60-default #1 SLE15-SP4 9096397fa6646928cc6d185ba417f2af65b536f1
...

Feb 03 19:57:08 ** kernel: memory: usage 17024340kB, limit 17024340kB, failcnt 1243
Feb 03 19:57:08 ** kernel: memory+swap: usage 17024340kB, limit 9007199254740988kB, failcnt 0                                                                 
Feb 03 19:57:08 ** kernel: kmem: usage 143556kB, limit 9007199254740988kB, failcnt 0
Feb 03 19:57:08 ** kernel: Memory cgroup stats for /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968a06fb_9ab9_4819_8caf_0392ddff3d9b.slice:
...
Feb 03 19:57:08 ** kernel: Tasks state (memory values in pages):
Feb 03 19:57:08 ** kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Feb 03 19:57:08 ** kernel: [  38886]     0 38886      243        1    28672        0          -998 pause
Feb 03 19:57:08 ** kernel: [  38917]     0 38917   310400     6921   192512        0           986 virt-launcher-m
Feb 03 19:57:08 ** kernel: [  38934]     0 38934  1200940    25126   954368        0           986 virt-launcher
Feb 03 19:57:08 ** kernel: [  38951]     0 38951   386525     8247   466944        0           986 libvirtd
Feb 03 19:57:08 ** kernel: [  38952]     0 38952    33619     3940   290816        0           986 virtlogd
Feb 03 19:57:08 ** kernel: [  39079]   107 39079  4457263  4201766 34439168        0           986 qemu-system-x86
Feb 03 19:57:08 ** kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-0f32894de86edf3d3832702af794874ef8d400b4969acdea4976b12040756e0d.scope,mems_allowed=0-1,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968a06fb_9ab9_4819_8caf_0392ddff3d9b.slice,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968a06fb_9ab9_4819_8caf_0392ddff3d9b.slice/cri-containerd-0f32894de86edf3d3832702af794874ef8d400b4969acdea4976b12040756e0d.scope,task=qemu-system-x86,pid=39079,uid=107

Example 2: CPU X/KVM invoked oom-killer

This occurs when a vCPU execution thread inside qemu-system-x86_64 attempts a memory operation that pushes the entire container beyond its Kubernetes memory limit.

[Thu May  9 14:52:38 2024] CPU 11/KVM invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=830
[Thu May  9 14:52:38 2024] CPU: 60 PID: 70888 Comm: CPU 11/KVM Not tainted 5.3.18-150300.59.101-default #1 SLE15-SP3
...
[Thu May  9 14:52:38 2024] memory: usage 67579904kB, limit 67579904kB, failcnt 67391
[Thu May  9 14:52:38 2024] memory+swap: usage 0kB, limit 9007199254740988kB, failcnt 0
[Thu May  9 14:52:38 2024] kmem: usage 633636kB, limit 9007199254740988kB, failcnt 0
...
[Thu May  9 14:52:38 2024] Tasks state (memory values in pages):
[Thu May  9 14:52:38 2024] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[Thu May  9 14:52:38 2024] [  70675]     0 70675      243        1    28672        0          -998 pause
[Thu May  9 14:52:38 2024] [  70728]     0 70728   310400     5467   188416        0           830 virt-launcher-m
[Thu May  9 14:52:38 2024] [  70746]     0 70746  1242373    25104  1073152        0           830 virt-launcher
[Thu May  9 14:52:38 2024] [  70762]     0 70762   455279    14110   770048        0           830 libvirtd
[Thu May  9 14:52:38 2024] [  70763]     0 70763    37704     3916   339968        0           830 virtlogd
[Thu May  9 14:52:38 2024] [  70870]   107 70870 18302464 16718510 135278592        0           830 qemu-system-x86
[Thu May  9 14:52:38 2024] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-100093783c22a3ae1a42e21dd887b7c26eef52d56ba44c7273ef54507b6efe7c.scope,mems_allowed=0-3,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef91e487_dec5_4613_800b_eb23e1a1617d.slice,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef91e487_dec5_4613_800b_eb23e1a1617d.slice/cri-containerd-100093783c22a3ae1a42e21dd887b7c26eef52d56ba44c7273ef54507b6efe7c.scope,task=qemu-system-x86,pid=70870,uid=107
[Thu May  9 14:52:38 2024] Memory cgroup out of memory: Killed process 70870 (qemu-system-x86) total-vm:73209856kB, anon-rss:66852088kB, file-rss:21948kB, shmem-rss:4kB
[Thu May  9 14:52:38 2024] oom_reaper: reaped process 70870 (qemu-system-x86), now anon-rss:0kB, file-rss:132kB, shmem-rss:4kB

Root Cause: Native Hypervisors vs. Harvester Architecture​

Traditional Linux Host (e.g., Virtual Machine Manager)​

On a standard Linux host, a VM managed via QEMU/KVM runs inside a systemd machine.slice. The hypervisor process (qemu-system-x86_64) has access to the host's wider pool of resources, managed loosely unless strict cgroup limits are manually added.

$ systemd-cgls
Control group /:
-.slice
├─1173 bpfilter_umh
├─system.slice

└─machine.slice
  └─machine-qemu\x2d1\x2dharv41.scope 
    └─8632 /usr/bin/qemu-system-x86_64 -name guest=harv41,debug-threads=on -S -…

Harvester (Kubernetes/KubeVirt Engine)​

In Harvester, every VM is encapsulated inside a Kubernetes Pod. This introduces a strict cgroup boundary (kubepods.slice).

As shown below, multiple helper processes must live alongside the primary qemu-system-x86_64 process within the same tightly limited container memory budget:

-.slice
└─kubepods.slice
  │ ├─kubepods-burstable-pod99ee3a64_645b_4699_9384_5a3875d78b41.slice
  │ │ ├─cri-containerd-0c316eb8a4711bff1ce968b46ddb658e49378897454b4caf2a20704c808f33f1.scope …
  │ │ │ └─ 8505 /pause
  │ │ ├─cri-containerd-eb5deef29f064adfd2456d9f9c535674ac4d0c95c81b13afbdab5a89dc6a774b.scope …
  │ │ │ └─ 8590 /usr/bin/virt-tail --logfile /var/run/kubevirt-private/2ce151aa…
  │ │ └─cri-containerd-fd57a5cfc2b9b1f53eaf7b575c3273e6784f4c56a04a17d502ecfbd19e55b066.scope …
  │ │   ├─ 8542 /usr/bin/virt-launcher-monitor --qemu-timeout 301s --name vm2 -…
  │ │   ├─ 8558 /usr/bin/virt-launcher --qemu-timeout 301s --name vm2 --uid 2ce…
  │ │   ├─ 8591 /usr/sbin/virtqemud -f /var/run/libvirt/virtqemud.conf
  │ │   ├─ 8592 /usr/sbin/virtlogd -f /etc/libvirt/virtlogd.conf
  │ │   └─ 8823 /usr/bin/qemu-system-x86_64 -name guest=default_vm2,debug-threa…

The "Hidden" Memory Overhead​

Breaking Down the Memory Overhead Buffer​

When you define a virtual machine, for example, a VM configured with 4 vCPUs, 2 GiB of memory, and 1 Ethernet interface, KubeVirt does not just allocate exactly 2 GiB of memory to the container.

Instead, KubeVirt calculates an additional baseline memory overhead required to operate the virtualization stack. This overhead budget covers:

• CPU Simulators: Thread pools tracking guest state and handling context switches.
• Memory Management: Tracking structures such as QEMU page tables mapping guest RAM.
• Auxiliary Devices: Buffers for virtual network interfaces (NICs), storage queues, and video devices.

The High Stakes of VM OOM Kills​

Depending on the guest OS type, specific kernel workloads, and heavy storage/network I/O spikes, the memory consumed by these helper tasks can quickly exceed KubeVirt's default calculations. Because Kubernetes enforces a strict hard ceiling on the Pod container, the entire container triggers a CONSTRAINT_MEMCG OOM kill the moment this boundary is breached.

Unlike traditional, stateless Kubernetes workloads where a container crash is quickly mitigated by a rapid pod restart, an OOM kill on a VM pod carries severe operational consequences:

• Prolonged Downtime: A virtual machine is a stateful workload. It does not instantaneously serve traffic upon a container restart; it must undergo a full operating system boot cycle, run init scripts, and re-initialize services, drastically extending your Recovery Time Objective (RTO).

• Risk of Data Corruption: Sudden terminations during flight can abruptly cut off active storage queues. If the guest OS or database engine is in the middle of a critical write operation when the host terminates the qemu process, it can result in uncommitted journals, filesystem degradation, or severe data corruption on your persistent volumes.

The Solution: Tunable Memory Architectures​

To address this, Harvester introduced dual-layer configurations that give administrators full flexibility over how overhead buffers are calculated.

Global Adjustment: additional-guest-memory-overhead-ratio​

This cluster-wide setting functions as a multiplier for KubeVirt's automatically calculated memory overhead. For deep structural details, refer to the Harvester Advanced Documentation.

• Definition: Scales the calculated overhead buffer to accommodate heavy I/O or virtualization tasks.
• Default Value: 1.5 (Provides a 50% safety cushion above baseline calculations).
• Valid Range: 0 or 1.0 to 10.0.

💡 Important Operational Notes:

• Lifecycle Impact: Changes to this setting only apply to newly created virtual machines or existing VMs after they undergo a migration or a full power cycle.
• System Overhead: A higher ratio increases the host container's memory allocation, guaranteeing safety for heavy workloads but scaling up the overall system resource reservation footprint.
• Resource Allocation Trade-off: Setting this ratio excessively high can lock up unneeded host memory blocks, leading to predictable underutilization and significant memory waste across your compute nodes.

Per-VM Optimization: Reserved Memory​

For specific virtual machines running intensive or non-standard workloads, a global multiplier might not offer the precision required. Harvester allows administrators to define a dedicated Reserved Memory value directly on individual VMs. For complete configuration steps, see the Harvester VM Management Documentation.

⚠️ Under the Hood Memory Carving: When you configure this setting, Harvester explicitly scales down the available memory presented to the Guest OS inside the VM. For example, if a VM is configured with 2 GiB of memory and you set a Reserved Memory value of 256 MiB, the Guest OS will only see and utilize 1.75 GiB (2 GiB - 256 MiB).

Why Use Per-VM Reserved Memory?​

• Guaranteed Overhead Headroom: By restricting the Guest OS from consuming the top slice of its configured allocation, you guarantee an isolated, un-evictable memory runway for host helper tasks.

• Targeted Safety for Heavy Workloads: This mechanism is highly practical for mission-critical, high-performance, or special-purpose workloads (such as nested virtualization layers or intensive database engines). It effectively prevents the VM from running into host-level cgroup OOM termination by proactively limiting its internal usage boundaries, removing the risk of unexpected node-level kills.

• Optimized Cluster Usability: Using per-VM reservations eliminates the major disadvantage of cranking up the global additional-guest-memory-overhead-ratio for the whole cluster. Instead of forcing a massive, wasteful memory overhead reservation across every idle or lightweight VM on your hosts, you can maintain a lean global default and surgically protect only the heavy workloads—striking an ideal balance between system density and ironclad stability.

Outcome: Guaranteed Workload Stability​

By leveraging this dual-layer tunable memory architecture, Harvester fundamentally alters how host-level overhead is calculated, moving from rigid, generalized defaults to a precise, tiered enforcement model:

Total Memory Overhead = Auto-calculated Overhead * Ratio + Reserved Memory

Best Practices & Configuration Matrix​

The following matrix showcases how combinations of Reserved Memory and the Overhead Ratio change the actual layout of the Guest OS space versus what Kubernetes reserves as a hard boundary.

When optimizing your Harvester cluster to eliminate host-level container OOM events, use the following operational checklist to tailor your memory strategies:

• For General Workloads:

  • Stick to the default ratio of 1.5, or configure a slightly higher value of 2.0. This ensures that standard Guest operating systems receive exactly the memory requested while scaling out a stable, predictable background overhead buffer across the cluster.

• For High I/O and Storage-Heavy VMs:

  • If you observe periodic KVM OOM events during massive backup windows, large-scale data syncs, or intensive disk read/write cycles, increase the individual VM's allocation or implement a targeted Reserved Memory configuration to safely expand the helper overhead pool.

• For GPU Passthrough Workloads:

  • Virtual machines utilizing direct hardware acceleration or GPU passthrough are prime candidates for explicit Reserved Memory carving. The underlying host-side device drivers and memory-mapped I/O (MMIO) windows for high-performance graphics hardware require a significantly higher, specialized memory footprint outside the guest OS space. Allocating dedicated per-VM reserved memory prevents driver-instigated cgroup allocation breaches, keeping both the hardware pipeline and the hypervisor completely stable.

Quick Summary​

• The Problem: In Harvester's Kubernetes-native architecture, every virtual machine is bound by a strict Pod container limit. While this rigid cgroup boundary is essential for security, ensuring a single rogue or leaking VM can never starve neighboring workloads or crash the bare-metal host, it means heavy storage/network I/O, device drivers, or GPU passthrough can cause internal helper processes to breach this hard ceiling, triggering a sudden host-level OOM kill.

• The Solution: Harvester eliminates these crashes without losing secure resource control using a dual-layer memory tuning strategy:

  • Globally: The additional-guest-memory-overhead-ratio scales out a safety cushion cluster-wide for newly created or migrated VMs.
  • Per-VM: The Reserved Memory setting surgically carves out a chunk of the VM's configured RAM exclusively for background helper tasks—preventing wasteful memory reservations across the cluster while safely anchoring high-performance, mission-critical workloads.

Appendix: Lab Simulation — Manually Triggering the Host-Level OOM​

For engineers looking to validate this behavior safely in a staging environment, you can replicate this multi-process cgroup breach. A detailed script and case study can be found in the Harvester Development Summary: OOM Investigation.

The simulation process highlights a fundamental truth about modern virtualization boundaries:

• The Guest OS is Trustworthy: Testing shows that modern guest operating systems handle internal resource limits reliably. If a runaway application inside the guest OS eats up all available RAM, the guest kernel safely steps in and kills that specific process internally. The VM itself survives, and from the host's perspective, the virtual machine continues running normally.

• The Host Cgroup Boundary is the Weak Link: The true host-level crash only happens if processes inside the host cgroup expand unexpectedly. If an infrastructure task or helper process inside the Pod container balloons, it consumes the memory buffer that KubeVirt set aside, causing the entire cgroup, the VM's carrier, to slam into the hard Kubernetes ceiling and trigger a host-level OOM kill.

This article provides information and mitigation steps for the following vulnerabilities in Harvester:

• CVE-2026-42945
• CVE-2026-42946
• CVE-2026-40701
• CVE-2026-42934

You can confirm the version of the RKE2 ingress-nginx pods by running this command on your Harvester cluster:

kubectl -n kube-system get po -l"app.kubernetes.io/name=rke2-ingress-nginx" -ojsonpath='{.items[].spec.containers[].image}'

If the command returns one of the affected versions, perform the following mitigation steps.

The primary resolution is to upgrade Harvester to one of these versions:

• 1.7.2 or newer
• 1.8.1 or newer

If upgrade is not possible, apply the following mitigation to protect your clusters.

All ingress resources with the nginx.ingress.kubernetes.io/rewrite-target annotation containing ? in the annotation value are at risk.

By default, Harvester does not include any ingress resources with this annotation. Run the following command on your clusters to identify affected custom ingress resources:

kubectl get ingress -A -o json | jq '.items[] | select(.metadata.annotations["nginx.ingress.kubernetes.io/rewrite-target"] // "" | contains("?")) | {namespace: .metadata.namespace, name: .metadata.name, rewrite: .metadata.annotations["nginx.ingress.kubernetes.io/rewrite-target"]}'

Any ingress resources reported by the above command are vulnerable. They should be updated to either remove the vulnerable annotation or change the annotation value to not contain a question mark ?.

The following validating admission policy can be applied to your cluster to reject ingress resources with the vulnerable configuration:

cat<<EOF | kubectl apply -f -
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: ingress-nginx-annotation-validation-20260514
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups:   ["networking.k8s.io"]
      apiVersions: ["v1"]
      operations:  ["CREATE", "UPDATE"]
      resources:   ["ingresses"]
  validations:
    - expression: |
        !has(object.metadata.annotations) ||
        !object.metadata.annotations.exists(k, k == 'nginx.ingress.kubernetes.io/rewrite-target') ||
        !object.metadata.annotations['nginx.ingress.kubernetes.io/rewrite-target'].contains('?')
      message: "Ingress resources with 'nginx.ingress.kubernetes.io/rewrite-target' annotation containing '?' in the annotation value are not allowed, due to the following CVEs: CVE-2026-42945, CVE-2026-42946, CVE-2026-40701, CVE-2026-42934"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: ingress-nginx-annotation-validation-20260514
spec:
  policyName: ingress-nginx-annotation-validation-20260514
  validationActions: [Deny]
EOF

The policy can be removed once you upgrade to Harvester 1.7.2, 1.8.1 or newer:

kubectl delete validatingadmissionpolicy ingress-nginx-annotation-validation-20260514

kubectl delete validatingadmissionpolicybinding ingress-nginx-annotation-validation-20260514

References​

• https://depthfirst.com/research/nginx-rift-achieving-nginx-rce-via-an-18-year-old-vulnerability
• https://www.suse.com/security/cve/CVE-2026-42945.html
• https://www.suse.com/security/cve/CVE-2026-42946.html
• https://www.suse.com/security/cve/CVE-2026-40701.html
• https://www.suse.com/security/cve/CVE-2026-42934.html

You have lost the admin password for the Harvester GUI​

The admin password for the Harvester GUI can be reset if you can still login via ssh as the rancher user, or if you have the Harvester cluster's kubeconfig file saved locally. For details, see How can I reset the administrator password? in the documentation.

You have lost the rancher user's ssh/console login password​

But you still have the Harvester cluster's kubeconfig​

The rancher user's ssh/console login password can be reset by creating a CloudInit CRD to update the password. For details, see Password of user rancher in the documentation. Alternately you can create additional users with ssh access as described in How to create an SSH user for Harvester nodes.

You don't have a kubeconfig, but do have physical or remote console access​

In this case, the rancher user's ssh/console login password can be changed by editing /oem/90_custom.yaml on each host.

If you can boot the Harvester installer ISO​

Boot the Harvester installer, but don't proceed with the regular installation process. Instead, once the installer comes up, press CTRL-ALT-F2 to switch to VT2 and login as the rancher user with password rancher. Then proceed with the following steps:

1. Run sudo -i to become root
2. Mount the COS_OEM partition from the host:

   # mkdir /tmp/oem
   # mount -L COS_OEM /tmp/oem
   

   Copy
3. Run vim /tmp/oem/90_custom.yaml to edit 90_custom.yaml and change the password for the rancher user. You can specify either a plaintext password (not recommended) or a password hash generated with e.g. openssl passwd -6. Following is the section that you need to edit:

   users:
     rancher:
       passwd: <PASSWORD_GOES_HERE>
   

   Copy
4. Reboot the host. You should now be able to log in as the rancher user with the new password.

You can't boot the Harvester installer ISO, but can still reboot the host and access the boot menu​

If you have no other option, then during system boot, edit the grub config and add rd.break at the end of the kernel command line (the one that starts with linux). This will drop you into the dracut emergency shell, with the root partition mounted under /sysroot. Unfortunately, this shell does not contain any text editor programs. Trying to edit /oem/90_custom.yaml under the circumstances would be unwise. Instead though, you can run this command:

# sed -i 's%rancher.*%rancher:$6$j0.h3TQv8RZPHJkB$3SbV978JLT2Qeq4KSCBZitErNlZZGfrDxnGW5HS0wHzWexGyPzeQBoQmQJetUhLFfquv/X5VWL6odxtlEec1u/:20468::::::%' /sysroot/etc/shadow

Then, hit CTRL-D to continue, and once the system finishes booting, the rancher user's password will be set back to rancher for this boot only. You can then login on the console and use vim to update /oem/90_custom.yaml and permanently set the password to something more secure as described in the previous section.

This article provides information and mitigation steps for the following vulnerabilities in Harvester:

• CVE-2025-15566
• CVE-2026-1580
• CVE-2026-24512
• CVE-2026-24513
• CVE-2026-24514

You can confirm the version of the RKE2 ingress-nginx pods by running this command on your Harvester cluster:

kubectl -n kube-system get po -l"app.kubernetes.io/name=rke2-ingress-nginx" -ojsonpath='{.items[].spec.containers[].image}'

If the command returns one of the affected versions, perform one of the following mitigation steps.

The primary resolution is to upgrade to Harvester 1.7.1 or newer, which includes the fixed RKE2 ingress-nginx controller.

If upgrade is not possible, deploy the following validating admission policy to your cluster to reject ingress resources with the vulnerable configuration:

cat<<EOF | kubectl apply -f -
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: ingress-nginx-annotation-validation
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups:   ["networking.k8s.io"]
      apiVersions: ["v1"]
      operations:  ["CREATE", "UPDATE"]
      resources:   ["ingresses"]
  validations:
    - expression: |
        !('nginx.ingress.kubernetes.io/auth-proxy-set-headers' in object.metadata.annotations) &&
        !('nginx.ingress.kubernetes.io/auth-method' in object.metadata.annotations) &&
        (object.spec.rules.all(rule, rule.http.paths.all(path, path.pathType != 'ImplementationSpecific')))
      message: "Ingress resources with the vulnerable annotations are not allowed. Please remove the 'nginx.ingress.kubernetes.io/auth-proxy-set-headers' and 'nginx.ingress.kubernetes.io/auth-method' annotations, and avoid using the 'ImplementationSpecific' path type."
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: ingress-nginx-annotation-validation
spec:
  policyName: ingress-nginx-annotation-validation
  validationActions: [Deny]
EOF

This validating policy rejects any ingress resources that contain the:

• nginx.ingress.kubernetes.io/auth-proxy-set-headers annotation
• nginx.ingress.kubernetes.io/auth-method annotation
• ImplementationSpecific path type

The policy can be removed once you upgrade to Harvester 1.7.1 or newer:

kubectl delete validatingadmissionpolicy ingress-nginx-annotation-validation

kubectl delete validatingadmissionpolicybinding ingress-nginx-annotation-validation

References​

• https://github.com/kubernetes/kubernetes/issues/136789
• https://github.com/kubernetes/kubernetes/issues/136677
• https://github.com/kubernetes/kubernetes/issues/136678
• https://github.com/kubernetes/kubernetes/issues/136679
• https://github.com/kubernetes/kubernetes/issues/136680
• https://support.scc.suse.com/s/kb/Upstream-Ingress-Nginx-CVEs-CVE-2025-15566-CVE-2026-1580-CVE-2026-24512-CVE-2026-24513-and-CVE-2026-24514

Issue​

Node selector constraints can prevent the scheduler from live-migrating a virtual machine to a target node. This often indicates a mismatch between the virtual machine's requirements and the node's capabilities.

A node selector may require a specific CPU feature, but the target node lacks the corresponding label (for example, cpu-feature.node.kubevirt.io/fpu: "true"). This mismatch can occur when the host-model CPU models and features computed by KubeVirt change over time.

Solutions​

You can resolve this issue using four different approaches.

• Reboot the virtual machine.

  KubeVirt automatically adds node selectors (during a previous migration or initial start) that can restrict scheduling. You can clear these node selectors by rebooting the virtual machine.

• Reboot the virtual machine and set up a common CPU model.

  You can override KubeVirt's default host-model CPU configuration by setting up a common CPU model for virtual machine migration. The model is applied to the virtual machine as its domain CPU, and to the pod as its node selector configuration.

  This is the recommended approach for environments that can tolerate restarting of virtual machines.

• Modify the node labels.

  If rebooting the virtual machine is not an option, you can manually manipulate the target node's labels to satisfy the scheduling requirements.

  1. Add the node-labeller.kubevirt.io/skip-node="true" annotation to the target node.

     This annotation, which persists even after upgrades, prevents KubeVirt's node-labeller from automatically adding or removing CPU-related labels on this node.

     kubectl annotate node <node-name> node-labeller.kubevirt.io/skip-node="true"
     

     Copy

     important

     The annotation itself does not affect the pod's node selector. It only controls the presence of specific CPU-related labels on the node, which the node selector checks against. For more information, see the References section.

  2. Identify labels that are missing from the virtual machine's node selector and add them to the target node.

     You can add the missing labels using the following command:

     kubectl label node <node-name> <key>=<value>
     

     Copy

     This circumvents the standard scheduling restrictions, allowing the virtual machine to migrate to the target node.

  If a new node that lacks the required features is added to the cluster, you must repeat these steps to allow the virtual machine to live-migrate to that node.

• Remove the node labels.

  If you want to ensure that the virtual machine does not acquire specific node selector constraints after live migration, you can remove the relevant CPU labels from the target node.

  1. Add the node-labeller.kubevirt.io/skip-node="true" annotation to the target node.

     This annotation, which persists even after upgrades, prevents KubeVirt's node-labeller from automatically adding or removing CPU-related labels on this node.

     kubectl annotate node <node-name> node-labeller.kubevirt.io/skip-node="true"
     

     Copy

     important

     This method works only if the virtual machine's pod does not have an existing node selector that contains the labels listed in the References section. Otherwise, you must reboot the virtual machine to clear the constraints.

  2. Check if the pod has a node selector.

     kubectl get pod <pod-name> -o yaml | grep nodeSelector -A 5 -B 5
     

     Copy

  3. If no node selector exists, remove the relevant CPU labels from the node.

     Performing this action prevents the pod from acquiring new node selector constraints, thus enabling its future migration to other nodes. However, the successful outcome of that migration is not guaranteed.

References​

KubeVirt Node Labels​

The KubeVirt CPU node-labeller manages the following labels:

• cpu-feature.node.kubevirt.io/*
• cpu-model-migration.node.kubevirt.io/*
• cpu-model.node.kubevirt.io/*
• host-model-cpu.node.kubevirt.io
• host-model-required-features.node.kubevirt.io

External Links​

• Labeling nodes with CPU models, CPU features and machine types

When guest virtual machines running on Harvester nodes experience very slow network throughput, disabling Generic Receive Offload (GRO) and Generic Segmentation Offload (GSO) on the host interfaces may resolve the issue.

Problem​

Symptoms​

In the testing environment, guest virtual machines experienced severely degraded download and transfer speeds, dropping as low as 100 bps. This extreme slowdown was particularly evident when apt-get update, curl, and scp were used to transfer data between virtual machines running on different nodes. In contrast, performance remained normal when the virtual machines were hosted on the same node.

Environment​

The issue was observed in a Harvester cluster hosted on Dell servers using Broadcom NetXtreme-E Series BCM57508 NICs (100 Gbps). mgmt, the built-in cluster network, was used for both management and virtual machine traffic.

Cause​

Root Cause​

Harvester relies on Linux’s bridge-based virtual networking to connect guest virtual machines to physical networks.

The NetXtreme-E BCM57508 NICs were connected to leaf switches configured to transmit jumbo frames. When the default MTU of 1500 is used, these frames should ideally be segmented to approximately 1450 bytes before reaching the Harvester host kernel. However, the packets actually arriving at the kernel were fragmented into unexpectedly small sizes. This forced the kernel to process a significantly higher volume of packets, leading to increased CPU overhead and reduced download throughput.

Packets captures collected using the following command confirmed the unexpectedly small size of the incoming packets.

tcpdump -xx -i <interface-name>

<interface-name> is the name of the physical interface on the host connected to the VMs.

GRO/GSO offload​

Generic Receive Offload (GRO) and Generic Segmentation Offload (GSO) are kernel-level software offloading mechanisms designed to optimize network performance. GRO aggregates multiple small incoming packets into larger ones before passing them to the network stack. GSO performs the opposite on transmission, splitting large packets into smaller frames before sending them to the NIC.

While these features are typically used to enhance performance, in this specific scenario, they interfered with the normal TCP segmentation process. This interference led to inefficient packet segmentation and the creation of an excessive number of small fragments, which ultimately degraded overall network performance.

When GRO and GSO were disabled, the Linux network stack automatically reverted to using standard transport-layer segmentation methods, specifically TCP Segmentation Offload (TSO) and Large Receive Offload (LRO). These mechanisms maintained efficient packet aggregation and segmentation at the appropriate layers, ensuring properly sized packets were presented to the kernel, which successfully restored expected network performance.

The NetXtreme-E BCM57508 NICs may experience suboptimal interaction with GRO and GSO due to a Broadcom driver bug. Enabling these offload mechanisms led to inefficient packetization, producing many small packets instead of fewer large ones, which ultimately reduced network throughput.

Solution​

• Option 1: Disable GRO and GSO on the affected Harvester host interfaces. This change does not persist across reboots.

  # Check current offload settings
  /usr/sbin/ethtool -k <interface-name>
  # Disable GRO and GSO
  /usr/sbin/ethtool -K <interface-name> gro off
  /usr/sbin/ethtool -K <interface-name> gso off
  

  Copy

• Option 2: Apply the following cloudinit resource and reboot the nodes. This change persists across reboots.

  Replace <interface-name> with the name of the physical interface the virtual machines are connected to.

  apiVersion: node.harvesterhci.io/v1beta1
  kind: CloudInit
  metadata:
    name: disable-offloads
  spec:
    matchSelector: {}
    filename: 99_disable_offloads.yaml
    contents: |
      stages:
        network:
          - commands:
              - /usr/sbin/ethtool -K <interface-name> gro off
              - /usr/sbin/ethtool -K <interface-name> gso off
  

  Copy

Verification​

• Run apt-get update or a curl command from a guest virtual machine. The download throughput should be normal (utilizing most of the link capacity).
• Transfer files between virtual machines running on different nodes using scp. The file transfers should be completed at the expected speed.
• Verify that no packets are dropped and no errors are reported using the following command:

  ip -s link show <interface-name>
  

  Copy

When working with Longhorn, you may encounter two different VolumeAttachment resources with similar names: Kubernetes VolumeAttachment (storage.k8s.io/v1) and Longhorn VolumeAttachment (longhorn.io/v1beta2). This often causes confusion about why both exist, when each is created, whether they always appear together, and which one to check when troubleshooting. This document clarifies their distinct roles, shows how they work together (and when they don't), and provides real-world examples to help you identify attachment sources and effectively troubleshoot volume attachment issues.

For additional context, see the official documentation at https://longhorn.io/docs/latest/advanced-resources/volumeattachment/

Workflow: How K8s and Longhorn VolumeAttachments Work Together​

When a Pod requires a Longhorn volume, two separate VolumeAttachment resources work together to complete the attachment process. The Kubernetes VolumeAttachment represents the CSI standard attachment request, while the Longhorn VolumeAttachment manages the actual attachment orchestration with ticket-based coordination.

The following diagram illustrates the complete flow from Pod scheduling to successful volume attachment:

┌─────────────────────────────────────────────────────────────┐
│                  Pod Scheduled to Node                      │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Kubernetes Attach/Detach Controller                         │
│ Creates K8s VolumeAttachment                                │
│   APIVersion: storage.k8s.io/v1                             │
│   Spec:                                                     │
│     Attacher: driver.longhorn.io                            │
│     NodeName: worker-node-1                                 │
│     Source.PersistentVolumeName: pvc-xxx                    │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ CSI External-Attacher (Longhorn)                            │
│ Watches K8s VolumeAttachment                                │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Longhorn CSI Plugin                                         │
│ Calls ControllerPublishVolume()                             │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Longhorn Manager API                                        │
│ Creates/Updates Longhorn VolumeAttachment                   │
│   APIVersion: longhorn.io/v1beta2                           │
│   Spec:                                                     │
│     Volume: my-volume                                       │
│     AttachmentTickets:                                      │
│       csi-attacher-<hash>:                                  │
│         ID: <pod-id>                                        │
│         Type: csi-attacher                                  │
│         NodeID: worker-node-1                               │
│         Parameters: {...}                                   │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Longhorn VolumeAttachment Controller                        │
│ 1. Evaluates all attachment tickets                         │
│ 2. Selects appropriate ticket to satisfy                    │
│ 3. Updates Volume.Spec.NodeID = worker-node-1               │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Longhorn Volume Controller                                  │
│ Performs actual volume attachment operation                 │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Longhorn VolumeAttachment Controller                        │
│ Updates ticket status: Satisfied = true                     │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Longhorn CSI Plugin                                         │
│ Returns attach success to external-attacher                 │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ CSI External-Attacher                                       │
│ Updates K8s VolumeAttachment.Status.Attached = true         │
└─────────────────────────────────────────────────────────────┘

The resulting Longhorn VolumeAttachment YAML:

apiVersion: longhorn.io/v1beta2
kind: VolumeAttachment
metadata:
  name: pvc-0b9c8d59-0ae8-413c-8bc5-af32b932b8ab
  namespace: longhorn-system
  labels:
    longhornvolume: pvc-0b9c8d59-0ae8-413c-8bc5-af32b932b8ab
spec:
  volume: pvc-0b9c8d59-0ae8-413c-8bc5-af32b932b8ab
  attachmentTickets:
    # This CSI ticket was triggered by K8s VolumeAttachment (Pod binding)
    csi-3d3120f43480db87c91a6902d670c35899917c03f9f6f81db7bf26d9d66e45ec:
      id: csi-3d3120f43480db87c91a6902d670c35899917c03f9f6f81db7bf26d9d66e45ec
      type: csi-attacher
      nodeID: harvester-node-1
      parameters:
        disableFrontend: "false"
status:
  attachmentTicketStatuses:
    csi-3d3120f43480db87c91a6902d670c35899917c03f9f6f81db7bf26d9d66e45ec:
      id: csi-3d3120f43480db87c91a6902d670c35899917c03f9f6f81db7bf26d9d66e45ec
      satisfied: true  # Volume successfully attached
      conditions:
      - type: Satisfied
        status: "True"

Notice the csi-attacher ticket type - this confirms the attachment was triggered by Kubernetes VolumeAttachment through the CSI flow, not by Longhorn internal operations.

Trigger Points​

Understanding when each VolumeAttachment is created or modified is crucial for troubleshooting attachment issues:

1. K8s VolumeAttachment Creation: Triggered when Pod is scheduled to a node requiring a PVC

   • Managed by Kubernetes Attach/Detach (AD) Controller
   • One VolumeAttachment per PV-node combination
   • Represents Kubernetes' intent to attach the volume

2. Longhorn VolumeAttachment Ticket Addition: Triggered by various Longhorn components based on operation needs:

   • CSIAttacher - when CSI ControllerPublishVolume is called
   • SnapshotController - when creating snapshots of volumes
   • BackupController - when backing up volumes
   • LonghornAPI - when users manually attach volumes via Longhorn UI
   • VolumeCloneController - when managing source volume during clone
   • VolumeRestoreController - when restoring data from backups
   • VolumeExpansionController - when expanding volume size
   • ShareManagerController - for RWX volume sharing
   • SalvageController - for volume salvage operations

Attachment Ticket Priority and Coordination​

When multiple operations require volume attachment simultaneously, Longhorn uses a ticket-based priority system to coordinate access intelligently. This ensures critical operations take precedence while allowing background tasks to coexist when possible.

How Priority Works​

Each ticket type has an assigned priority level that determines selection order when the volume is detached:

• Priority 2000 (Highest):
  • VolumeRestoreController
  • VolumeExpansionController
• Priority 1000:
  • LonghornAPI
• Priority 900:
  • CSIAttacher
  • ShareManagerController
  • SalvageController
• Priority 800 (Lowest):
  • BackupController
  • SnapshotController
  • VolumeCloneController
  • VolumeEvictionController

When the volume is detached, the ticket with the highest priority is selected for attachment. If multiple tickets share the same priority, the first one (sorted by ID) is chosen.

Interruption Mechanism​

Priority levels alone don't tell the complete story. Longhorn also implements an interruption mechanism to handle cases where request arrives while the volume is already attached to a different node.

Interruptible operations (can be interrupted):

• BackupController
• SnapshotController
• VolumeCloneController - clone operations, but only when the volume is in VolumeCloneStateCopyCompletedAwaitingHealthy state

Workload operations (can trigger interruption):

• CSIAttacher - Pod workloads requiring the volume on a different node
• LonghornAPI - manual attachment requests via UI/API
• ShareManagerController - RWX volume sharing operations

The interruption only occurs when:

1. The volume's currently attached node has only interruptible tickets
2. A different node has a workload ticket requesting the volume

This design ensures background operations never block workload rescheduling, while protecting active workloads from being interrupted by other background tasks.

Real-World Scenarios​

Scenario 1: Backup During Active Pod Usage

• Pod is running on node-A with a CSIAttacher ticket
• BackupController creates a ticket for node-A (same node)
• Both tickets coexist peacefully - backup runs alongside the Pod
• CSI attachment and backup execution use the engine on the same node, avoiding a node transition.

Scenario 2: Backup Interrupted by Pod Workload

• BackupController is running on node-A (only ticket present)
• A Pod requiring this volume is scheduled to node-B, CSIAttacher creates a ticket for node-B
• VolumeAttachment Controller detects: interruptible ticket on node-A, workload ticket on node-B
• Volume detaches from node-A (backup interrupted), attaches to node-B (csi attacher)
• Backup will retry later automatically

Scenario 3: Detached Volume Snapshot

• Volume is detached, SnapshotController creates a ticket
• Volume attaches temporarily for snapshot creation
• After snapshot completes, ticket is removed
• Volume auto-detaches if no other tickets exist

Usage Examples​

The following examples demonstrate how VolumeAttachment resources behave in common scenarios. Each example shows the complete YAML resource state at different stages, helping you understand what to look for when troubleshooting or monitoring Longhorn operations.

Example 1: VolumeSnapshot Creation (Longhorn VolumeAttachment Only)​

VolumeSnapshot operations use only Longhorn VolumeAttachment without involving Kubernetes VolumeAttachment. This demonstrates that Longhorn VolumeAttachment can operate independently for internal operations.

┌─────────────────────────────────────────────────────────────┐
│           User Creates VolumeSnapshot via kubectl           │
│           kubectl apply -f volumesnapshot.yaml              │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Longhorn Snapshot Controller                                │
│ Detects new VolumeSnapshot resource                         │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Snapshot Controller Checks Volume State                     │
│ If Volume is detached → needs attachment for snapshot       │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Snapshot Controller Creates Attachment Ticket               │
│ Updates Longhorn VolumeAttachment:                          │
│   AttachmentTickets:                                        │
│     snapshot-<snapshot-name>:                               │
│       Type: snapshot-controller                             │
│       NodeID: <volume-owner-node>                           │
│       Parameters: {disableFrontend: "false"}                │
│                                                             │
│ ❌ No K8s VolumeAttachment created                          │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Longhorn VolumeAttachment Controller                        │
│ Selects snapshot ticket → Updates Volume.Spec.NodeID        │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Longhorn Volume Controller                                  │
│ Attaches volume → Starts Engine                             │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Snapshot Controller                                         │
│ Engine running → Creates snapshot via Engine API            │
│ Snapshot complete → Removes attachment ticket               │
└─────────────────────┬───────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────┐
│ Volume May Auto-Detach (if no other tickets exist)          │
└─────────────────────────────────────────────────────────────┘

The Longhorn VolumeAttachment YAML during snapshot creation:

During Snapshot Creation (ticket exists):

apiVersion: longhorn.io/v1beta2
kind: VolumeAttachment
metadata:
  name: pvc-0b9c8d59-0ae8-413c-8bc5-af32b932b8ab
  namespace: longhorn-system
  generation: 30
spec:
  volume: pvc-0b9c8d59-0ae8-413c-8bc5-af32b932b8ab
  attachmentTickets:
    # Temporary ticket created by Snapshot Controller
    snapshot-controller-snapshot-a36bedf5-fb3b-4b30-a10d-ed98f9c0323a:
      id: snapshot-controller-snapshot-a36bedf5-fb3b-4b30-a10d-ed98f9c0323a
      type: snapshot-controller
      nodeID: harvester-node-1
      parameters:
        disableFrontend: any
status:
  attachmentTicketStatuses:
    snapshot-controller-snapshot-a36bedf5-fb3b-4b30-a10d-ed98f9c0323a:
      id: snapshot-controller-snapshot-a36bedf5-fb3b-4b30-a10d-ed98f9c0323a
      satisfied: false  # Snapshot in progress
      conditions:
      - type: Satisfied
        status: "False"

After Snapshot Completes (ticket removed):

apiVersion: longhorn.io/v1beta2
kind: VolumeAttachment
metadata:
  name: pvc-0b9c8d59-0ae8-413c-8bc5-af32b932b8ab
  namespace: longhorn-system
  generation: 31  # Incremented after ticket removal
spec:
  volume: pvc-0b9c8d59-0ae8-413c-8bc5-af32b932b8ab
  attachmentTickets: {}  # Ticket removed after snapshot completes
status:
  attachmentTicketStatuses: {}

Key Observations:

• The snapshot-controller ticket type clearly identifies this as a Longhorn internal operation
• Unlike csi-attacher tickets (triggered by K8s), this ticket is created purely by Longhorn
• The ticket is temporary - it appears during snapshot creation and disappears when complete
• No corresponding Kubernetes VolumeAttachment exists for this operation

Example 2: VM Migration​

During VM migration, Harvester has two virt-launcher pods for the same VM: the original pod on the source node and a new pod on the target node. This multi-attach capability is enabled for RWX (ReadWriteMany) block mode volumes when the StorageClass has migratable: true parameter, which allows Longhorn to support live VM migration. In the following example, we migrate a VM from harvester-node-2 to harvester-node-0.

apiVersion: longhorn.io/v1beta2
kind: VolumeAttachment
metadata:
  creationTimestamp: "2025-12-10T04:19:42Z"
  finalizers:
  - longhorn.io
  generation: 3
  labels:
    longhornvolume: pvc-0dc9e1f0-4932-4567-aa1e-e70b570058da
  name: pvc-0dc9e1f0-4932-4567-aa1e-e70b570058da
  namespace: longhorn-system
  ownerReferences:
  - apiVersion: longhorn.io/v1beta2
    kind: Volume
    name: pvc-0dc9e1f0-4932-4567-aa1e-e70b570058da
    uid: 7cd2ed46-194f-4528-83f7-bbaa5945e7e3
  resourceVersion: "2736440"
  uid: b2492681-8fcb-4330-9ec6-496afa93e96b
spec:
  attachmentTickets:
    csi-5852f2d48d96311bb582eeeaad0e38361031d502899416c71cea10795748a84b:
      generation: 0
      id: csi-5852f2d48d96311bb582eeeaad0e38361031d502899416c71cea10795748a84b
      nodeID: harvester-node-2
      parameters:
        disableFrontend: "false"
        lastAttachedBy: ""
      type: csi-attacher
    csi-f080d69495b619fad93621ff3d57201793952e422304cceac8807e975ccf795d:
      generation: 0
      id: csi-f080d69495b619fad93621ff3d57201793952e422304cceac8807e975ccf795d
      nodeID: harvester-node-0
      parameters:
        disableFrontend: "false"
        lastAttachedBy: ""
      type: csi-attacher
  volume: pvc-0dc9e1f0-4932-4567-aa1e-e70b570058da
status:
  attachmentTicketStatuses:
    csi-5852f2d48d96311bb582eeeaad0e38361031d502899416c71cea10795748a84b:
      conditions:
      - lastProbeTime: ""
        lastTransitionTime: "2025-12-10T04:19:49Z"
        message: ""
        reason: ""
        status: "True"
        type: Satisfied
      generation: 0
      id: csi-5852f2d48d96311bb582eeeaad0e38361031d502899416c71cea10795748a84b
      satisfied: true
    csi-f080d69495b619fad93621ff3d57201793952e422304cceac8807e975ccf795d:
      conditions:
      - lastProbeTime: ""
        lastTransitionTime: "2025-12-10T04:21:00Z"
        message: The migrating attachment ticket is satisfied
        reason: ""
        status: "True"
        type: Satisfied
      generation: 0
      id: csi-f080d69495b619fad93621ff3d57201793952e422304cceac8807e975ccf795d
      satisfied: true

After Migration Completes (ticket removed):

apiVersion: longhorn.io/v1beta2
kind: VolumeAttachment
metadata:
  creationTimestamp: "2025-12-10T04:19:42Z"
  finalizers:
  - longhorn.io
  generation: 4
  labels:
    longhornvolume: pvc-0dc9e1f0-4932-4567-aa1e-e70b570058da
  name: pvc-0dc9e1f0-4932-4567-aa1e-e70b570058da
  namespace: longhorn-system
  ownerReferences:
  - apiVersion: longhorn.io/v1beta2
    kind: Volume
    name: pvc-0dc9e1f0-4932-4567-aa1e-e70b570058da
    uid: 7cd2ed46-194f-4528-83f7-bbaa5945e7e3
  resourceVersion: "2736824"
  uid: b2492681-8fcb-4330-9ec6-496afa93e96b
spec:
  attachmentTickets:
    csi-f080d69495b619fad93621ff3d57201793952e422304cceac8807e975ccf795d:
      generation: 0
      id: csi-f080d69495b619fad93621ff3d57201793952e422304cceac8807e975ccf795d
      nodeID: harvester-node-0
      parameters:
        disableFrontend: "false"
        lastAttachedBy: ""
      type: csi-attacher
  volume: pvc-0dc9e1f0-4932-4567-aa1e-e70b570058da
status:
  attachmentTicketStatuses:
    csi-f080d69495b619fad93621ff3d57201793952e422304cceac8807e975ccf795d:
      conditions:
      - lastProbeTime: ""
        lastTransitionTime: "2025-12-10T04:21:00Z"
        message: ""
        reason: ""
        status: "True"
        type: Satisfied
      generation: 0
      id: csi-f080d69495b619fad93621ff3d57201793952e422304cceac8807e975ccf795d
      satisfied: true

Key Observations:

• Two CSI attachment tickets coexist: One pointing to the source node (harvester-node-2) and another to the target node (harvester-node-0)
• Both tickets are csi-attacher type: Indicating they were both triggered by Kubernetes VolumeAttachment through the CSI flow
• Both tickets have satisfied: true status: This demonstrates Longhorn's support for attaching the same volume to multiple nodes simultaneously (RWX-like behavior for migration)
• Target node ticket has special message: "The migrating attachment ticket is satisfied" explicitly identifies this as a migration scenario
• Multi-attach is temporary: This dual-attachment state only exists during VM migration; the source node's ticket will be removed after migration completes

Summary​

Longhorn uses two different VolumeAttachment resources for different purposes:

Kubernetes VolumeAttachment (storage.k8s.io/v1) follows the standard CSI specification and is created only when Pods are scheduled to nodes. It represents Kubernetes' official attachment intent and is managed by K8s Attach/Detach Controller and CSI External-Attacher.

Longhorn VolumeAttachment (longhorn.io/v1beta2) extends beyond CSI to support Longhorn's advanced features. It's created for multiple scenarios, including Pod workloads, snapshots, backups, clones, and manual operations. It uses a ticket-based system to coordinate concurrent attachment requests and is managed collaboratively by multiple Longhorn controllers.

Why both are needed: K8s VolumeAttachment ensures CSI compliance with the Kubernetes ecosystem, while Longhorn VolumeAttachment enables automation for background operations without manual intervention. Importantly, not all Longhorn operations trigger K8s VolumeAttachment—for example, creating a VolumeSnapshot only creates a Longhorn VolumeAttachment ticket (snapshot-controller), not a K8s VolumeAttachment.

When troubleshooting: Check both resources. K8s VolumeAttachment shows the CSI standard workflow status, while Longhorn VolumeAttachment shows the complete picture, including all internal operations via attachment tickets. Look at the ticket type to identify the operation source: csi-attacher means triggered by the K8s VolumeAttachment (Pod workload), while snapshot-controller, backup-controller, etc. indicate Longhorn internal operations.

In this Harvester Knowledge Base article, Ivan Sim provided comprehensive guidance on using Velero to perform backup and restore operations for VMs with external storage in Harvester.

However, in certain scenarios, users may require the VM filesystem to be quiesced during Velero backup creation to prevent data corruption, especially when the VM is experiencing heavy I/O operations.

This article describes how to customize Velero Backup Hooks to implement filesystem freeze during Velero backup processing, ensuring data consistency in the backup content.

Background Knowledge​

KubeVirt's virt-freezer provides a mechanism to freeze and thaw guest filesystems. This capability can be leveraged to ensure filesystem consistency during VM backups. However, certain prerequisites must be met for filesystem freeze/thaw operations to function properly:

Prerequisites for Filesystem Freeze​

• QEMU Guest Agent must be enabled in the guest VM
  • Verify this by checking if the VMI has AgentConnected in its status
• Guest VM must be properly configured for related libvirt commands
  • When virt-freezer is triggered, KubeVirt communicates with the QEMU Guest Agent via libvirt commands such as guest-fsfreeze-freeze
  • The guest agent translates these commands to OS-specific calls:
    • Linux systems: Uses fsfreeze syscalls
    • Windows systems: Uses VSS (Volume Shadow Copy Service) APIs

Common Configuration Challenges​

Based on Harvester project experience, some guest operating systems require additional configuration:

• Linux distributions (e.g., RHEL, SLE Micro): May lack sufficient permissions for filesystem freeze operations by default, requiring custom policies
• Windows guests: Require the VSS service to be enabled for filesystem freeze functionality

Important: Filesystem freeze/thaw functionality depends on guest VM configuration, which is outside Harvester's control. Users are responsible for ensuring compatibility before implementing Velero backup hooks with filesystem freeze.

Verifying Filesystem Freeze Compatibility​

To confirm that your VM supports filesystem freeze operations:

1. Access the virtual machine's virt-launcher compute container:

   POD=$(kubectl get pods -n <VM Namespace> \
     -l vm.kubevirt.io/name=<VM Name> \
     -o jsonpath='{.items[0].metadata.name}')
   kubectl exec -it $POD -n default -c compute -- bash
   

   Copy

2. Test filesystem freeze using the virt-freezer application available in the compute container:

   virt-freezer --freeze --namespace <VM namespace> --name <VM name>
   

   Copy

3. Critical: Always verify the freeze operation result and thaw the VM filesystems before performing any other operations

Prerequisites​

All preparation steps outlined in External CSI Storage Backup and Restore With Velero are mandatory, including:

• Harvester installation and configuration
• Velero installation and setup
• S3-compatible storage configuration
• Proper networking and permissions

Implementing Filesystem Freeze Hooks for VM Backup Consistency​

Velero supports pre and post backup hooks that can be integrated with KubeVirt's virt-freezer to ensure filesystem consistency during VM backups.

Configuring VM Template Annotations​

For all VMs requiring data consistency, add the following annotations to the VM template:

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: vm-nfs
  namespace: demo
spec:
  template:
    metadata:
      annotations:
        # These annotations will be applied to the virt-launcher pod
        pre.hook.backup.velero.io/command: '["/usr/bin/virt-freezer", "--freeze", "--namespace", "<VM Namespace>", "--name", "<VM Name>"']'
        pre.hook.backup.velero.io/container: compute
        pre.hook.backup.velero.io/on-error: Fail
        pre.hook.backup.velero.io/timeout: 30s
        
        post.hook.backup.velero.io/command: '["/usr/bin/virt-freezer", "--unfreeze", "--namespace", "<VM Namespace>", "--name", "<VM Name>"]'
        post.hook.backup.velero.io/container: compute
        post.hook.backup.velero.io/timeout: 30s
    spec:
      # ...rest of VM spec...

These annotations will be propagated to the related virt-launcher pod and instruct Velero to:

• Freeze the VM filesystem before backup creation begins
• Thaw the VM filesystem after backup completion

Important: Replace <VM Namespace> and <VM Name> with the actual namespace and name of your VM.

Creating a Velero Backup with Filesystem Freeze​

After applying the Velero pre/post hook annotations to the VM manifest, follow the backup procedures described in External CSI Storage Backup and Restore With Velero.

Verifying Successful Hook Execution​

If the guest VM is configured correctly, the Velero backup will complete successfully with HooksAttempted indicating successful hook execution.

Check the backup status using:

velero backup describe [Backup Name] --details

Example output showing successful hook execution:

Name:         demo
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.33.3+rke2r1
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=33

Phase:  Completed


Namespaces:
  Included:  demo
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Or label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  auto
Snapshot Move Data:          true
Data Mover:                  velero

....

Backup Volumes:
  Velero-Native Snapshots: <none included>

  CSI Snapshots:
    demo/vm-nfs-disk-0-au2ej:
      Data Movement:
        Operation ID: du-be5417aa-498e-4b93-b59f-e6498f95a6df.d7f97dab-3bb1-41e189381
        Data Mover: velero
        Uploader Type: kopia
        Moved data Size (bytes): 5368709120
        Result: succeeded

  Pod Volume Backups: <none included>

HooksAttempted:  2
HooksFailed:     0

The output shows that Velero pre/post backup hooks completed successfully. In this case, the hooks are connected to guest VM filesystem freeze and thaw operations to ensure data consistency.

Restoring the Velero Backup​

Follow the restoration procedures described in External CSI Storage Backup and Restore With Velero to restore the namespace using Velero.

Troubleshooting​

If you encounter issues with filesystem freeze operations:

1. Verify QEMU Guest Agent status in the VMI
2. Check guest OS configuration for filesystem freeze support
3. Review Velero hook logs for specific error messages
4. Test virt-freezer manually as described in the verification section

Conclusion​

Implementing filesystem freeze hooks with Velero ensures data consistency during VM backups by quiescing the filesystem before snapshot creation. This approach is particularly valuable for VMs with high I/O activity or critical data that requires point-in-time consistency guarantees.