Skip to main content

Longhorn V2 Data Engine Kernel Workqueue Lockup

· 20 min read
Cooper Tseng

After enabling the Longhorn V2 Data Engine in Harvester, some nodes may report kernel workqueue lockups and become unstable. In severe cases, rke2-server may terminate, Longhorn may stop making progress, and the node may require recovery.

This issue is most likely to occur when SPDK is pinned to dedicated CPUs, but the Linux kernel is still allowed to run IRQ handlers or unbound workqueue workers on those same CPUs.

caution

The mitigation described in this article changes host CPU affinity for IRQs and kernel workqueues. These settings are node-wide and can affect every workload on the node. Apply the workaround only to nodes that run the Longhorn V2 Data Engine, and validate the CPU mask carefully before changing any host settings.

Affected Scenario​

This issue can affect Harvester clusters that meet all of the following conditions:

  • The Longhorn V2 Data Engine is enabled.
  • The Longhorn V2 instance-manager pod runs SPDK (spdk_tgt), which busy-polls on the CPUs selected by the Longhorn V2 data engine CPU mask.
  • Host IRQ handling or unbound kernel workqueues are still allowed to run on the same busy CPUs used by spdk_tgt.

The default Longhorn V2 CPU mask is 0x3, which maps to CPUs 0 and 1. However, the issue is not limited to CPUs 0 and 1; it can occur with any CPU mask if kernel IRQs or unbound workqueues still run on the CPUs occupied by spdk_tgt.

The issue was observed on bare-metal Harvester nodes with Longhorn V2 enabled. It is not always reproducible in every environment.

Symptoms​

Affected nodes may show one or more of the following symptoms:

  • Kernel logs repeatedly contain BUG: workqueue lockup.
  • The lockup is reported on a CPU that is part of the Longhorn V2 data engine CPU mask.
  • rke2-server becomes unstable or terminates without a clear user-visible reason.
  • Longhorn volumes, engines, replicas, or instance managers stop progressing.
  • User switching commands, such as sudo -i, may become very slow before the root shell appears.
  • The node becomes unstable or unhealthy.

Example kernel message:

Jul 27 07:27:55 hp-114-tink-system kernel: BUG: workqueue lockup - pool cpus=0 node=0 flags=0x0 nice=0 stuck for 2117s!
Jul 27 07:27:55 hp-114-tink-system kernel: Showing busy workqueues and worker pools:
Jul 27 07:27:55 hp-114-tink-system kernel: workqueue events: flags=0x0
Jul 27 07:27:55 hp-114-tink-system kernel:   pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=7 refcnt=8
Jul 27 07:27:55 hp-114-tink-system kernel:     in-flight: 2935011:output_poll_execute ,32008:drm_fb_helper_damage_work drm_fb_helper_damage_work
Jul 27 07:27:55 hp-114-tink-system kernel:     pending: vmstat_shepherd, switchdev_deferred_process_work, rht_deferred_worker, key_garbage_collector
Jul 27 07:27:55 hp-114-tink-system kernel: workqueue events_unbound: flags=0x2
Jul 27 07:27:55 hp-114-tink-system kernel:   pwq 98: cpus=0-23 node=1 flags=0x4 nice=0 active=4 refcnt=8
Jul 27 07:27:55 hp-114-tink-system kernel: workqueue events_power_efficient: flags=0x80
Jul 27 07:27:55 hp-114-tink-system kernel:   pwq 2: cpus=0 node=0 flags=0x0 nice=0 active=4 refcnt=5
Jul 27 07:27:55 hp-114-tink-system kernel:     pending: neigh_managed_work, neigh_periodic_work, gc_worker [nf_conntrack], check_lifetime
Jul 27 07:27:55 hp-114-tink-system kernel: pool 2: cpus=0 node=0 flags=0x0 nice=0 hung=2117s workers=5 idle: 1025200 1541794
Jul 27 07:27:55 hp-114-tink-system kernel: task:kworker/0:0     state:R  running task     stack:0     pid:32008 tgid:32008 ppid:2      flags:0x00004000
Jul 27 07:27:55 hp-114-tink-system kernel: Workqueue: events drm_fb_helper_damage_work
Jul 27 07:28:12 hp-114-tink-system rke2[2909320]: time="2026-07-27T07:28:12Z" level=warning msg="Proxy error: write failed: write tcp 127.0.0.1:9345->127.0.0.1:46562: write: connection reset by peer"

Root Cause​

Longhorn V2 uses SPDK. SPDK reactor threads are designed to busy-poll on the CPUs selected by the Longhorn V2 data engine CPU mask. This is expected for high-performance storage I/O.

The problem occurs when normal host kernel work is also allowed to run on those same CPUs. For example:

  • NIC or storage IRQs may still target the SPDK CPUs.
  • Unbound kernel workqueues may still include the SPDK CPUs in /sys/devices/virtual/workqueue/cpumask.
  • Per-workqueue CPU masks may still allow existing workers to run on the SPDK CPUs.

When SPDK fully occupies those CPUs, kernel workers can be delayed long enough for the kernel workqueue lockup detector to fire. Short or occasional workqueue lockup reports do not always mean the node is unrecoverable. The operational problem is when the delayed kernel work keeps accumulating and cascades into networking, RKE2, kubelet, and Longhorn instability.

For example, if the Longhorn V2 CPU mask is the default value 0x3, SPDK uses CPUs 0 and 1. IRQs and kernel workqueues should be moved away from CPUs 0 and 1.

Confirming the Issue​

Run the following checks on each affected Harvester node.

1. Check Kernel Logs​

journalctl -k --since "24 hours ago" | grep -E "BUG: workqueue lockup|soft lockup"

Look for the CPU reported in messages such as pool cpus=0, pool cpus=1, kworker/0, or kworker/1.

2. Check the Longhorn V2 CPU Mask​

kubectl -n longhorn-system get settings.longhorn.io data-engine-cpu-mask -o jsonpath='{.value}{"\n"}'

The value may be data-engine-specific. For example:

{"v2":"0x3"}

Some Longhorn versions support CPU-list input for this setting. If the value is already a CPU list, use that list directly in the checks below.

Convert the mask to CPU IDs:

python3 - <<'PY'
mask = int("0x3", 0)
print(",".join(str(cpu) for cpu in range(mask.bit_length()) if mask & (1 << cpu)))
PY

For 0x3, the output is:

0,1

3. Check SPDK Placement​

SPDK_PIDS=$(pgrep -f '[s]pdk_tgt' || true)

for pid in ${SPDK_PIDS}; do
  ps -T -p "${pid}" -o pid,tid,psr,stat,pcpu,comm,args
done

This command matches the full command line because the SPDK process or its threads may appear as reactor_<cpu> in the kernel process name. In that case, pgrep -x spdk_tgt can return empty even when spdk_tgt is running.

Example output for the default CPU mask 0x3:

    PID     TID PSR STAT %CPU COMMAND         COMMAND
2913161 2913161   0 RLl  99.9 reactor_0       spdk_tgt -L all --mem-size 2048 -m 0x3
2913161 2913166  21 SLl   0.0 dpdk-intr       spdk_tgt -L all --mem-size 2048 -m 0x3
2913161 2913220   1 RLl  99.9 reactor_1       spdk_tgt -L all --mem-size 2048 -m 0x3

The PSR column shows the CPU that each thread is running on. In the example above, reactor_0 is running on CPU 0 and reactor_1 is running on CPU 1, so the SPDK CPU list is 0,1.

To check whether running or blocked kernel workers are currently on the SPDK CPUs, use the following more targeted command. Replace SPDK_CPUS with the comma-separated CPU IDs from the reactor_* rows.

SPDK_CPUS="0,1"

ps -eLo pid,tid,psr,stat,pcpu,wchan:30,comm,args | \
  awk -v cpus="${SPDK_CPUS}" '
    BEGIN {
      printf "%7s %7s %3s %-5s %5s %-30s %-16s %s\n", "PID", "TID", "PSR", "STAT", "%CPU", "WCHAN", "COMMAND", "ARGS"
      split(cpus, cpu_list, ",")
      for (i in cpu_list) {
        spdk_cpu[cpu_list[i]] = 1
      }
    }
    NR > 1 && spdk_cpu[$3] && $4 ~ /[RD]/ && ($7 ~ /^kworker\// || $7 ~ /^ksoftirqd\//) {
      print
    }
  '

On nodes that are not affected, this command normally prints only the header line. If the header line is removed from the command, empty output is expected. This means there are no running or uninterruptible kworker/* or ksoftirqd/* threads on the SPDK CPUs at that moment.

Example output showing kernel work on an SPDK CPU:

    PID     TID PSR STAT   %CPU WCHAN                          COMMAND          ARGS
     17      17   0 R     0.0 -                              ksoftirqd/0     [ksoftirqd/0]
  32008   32008   0 R     0.0 -                              kworker/0:0+eve [kworker/0:0+events]
2935011 2935011   0 D     0.0 mgag200_ddc_algo_bit_data_pre_ kworker/0:2+eve [kworker/0:2+events]
4012518 4012518   0 R     0.0 -                              kworker/0:3+mm_ [kworker/0:3+mm_percpu_wq]

In this example, the PSR value is 0, and the process names are ksoftirqd/0 and kworker/0:*. This means kernel softirq and workqueue threads are running or blocked on CPU 0.

If the SPDK reactor threads are running on the CPUs reported in the workqueue lockup, or if the targeted command shows kworker/* or ksoftirqd/* activity on the SPDK CPUs, continue with the IRQ and workqueue checks.

4. Check IRQ Affinity​

The following example checks CPUs 0 and 1 because the reactor_* rows above are running on those CPUs. Replace SPDK_CPUS with the CPU IDs used in your environment.

SPDK_CPUS="0,1"

cpu_list_overlaps() {
  python3 - "$1" "$2" <<'PY'
import sys

target = {int(cpu) for cpu in sys.argv[1].split(",") if cpu}
seen = set()

for part in sys.argv[2].split(","):
    part = part.strip()
    if not part:
        continue
    if "-" in part:
        start, end = map(int, part.split("-", 1))
        seen.update(range(start, end + 1))
    else:
        seen.add(int(part))

sys.exit(0 if target & seen else 1)
PY
}

for irqdir in /proc/irq/[0-9]*; do
  irq=${irqdir##*/}
  eff=$(cat "${irqdir}/effective_affinity_list" 2>/dev/null || true)
  conf=$(cat "${irqdir}/smp_affinity_list" 2>/dev/null || true)

  if cpu_list_overlaps "${SPDK_CPUS}" "${eff}"; then
    echo "IRQ=${irq} configured=${conf} effective=${eff}"
    grep -w "^ *${irq}:" /proc/interrupts 2>/dev/null || true
  fi
done

Example output:

IRQ=108 configured=0-5,12-17 effective=0
 108:   25762242          0 ... IR-PCI-MSIX-0000:04:00.0    3-edge      netboot-TxRx-3
IRQ=109 configured=0-5,12-17 effective=1
 109:          0   15555565 ... IR-PCI-MSIX-0000:04:00.0    4-edge      netboot-TxRx-4
IRQ=120 configured=0-5,12-17 effective=0
 120:   10898692          0 ... IR-PCI-MSIX-0000:04:00.0   15-edge      netboot-TxRx-15
IRQ=121 configured=0-5,12-17 effective=1
 121:          0    8135324 ... IR-PCI-MSIX-0000:04:00.0   16-edge      netboot-TxRx-16
IRQ=92 configured=0 effective=0
  92:    2242087          0 ... IR-PCI-MSIX-0000:08:00.0   13-edge      nvme0q13
IRQ=93 configured=1 effective=1
  93:          0     448454 ... IR-PCI-MSIX-0000:08:00.0   14-edge      nvme0q14

The effective value shows where the IRQ is actually running. In this example, NIC queues and NVMe queues are effectively landing on CPUs 0 and 1, which are the default SPDK reactor CPUs for mask 0x3. If device IRQs, especially high-traffic NIC or storage IRQs, are effectively landing on SPDK CPUs, the node is at risk.

5. Check Kernel Workqueue Masks​

cat /sys/devices/virtual/workqueue/cpumask

for f in /sys/devices/virtual/workqueue/*/cpumask; do
  echo "${f}: $(cat "${f}")"
done

Example output:

/sys/devices/virtual/workqueue/blkcg_punt_bio/cpumask: ffffff
/sys/devices/virtual/workqueue/ib-comp-unb-wq/cpumask: ffffff
/sys/devices/virtual/workqueue/iscsi_conn_cleanup/cpumask: ffffff
/sys/devices/virtual/workqueue/nvme-auth-wq/cpumask: ffffff
/sys/devices/virtual/workqueue/nvme-delete-wq/cpumask: ffffff
/sys/devices/virtual/workqueue/nvme-reset-wq/cpumask: ffffff
/sys/devices/virtual/workqueue/nvme-wq/cpumask: ffffff
/sys/devices/virtual/workqueue/scsi_tmf_0/cpumask: ffffff
/sys/devices/virtual/workqueue/writeback/cpumask: ffffff

On a 24-CPU node, ffffff means the workqueue can run on CPUs 0-23. If SPDK uses the default CPU mask 0x3, CPUs 0 and 1 are included in this workqueue mask. If the global or per-workqueue masks include the SPDK CPUs, unbound kernel work may still run on the SPDK CPUs.

Preferred Risk-Reduction Setting​

Longhorn added the data-engine-cpu-isolation-enabled setting to reduce the chance of this issue. When enabled for the V2 Data Engine, the Longhorn V2 instance-manager:

  • Persists the SPDK CPU mask under /var/lib/longhorn/instance-manager/v2/spdk_cpu_mask on the host.
  • Programs /proc/irq/*/smp_affinity to the inverse of the SPDK CPU mask.
  • Writes the same inverse mask to /sys/devices/virtual/workqueue/cpumask.
  • Updates per-workqueue CPU masks when possible.
  • Reconciles stale affinity state on the next instance-manager restart if the setting is later disabled.

This setting steers IRQs and unbound workqueues away from SPDK CPUs. In some environments, this may stop the workqueue lockup messages entirely. In others, it may only reduce how often they occur because CPU-bound or per-CPU kernel workers such as kworker/0:* are tied to a specific CPU and cannot be moved by the unbound workqueue CPU mask.

Use this setting when the Longhorn version bundled with Harvester includes it. If the Longhorn setting exists but Harvester does not expose it in the UI, you can still configure it through the Longhorn setting resource. For Harvester versions earlier than v1.9.0, the bundled Longhorn version does not include the setting, so use the manual workaround in the next section.

caution

The Longhorn setting is a danger-zone setting. It changes host-wide IRQ and workqueue affinity, takes effect only after the V2 instance-manager pod is recreated, and Longhorn refuses to apply the change while V2 volumes are attached. Stop workloads that use Longhorn V2 volumes and detach those volumes before changing the setting.

If the setting exists in your Longhorn version, you can check it with:

kubectl -n longhorn-system get settings.longhorn.io data-engine-cpu-isolation-enabled

Enable it for the V2 Data Engine:

kubectl -n longhorn-system patch settings.longhorn.io data-engine-cpu-isolation-enabled \
  --type=merge \
  -p '{"value":"{\"v2\":\"true\"}"}'

Then wait for the V2 instance-manager pods to be recreated after all V2 volumes are detached.

Verify the instance-manager log:

kubectl -n longhorn-system logs <v2-instance-manager-pod> | \
  grep -E "Setting IRQ affinity|Setting workqueue cpumask|Applied IRQ affinity|Applied global workqueue cpumask"

Example expected messages:

Setting IRQ affinity to exclude SPDK CPUs
Applied IRQ affinity mask
Setting workqueue cpumask to exclude SPDK CPUs
Applied global workqueue cpumask
Applied per-workqueue cpumask

Manual Workaround for Versions Without the Longhorn Setting​

Use this workaround for Harvester versions earlier than v1.9.0, where the bundled Longhorn version does not include data-engine-cpu-isolation-enabled. For later versions, first check whether the Longhorn setting exists and prefer the setting-based risk-reduction path when possible.

The goal is to move IRQs and unbound workqueues away from the SPDK CPUs. For example, if SPDK uses the default CPUs 0 and 1, IRQs and unbound workqueues should use CPU 2 through the last online CPU.

This workaround is a risk-reduction step, not a guaranteed fix for every workqueue lockup. CPU-bound or per-CPU kernel workers can still run on the SPDK CPUs because they are tied to those CPUs by the kernel. The expected result is that the node remains stable and any remaining workqueue stalls recover quickly instead of hanging for a long time.

Repeat this workaround whenever the Longhorn V2 CPU mask changes. If CPU allocation is managed dynamically, the IRQ and workqueue masks must be recalculated after each placement change.

1. Identify the Non-SPDK CPUs and Mask​

The default Longhorn V2 CPU mask is 0x3, so SPDK uses CPUs 0 and 1.

On the example node, all online CPUs are 0-23:

cat /sys/devices/system/cpu/online
0-23

Use the following values for this default example:

ItemValue
SPDK CPUs0,1
Non-SPDK CPU list2-23
Linux affinity maskfffffc

The Linux affinity mask is a CPU bitmap written in hexadecimal. CPU 0 is bit 0, CPU 1 is bit 1, and so on. On a 24-CPU node, all CPUs enabled is ffffff. Excluding CPUs 0 and 1 clears the lowest two bits, so the mask becomes fffffc. Leading zeros are optional, so fffffc and 00fffffc are equivalent.

Do not copy these values blindly if your Longhorn V2 CPU mask or online CPU list is different. The non-SPDK CPU list must be all online CPUs except the CPUs used by the SPDK reactor threads.

2. Apply IRQ Affinity at Runtime​

Set NON_SPDK_AFFINITY_MASK to the Linux affinity mask from the previous step.

NON_SPDK_AFFINITY_MASK=fffffc

echo "${NON_SPDK_AFFINITY_MASK}" > /proc/irq/default_smp_affinity

for f in /proc/irq/[0-9]*/smp_affinity; do
  echo "${NON_SPDK_AFFINITY_MASK}" > "${f}" 2>/dev/null || true
done

Some IRQs may reject affinity updates because they are managed by the kernel. This is expected. Always verify effective_affinity_list after applying the change.

Changing IRQ affinity alone may not make existing workqueue lockup messages disappear. Kernel workqueues can still run on the SPDK CPUs until the workqueue CPU masks are updated, and workers that are already stuck may continue to be reported by the kernel. Apply the workqueue affinity change as well, and then verify whether new lockup messages stop appearing.

These runtime IRQ affinity changes are not persistent across reboot. Use the persistence step later in this section if the runtime change mitigates the issue.

3. Apply Workqueue Affinity at Runtime​

NON_SPDK_AFFINITY_MASK=fffffc

echo "${NON_SPDK_AFFINITY_MASK}" > /sys/devices/virtual/workqueue/cpumask

for f in /sys/devices/virtual/workqueue/*/cpumask; do
  echo "${NON_SPDK_AFFINITY_MASK}" > "${f}" 2>/dev/null || true
done

These runtime workqueue affinity changes are not persistent across reboot. Use the persistence step later in this section if the runtime change mitigates the issue.

4. Restart the V2 Instance-Manager Pod​

After changing both IRQ and workqueue affinity, restart the Longhorn V2 instance-manager pod on the affected node. This is required because the existing spdk_tgt process can keep the SPDK reactor threads on the busy CPUs, and already-stuck CPU-bound workers may continue to be reported by the kernel until spdk_tgt is recreated.

caution

Detach all Longhorn V2 volumes attached to the affected node before restarting the V2 instance-manager pod. You do not need to detach V2 volumes attached to other nodes. Restarting an instance-manager while V2 volumes are still attached to the affected node can interrupt storage I/O and affect running VMs.

Check the Longhorn V2 volumes first:

kubectl -n longhorn-system get volumes.longhorn.io \
  -o custom-columns=NAME:.metadata.name,DATAENGINE:.spec.dataEngine,STATE:.status.state,NODE:.status.currentNodeID

Only continue after all v2 volumes whose NODE is the affected node are detached.

Find the V2 instance-manager pod on the affected node:

AFFECTED_NODE=hp-114-tink-system

kubectl -n longhorn-system get pods \
  -l longhorn.io/component=instance-manager,longhorn.io/data-engine=v2 \
  --field-selector spec.nodeName="${AFFECTED_NODE}" \
  -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName

Restart the V2 instance-manager pod:

V2_IM_POD=<v2-instance-manager-pod-name>

kubectl -n longhorn-system delete pod "${V2_IM_POD}"

5. Make IRQ and Workqueue Affinity Persistent​

The previous IRQ and workqueue affinity commands only change runtime procfs and sysfs state, so they are lost after reboot. If those runtime changes reduce or stop new lockup messages, use a Harvester CloudInit resource to reapply both IRQ and workqueue affinity after reboot. This follows the same model as the Longhorn CPU isolation setting, which updates host IRQ and workqueue affinity at runtime.

The following manifest is a Harvester CloudInit resource, not a file that you manually place on each node. Save it on a machine with kubectl access to the Harvester cluster and apply it with kubectl apply -f <file-name>.yaml. The Harvester controller writes the content to /oem/99_longhorn_v2_cpu_affinity.yaml on each node matched by matchSelector.

You can add this resource after the cluster is already installed. The file is synchronized to the matched nodes after the resource is applied, but the cloud-init commands take effect only after those nodes are rebooted. After reboot, the commands are applied again on each boot.

Example CloudInit resource for the default SPDK CPUs 0 and 1 on a 24-CPU node:

apiVersion: node.harvesterhci.io/v1beta1
kind: CloudInit
metadata:
  name: longhorn-v2-cpu-affinity
spec:
  matchSelector:
    kubernetes.io/hostname: "hp-114-tink-system"
  filename: 99_longhorn_v2_cpu_affinity.yaml
  contents: |
    stages:
      network:
        - commands:
            - echo fffffc > /proc/irq/default_smp_affinity
            - for f in /proc/irq/[0-9]*/smp_affinity; do echo fffffc > "${f}" 2>/dev/null || true; done
            - echo fffffc > /sys/devices/virtual/workqueue/cpumask
            - for f in /sys/devices/virtual/workqueue/*/cpumask; do echo fffffc > "${f}" 2>/dev/null || true; done

Adjust the affinity mask and matchSelector for your environment. To target a different affected node, replace hp-114-tink-system with that node's kubernetes.io/hostname label value. Do not use matchSelector: {} unless every node should receive this workaround. This workaround is not dynamically reconciled. If the Longhorn V2 CPU mask changes, update the CloudInit resource and reboot the affected nodes.

After configuring the CloudInit resource, reboot the affected nodes for the commands to take effect. To apply the workaround immediately before reboot, run the runtime commands in the previous steps.

If Lockups Still Appear​

The IRQ and workqueue affinity changes reduce the chance of a workqueue lockup, but they may not eliminate it in every environment. The workqueue CPU mask mainly controls unbound workqueues. Bound or per-CPU workqueues can still run on their associated CPU, so kernel workers such as kworker/20:* may still appear on an SPDK CPU even after unbound workqueues are moved away.

If the node still reports occasional workqueue lockups after the IRQ and workqueue affinity mitigation is applied, use the following additional mitigations.

1. Change the Longhorn V2 Disk Driver to auto​

If the affected Longhorn V2 disk is using the aio disk driver, consider changing the requested disk driver to auto. For an NVMe disk, verify after reprovisioning that Longhorn reports the actual disk driver as nvme in the Longhorn Node custom resource. In recent validation, after applying IRQ affinity and confirming that the actual disk driver was nvme, no new workqueue lockup messages were observed during the test window.

caution

Only use the SPDK NVMe disk driver when the NVMe device satisfies the Longhorn V2 IOMMU group isolation requirement. Longhorn uses vfio-pci for the SPDK NVMe path, and VFIO must claim the whole IOMMU group. If the NVMe device shares an IOMMU group with a PCIe bridge or another device that cannot be bound to VFIO, Longhorn cannot use the SPDK NVMe driver for that disk and the disk must stay on the aio driver. For details, see the Longhorn V2 Data Engine requirements.

Before changing the disk driver, remove the affected disk from Harvester. Follow the Remove Disks guide, and make sure the disk no longer contains active Longhorn replicas or backing images.

Find the BlockDevice resource for the disk:

kubectl -n longhorn-system get blockdevices.harvesterhci.io \
  -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,PROVISION:.spec.provision,PHASE:.status.provisionPhase,STATE:.status.state,ENGINE:.spec.provisioner.longhorn.engineVersion,DRIVER:.spec.provisioner.longhorn.diskDriver

After the disk is removed and the BlockDevice is no longer provisioned, patch the disk driver:

BLOCKDEVICE=<blockdevice-name>

kubectl -n longhorn-system patch blockdevice.harvesterhci.io "${BLOCKDEVICE}" \
  --type=merge \
  -p '{"spec":{"provisioner":{"longhorn":{"engineVersion":"LonghornV2","diskDriver":"auto"}}}}'

Provision the disk again from the Harvester UI. Harvester will add the disk back to Longhorn V2. Then verify the requested driver in the BlockDevice resource and the actual driver in the Longhorn Node resource.

Verify the BlockDevice after provisioning:

kubectl -n longhorn-system get blockdevice.harvesterhci.io "${BLOCKDEVICE}" -o yaml

The matching BlockDevice should show:

spec:
  provision: true
  provisioner:
    longhorn:
      engineVersion: LonghornV2
      diskDriver: auto
status:
  provisionPhase: Provisioned
  state: Active

Also check the Longhorn Node custom resource to confirm the actual disk driver selected by Longhorn:

kubectl -n longhorn-system get nodes.longhorn.io \
  -o go-template='{{printf "NODE\tDISK\tTYPE\tPATH\tSPEC_DRIVER\tSTATUS_DRIVER\n"}}{{range .items}}{{ $node := . }}{{range $diskName, $disk := .spec.disks}}{{ $status := index $node.status.diskStatus $diskName }}{{printf "%s\t%s\t%s\t%s\t%s\t%s\n" $node.metadata.name $diskName $disk.diskType $disk.path $disk.diskDriver $status.diskDriver}}{{end}}{{end}}'

For the affected disk, SPEC_DRIVER should be auto, and STATUS_DRIVER should be nvme:

NODE                  DISK                                  TYPE    PATH          SPEC_DRIVER   STATUS_DRIVER
hp-161-tink-system    32f43222-1eb1-4ab6-9e65-c4f8ddad700d  block   0000:08:00.0  auto          nvme

The spec.disks.<blockdevice-name>.diskDriver value is the requested driver mode from Harvester. The status.diskStatus.<blockdevice-name>.diskDriver value is the driver Longhorn actually uses. For an NVMe disk, the status value should be nvme.

2. Enable Longhorn V2 Interrupt Mode​

If the lockup still occurs after changing the disk driver and applying IRQ affinity, consider enabling Longhorn V2 interrupt mode.

caution

Longhorn V2 interrupt mode should still be treated as experimental. It changes the SPDK execution model from continuous polling to interrupt-driven handling, which can reduce constant CPU pressure but may affect latency and performance. Longhorn also requires all V2 volumes to be detached before this setting can be changed.

Check the setting:

kubectl -n longhorn-system get settings.longhorn.io data-engine-interrupt-mode-enabled

Enable interrupt mode for the V2 Data Engine:

kubectl -n longhorn-system patch settings.longhorn.io data-engine-interrupt-mode-enabled \
  --type=merge \
  -p '{"value":"{\"v2\":\"true\"}"}'

Wait for the V2 instance-manager pods to be recreated after all V2 volumes are detached.

Verification​

After applying the mitigation, verify the items that directly confirm the node recovered and the settings are still applied.

  1. Kernel logs no longer report new workqueue lockups, or any remaining reports are short and do not keep increasing for a long time.

    journalctl -k --since "30 minutes ago" | grep -E "BUG: workqueue lockup|soft lockup" || true

    If occasional messages still appear, compare the stuck for <seconds>s value over time. The mitigation is still useful if the stuck time stops growing, the node remains responsive, and RKE2, kubelet, and Longhorn keep making progress.

  2. The Longhorn V2 instance-manager pod is running on the affected node.

    AFFECTED_NODE=hp-114-tink-system

    kubectl -n longhorn-system get pods \
      -l longhorn.io/component=instance-manager,longhorn.io/data-engine=v2 \
      --field-selector spec.nodeName="${AFFECTED_NODE}"

  3. If you configured the persistent workaround and rebooted the node, the IRQ and workqueue masks are still applied.

    cat /proc/irq/default_smp_affinity
    cat /sys/devices/virtual/workqueue/cpumask

References​