perf: speed up the VM list with many VMs (#1189)

* perf: speed up the VM list with many VMs

The VM list scanned whole store lists for every row and re-evaluated the same values per row on every render and sort. With several hundred VMs this blocked the browser on sorting and on every VM update.

Performance:
- look up launcher pods and migrations through shared indexes (namespace/name -> resource) instead of scanning per row; the indexes are computed per store list, so they stay in sync with in-place store updates
- use the same migration lookup (getLatestMigration) in the VM model and on the VM detail page
- look up PVCs, PVs, VMIs and restore backups by id
- merge VMs and VMIs with a Set
- evaluate the VM state once when sorting by state
- check the unschedulable condition before the pod based state checks
- cache feature flag results and per-schema/per-setting values that are the same for every row

Bug fixes:
- the launcher pod map was never rebuilt, because the store keeps the same array
- pods, volumes and migrations of a VM with the same name in another namespace could be picked up

The order of the volumes of a VM (vm.volumes) now follows the VM spec instead of the store order. Nothing depends on it.

Measured in a synthetic setup, 100 rows per page, sorting by state (Chrome, dev build):
- ~490 VMs, ~500 PVCs/PVs: 1.9 s -> 1.0 s
- ~490 VMs, 600 PVs, 200 migrations: 6.0 s -> 0.7 s, main thread blocked during VM updates 78% -> 32%
With few volumes the gain is much smaller (8-20%).

Related to: https://github.com/harvester/harvester/issues/9513

Signed-off-by: Volker Theile <vtheile@suse.com>

* fix: read VM data from the store of the resource instead of a fixed one

The migrations of a VMI and the migration badge of the state column were read from the hardcoded `harvester` store. In the Rancher explorer (embedded and standalone) the data is loaded into the `cluster` store, so a running migration was shown neither in the VM state nor as badge.

The VM model remembered the store of the current product in a module variable on first use and never updated it. The same model is used in the Harvester product (`harvester` store) and in the Rancher explorer (`cluster` store), and Rancher keeps the page loaded when going from the explorer of a Harvester cluster to Virtualization Management and into the Harvester product. The VM list then read the VMIs, launcher pods, volumes and restores from the wrong store, and running VMs were shown as Starting until the page was reloaded.

Read the data from the store the resource lives in (`this.$getters`, in the formatter the store of the current product). In the Harvester product this is still the `harvester` store, so nothing changes there.

Not covered: volume details like the lock icon of encrypted volumes. The explorer uses the generic Rancher PVC model, which does not know them, and settings based checks (feature flags, backup target) are not loaded there either.

Related to: https://github.com/harvester/harvester/issues/11882

Signed-off-by: Volker Theile <vtheile@suse.com>

---------

Signed-off-by: Volker Theile <vtheile@suse.com>
This commit is contained in:
Volker Theile
2026-10-14 13:31:09 +02:00
committed by GitHub
parent 71141b257a
commit 5406c08f76
14 changed files with 388 additions and 116 deletions
@@ -5,18 +5,15 @@ import { HCI } from '../../types';
// The Steve base model and the shell resource class pull in most of the dashboard, none of that is needed here
jest.mock('../harvester', () => ({ __esModule: true, default: class {} }));
jest.mock('@shell/plugins/dashboard-store/resource-class', () => ({ colorForState: (state) => (state === 'Running' ? 'text-success' : 'text-info') }));
jest.mock('@shell/plugins/dashboard-store/resource-class', () => ({
colorForState: (state) => (state === 'Running' ? 'text-success' : 'text-info'),
stateSort: (color, display) => `${ color.replace(/^text-/, '') } ${ display }`,
}));
const VM_ID = 'ns1/vm1';
/**
* Create a VM model instance without the full Steve model plumbing. Only the store getters that the
* VM model reads are mocked, backed by plain lists (like the real store, `all` returns the same array
* that is mutated in place on updates).
*
* The model reads its store either through `$rootGetters` (`harvester/...`) or through `$getters`, the getters of the
* store it lives in. Both are backed by the same data, so the tests do not depend on which of them the model uses.
*/
// Creates a VM model with mocked store getters. Like the real store, `all` returns the same array that is mutated in place.
// Reads from the global `$rootGetters` are recorded in `globalReads`.
function createVm({
vm = {}, vmi, pods = [], pvcs = [], restores = [], backups = [], claims = ['disk-pvc'], printableStatus
} = {}) {
@@ -33,12 +30,14 @@ function createVm({
byId: (type, id) => (byId[type] || lists[type] || []).find((x) => x.id === id),
schemaFor: () => undefined,
};
const rootGetters = {
'harvester-common/getFeatureEnabled': () => false,
currentProduct: { inStore: 'harvester' },
'harvester/all': getters.all,
'harvester/byId': getters.byId,
};
const globalReads = [];
const rootGetters = new Proxy({ 'harvester-common/getFeatureEnabled': () => false }, {
get(target, key) {
globalReads.push(key);
return target[key];
}
});
const obj = Object.create(VirtVm.prototype);
@@ -66,7 +65,9 @@ function createVm({
Object.defineProperty(obj, '$getters', { value: getters });
Object.defineProperty(obj, 't', { value: (k) => k });
return { vm: obj, lists };
return {
vm: obj, lists, globalReads
};
}
const readyVmi = (extra = {}) => ({
@@ -287,6 +288,24 @@ describe('class VirtVm', () => {
});
});
describe('stateSort', () => {
it('should be built from the same state as stateDisplay and stateColor', () => {
const { vm } = createVm({ vmi: readyVmi() });
expect(vm.stateSort).toBe(`${ vm.stateColor.replace(/^text-/, '') } ${ vm.stateDisplay }`);
});
it('should evaluate the state only once', () => {
const { vm } = createVm({ vmi: readyVmi() });
const spy = jest.spyOn(vm, 'actualState', 'get');
expect(vm.stateSort).toBe('success Running');
expect(spy).toHaveBeenCalledTimes(1);
spy.mockRestore();
});
});
describe('podResource', () => {
it('should find the launcher pod by owner name', () => {
const pod = { metadata: { namespace: 'ns1', ownerReferences: [{ name: 'vm1' }] } };
@@ -311,6 +330,62 @@ describe('class VirtVm', () => {
expect(vm.podResource.metadata.name).toBe('launcher-target');
});
it('should find pods that are added to the store later', () => {
const { vm, lists } = createVm({ vmi: readyVmi(), pods: [] });
expect(vm.podResource).toBeUndefined();
const pod = { metadata: { namespace: 'ns1', ownerReferences: [{ name: 'vm1' }] } };
lists[POD].push(pod);
expect(vm.podResource).toStrictEqual(pod);
});
it('should not return a pod of a VM with the same name in another namespace', () => {
const other = { metadata: { namespace: 'ns2', ownerReferences: [{ name: 'vm1' }] } };
const mine = { metadata: { namespace: 'ns1', ownerReferences: [{ name: 'vm1' }] } };
const { vm } = createVm({ vmi: readyVmi(), pods: [mine, other] });
expect(vm.podResource).toStrictEqual(mine);
});
it('should not return pods that were removed from the store', () => {
const pod = { metadata: { namespace: 'ns1', ownerReferences: [{ name: 'vm1' }] } };
const { vm, lists } = createVm({ vmi: readyVmi(), pods: [pod] });
expect(vm.podResource).toStrictEqual(pod);
lists[POD].splice(0, 1);
expect(vm.podResource).toBeUndefined();
});
});
describe('store', () => {
it('should read the data of the VM from the store the VM lives in, not from a fixed one', () => {
// The VM model is used with the `harvester` store (Harvester product) and the `cluster` store (Rancher explorer)
const pod = { metadata: { namespace: 'ns1', ownerReferences: [{ name: 'vm1' }] } };
const { vm, globalReads } = createVm({
vmi: readyVmi(),
pods: [pod],
pvcs: [pvc('disk-pvc', false)],
restores: [{ id: 'ns1/r1', isComplete: true }],
vm: {
metadata: {
name: 'vm1', namespace: 'ns1', annotations: { 'restore.harvesterhci.io/name': 'r1' }
}
},
});
expect(vm.vmi.id).toBe(VM_ID);
expect(vm.podResource).toStrictEqual(pod);
expect(vm.volumes).toHaveLength(1);
expect(vm.restoreResource.id).toBe('ns1/r1');
expect(vm.actualState).toBe('Running');
expect(globalReads.filter((key) => /^(harvester|cluster)\//.test(key))).toStrictEqual([]);
});
});
describe('isRestartRequired', () => {
@@ -24,14 +24,24 @@ function createVmi(migrations) {
name: 'vm1',
annotations: { [HCI_ANNOTATIONS.MIGRATION_STATE]: 'Migrating' },
},
$getters: { all: (type) => type === HCI.VMIM ? list : [] },
$rootGetters: { 'harvester/all': (type) => type === HCI.VMIM ? list : [] },
$getters: { all: (type) => type === HCI.VMIM ? list : [] },
});
return { vmi, list };
}
describe('class VirtVmInstance', () => {
it('should select the newest migration in the VMI namespace regardless of list order', () => {
const { vmi } = createVmi([
migration('ns1', '2026-01-03T00:00:00Z'),
migration('ns2', '2026-01-04T00:00:00Z', 'Failed'),
migration('ns1', '2026-01-01T00:00:00Z', 'Failed'),
]);
expect(vmi.vmimResource.metadata.creationTimestamp).toBe('2026-01-03T00:00:00Z');
expect(vmi.migrationState.status).toBe('Migrating');
});
it('should follow additions, timestamp changes, phase changes and removals in the same list', () => {
const { vmi, list } = createVmi([]);
@@ -54,4 +64,32 @@ describe('class VirtVmInstance', () => {
list.splice(0, 2);
expect(vmi.migrationState).toBeNull();
});
it('should follow a migration reassigned to another VMI or namespace', () => {
const { vmi, list } = createVmi([migration('ns1', '2026-01-01T00:00:00Z')]);
expect(vmi.migrationState.status).toBe('Migrating');
list[0].spec.vmiName = 'vm2';
expect(vmi.migrationState).toBeNull();
list[0].spec.vmiName = 'vm1';
list[0].metadata.namespace = 'ns2';
expect(vmi.migrationState).toBeNull();
});
it('should invalidate the index when the store replaces properties on an existing migration', () => {
const { vmi, list } = createVmi([migration('ns1', '2026-01-01T00:00:00Z')]);
expect(vmi.migrationState.status).toBe('Migrating');
const existing = list[0];
for (const key of Object.keys(existing)) {
delete existing[key];
}
Object.assign(existing, migration('ns2', '2026-01-02T00:00:00Z', 'Failed'));
expect(vmi.migrationState).toBeNull();
});
});
@@ -326,7 +326,7 @@ export default class HciPv extends HarvesterResource {
}
get relatedPV() {
return this.$rootGetters['harvester/all'](PV).find((pv) => pv.metadata?.name === this.spec?.volumeName);
return this.$rootGetters['harvester/byId'](PV, this.spec?.volumeName);
}
get volumeProvider() {
@@ -1,7 +1,7 @@
import { load } from 'js-yaml';
import { omitBy, pickBy } from 'lodash';
import { PRODUCT_NAME as HARVESTER_PRODUCT } from '../config/harvester';
import { colorForState } from '@shell/plugins/dashboard-store/resource-class';
import { colorForState, stateSort } from '@shell/plugins/dashboard-store/resource-class';
import { POD, NODE, PVC } from '@shell/config/types';
import { findBy } from '@shell/utils/array';
import { parseSi } from '@shell/utils/units';
@@ -14,6 +14,7 @@ import { BACKUP_TYPE } from '../config/types';
import { HCI } from '../types';
import HarvesterResource from './harvester';
import { getVmCPUMemoryValues } from '../utils/cpuMemory';
import { sharedComputed } from '../utils/shared-computed';
import { isBackupTargetSettingUnavailable } from '../utils/setting';
export const OFF = 'Off';
@@ -81,42 +82,48 @@ const VMIPhase = {
Unknown: 'Unknown'
};
let productInStore;
// Index of the pods (namespace/ownerName -> pod), shared by all rows. `podList` must be the store list, not a copy.
// If several pods have the same owner (e.g. while a VM migrates), the last one wins.
function getPodOwnerIndex(podList) {
return sharedComputed(podList, 'podOwnerIndex', (pods) => {
const map = new Map(); // namespace/ownerName -> pod
let _podOwnerMap = null;
let _podOwnerMapSource = null;
for (const pod of pods) {
const refName = pod.metadata?.ownerReferences?.[0]?.name;
function getPodByOwnerName(rootGetters, inStore, ownerName) {
const podList = rootGetters[`${ inStore }/all`](POD);
if (refName) {
map.set(`${ pod.metadata.namespace }/${ refName }`, pod);
}
}
return map;
});
}
// `getters` are the getters of the store of the VM (`this.$getters`)
function getPodByOwnerName(getters, namespace, ownerName) {
const podList = getters['all'](POD);
if (!Array.isArray(podList)) {
return undefined;
}
// if not equals (usually means the pod list has been updated), we need to rebuild the map, otherwise we can reuse the map for better performance
if (_podOwnerMapSource !== podList) {
_podOwnerMap = new Map(); // use Map to store ownerReference name and pod mapping
for (const pod of podList) {
const refName = pod.metadata?.ownerReferences?.[0]?.name;
if (refName) {
_podOwnerMap.set(refName, pod);
}
}
_podOwnerMapSource = podList;
}
return _podOwnerMap.get(ownerName);
return getPodOwnerIndex(podList).get(`${ namespace }/${ ownerName }`);
}
function getPvcsByNames(rootGetters, inStore, names) {
const pvcList = rootGetters[`${ inStore }/all`](PVC);
// A VM only uses PVCs of its own namespace, so look them up by id instead of scanning all PVCs
function getPvcsByNames(getters, namespace, names) {
const out = [];
if (!Array.isArray(pvcList)) {
return [];
for (const name of new Set(names)) {
const pvc = getters['byId'](PVC, `${ namespace }/${ name }`);
if (pvc) {
out.push(pvc);
}
}
const uniqueNames = new Set(names);
return pvcList.filter((pvc) => uniqueNames.has(pvc.metadata?.name));
return out;
}
const IgnoreMessages = ['pod has unbound immediate PersistentVolumeClaims'];
@@ -135,7 +142,8 @@ export default class VirtVm extends HarvesterResource {
clone.action = 'goToCloneVM';
}
const canCreateVMSSchedule = !!this.$getters?.['schemaFor']?.(HCI.SCHEDULE_VM_BACKUP)?.collectionMethods?.find((x) => ['post'].includes(x.toLowerCase()));
const scheduleSchema = this.$getters?.['schemaFor']?.(HCI.SCHEDULE_VM_BACKUP);
const canCreateVMSSchedule = !!scheduleSchema && sharedComputed(scheduleSchema, 'canPost', (schema) => !!schema.collectionMethods?.find((x) => ['post'].includes(x.toLowerCase())));
return [
{
@@ -288,14 +296,6 @@ export default class VirtVm extends HarvesterResource {
];
}
get productInStore() {
if (!productInStore) {
productInStore = this.$rootGetters['currentProduct'].inStore;
}
return productInStore;
}
applyDefaults(resources = this, realMode) {
const spec = {
runStrategy: 'RerunOnFailure',
@@ -729,14 +729,13 @@ export default class VirtVm extends HarvesterResource {
}
get podResource() {
const inStore = this.productInStore;
const vmiResource = this.$rootGetters[`${ inStore }/byId`](HCI.VMI, this.id);
const vmiResource = this.$getters['byId'](HCI.VMI, this.id);
if (!vmiResource?.metadata?.name) {
return undefined;
}
return getPodByOwnerName(this.$rootGetters, inStore, vmiResource.metadata.name);
return getPodByOwnerName(this.$getters, vmiResource.metadata.namespace, vmiResource.metadata.name);
}
get isPaused() {
@@ -764,8 +763,7 @@ export default class VirtVm extends HarvesterResource {
}
get nsResourceQuota() {
const inStore = this.productInStore;
const allResQuotas = this.$rootGetters[`${ inStore }/all`](HCI.RESOURCE_QUOTA);
const allResQuotas = this.$getters['all'](HCI.RESOURCE_QUOTA);
return allResQuotas.find( (RQ) => RQ.namespace === this.metadata.namespace);
}
@@ -775,15 +773,13 @@ export default class VirtVm extends HarvesterResource {
}
get vmi() {
const inStore = this.productInStore;
return this.$rootGetters[`${ inStore }/byId`](HCI.VMI, this.id);
return this.$getters['byId'](HCI.VMI, this.id);
}
get volumes() {
const volumeClaimNames = this.spec.template.spec.volumes?.map((v) => v.persistentVolumeClaim?.claimName).filter((v) => !!v) || [];
return getPvcsByNames(this.$rootGetters, this.productInStore, volumeClaimNames);
return getPvcsByNames(this.$getters, this.metadata.namespace, volumeClaimNames);
}
get lvmVolumes() {
@@ -807,13 +803,15 @@ export default class VirtVm extends HarvesterResource {
}
get encryptedVolumeType() {
if (!this.volumes || this.volumes.length === 0) {
const volumes = this.volumes;
if (!volumes || volumes.length === 0) {
return 'none';
}
if (this.volumes.every((vol) => vol.isEncrypted)) {
if (volumes.every((vol) => vol.isEncrypted)) {
return 'all';
} else if (this.volumes.some((vol) => vol.isEncrypted)) {
} else if (volumes.some((vol) => vol.isEncrypted)) {
return 'partial';
} else {
return 'none';
@@ -918,15 +916,14 @@ export default class VirtVm extends HarvesterResource {
}
get isUnschedulable() {
if (this.isStopping || this.isStarting) {
const condition = this.status?.conditions?.find((c) => c.reason === UNSCHEDULABLE);
// Check the cheap condition first
const condition = this.status?.conditions?.find((c) => c.reason === UNSCHEDULABLE);
if (!!condition) {
return {
status: UNSCHEDULABLE,
message: condition.message || 'VM is unschedulable',
};
}
if (condition && (this.isStopping || this.isStarting)) {
return {
status: UNSCHEDULABLE,
message: condition.message || 'VM is unschedulable',
};
}
return null;
@@ -971,16 +968,14 @@ export default class VirtVm extends HarvesterResource {
`metadata.annotations."${ HCI_ANNOTATIONS.RESTORE_NAME }"`
) }`;
const inStore = this.productInStore;
const res = this.$rootGetters[`${ inStore }/byId`](HCI.RESTORE, id);
const res = this.$getters['byId'](HCI.RESTORE, id);
if (res) {
const allBackups = this.$rootGetters[`${ inStore }/all`](HCI.BACKUP);
const backupId = `${ res.spec?.virtualMachineBackupNamespace }/${ res.spec?.virtualMachineBackupName }`;
const backup = this.$getters['byId'](HCI.BACKUP, backupId);
res.fromSnapshot = !!allBackups
.filter((b) => b.spec?.type !== BACKUP_TYPE.BACKUP)
.find((s) => s.id === `${ res.spec?.virtualMachineBackupNamespace }/${ res.spec?.virtualMachineBackupName }`);
// Snapshots and backups are told apart by spec.type
res.fromSnapshot = !!backup && backup.spec?.type !== BACKUP_TYPE.BACKUP;
}
return res;
@@ -1005,11 +1000,13 @@ export default class VirtVm extends HarvesterResource {
}
get restoreState() {
if (!this.restoreResource) {
const restore = this.restoreResource;
if (!restore) {
return true;
}
return this.restoreResource?.isComplete;
return restore.isComplete;
}
get actualState() {
@@ -1110,6 +1107,13 @@ export default class VirtVm extends HarvesterResource {
return colorForState(state);
}
// Evaluate actualState once, the base implementation does it in stateColor and stateDisplay
get stateSort() {
const state = this.actualState;
return stateSort(colorForState(state), state);
}
get networkIps() {
let networkData = '';
const out = [];
@@ -1160,7 +1164,6 @@ export default class VirtVm extends HarvesterResource {
get rootImageId() {
let imageId = '';
const inStore = this.productInStore;
const volumes = this.spec.template.spec.volumes || [];
@@ -1170,7 +1173,7 @@ export default class VirtVm extends HarvesterResource {
});
if (!isNoExistingVolume) {
const existingVolume = this.$rootGetters[`${ inStore }/byId`](PVC, `${ this.metadata.namespace }/${ firstVolumeName }`);
const existingVolume = this.$getters['byId'](PVC, `${ this.metadata.namespace }/${ firstVolumeName }`);
if (existingVolume) {
return existingVolume?.metadata?.annotations?.[
@@ -1366,7 +1369,12 @@ export default class VirtVm extends HarvesterResource {
get isBackupTargetUnavailable() {
const backupTargetSetting = this.$rootGetters['harvester/byId'](HCI.SETTING, 'backup-target');
return isBackupTargetSettingUnavailable(backupTargetSetting);
// sharedComputed needs an object as key
if (!backupTargetSetting) {
return isBackupTargetSettingUnavailable(backupTargetSetting);
}
return sharedComputed(backupTargetSetting, 'unavailable', isBackupTargetSettingUnavailable);
}
setInstanceLabels(val) {
@@ -1,6 +1,7 @@
import { colorForState } from '@shell/plugins/dashboard-store/resource-class';
import { NODE } from '@shell/config/types';
import { HCI as HCI_ANNOTATIONS } from '@pkg/harvester/config/labels-annotations';
import { getLatestMigration } from '@pkg/utils/vm';
import { HCI } from '../types';
import HarvesterResource from './harvester';
@@ -45,32 +46,25 @@ export default class VirtVmInstance extends HarvesterResource {
return colorForState(state);
}
// The newest migration of this VMI, an empty array if there is none
get vmimResource() {
const all = this.$rootGetters['harvester/all'](HCI.VMIM) || [];
const vmimList = all.filter((vmim) => vmim.spec?.vmiName === this.metadata?.name);
const all = this.$getters['all'](HCI.VMIM) || [];
if (vmimList.length === 0) {
return [];
}
vmimList.sort((a, b) => {
return a?.metadata?.creationTimestamp > b?.metadata?.creationTimestamp ? -1 : 1;
});
return vmimList[0];
return getLatestMigration(all, this.metadata?.namespace, this.metadata?.name) || [];
}
get migrationState() {
const state = this.metadata?.annotations?.[HCI_ANNOTATIONS.MIGRATION_STATE];
const migration = this.vmimResource;
if (this.vmimResource?.status?.phase === VMIPhase.Failed) {
if (migration?.status?.phase === VMIPhase.Failed) {
return {
type: 'migration',
status: VMIPhase.Failed
};
}
if (this.vmimResource?.status?.phase && state) {
if (migration?.status?.phase && state) {
return {
type: 'migration',
status: state