We ran into the same issue on a production counting system (Hailo-8, hailo_pci 4.23.0). The trigger was the same pattern you describe: allocation churn in the inference process. For us it came from clients connecting to and disconnecting from an MJPEG/HTTP video stream served by that process.