ID 589187 - Convolve doesn't always fallback to CPU when running out of VRAM, instead displaying GPU (CUDA) errors and rendering black

Problem summary
Convolve doesn't always fallback to CPU when running out of VRAM, instead displaying GPU (CUDA) errors and rendering black
 
Errors like:

Convolve1: CUDA FFT error: CUFFT_INTERNAL_ERROR.Convolve1: CUDA Error: Unknown error code: 709Convolvel: Insufficient memory available for FFT on Quadro P4000.

 
Customer reported version
nuke.15.1v1
 
Customer reported platform
rocky8
 
Steps to reproduce
 
1) Monitor the amount of VRAM being used on your system

2) Run the attached vram_allocator.py Python script to use up around 90% of your system's VRAM (You may need to install PyOpenGL, numpy and PySide6 to run the script)

2b) If your VRAM isn't filled, you will need to adjust the number in line 55 of the Python file, and try re-running the script  until you see about 90% VRAM usage
 
3) Launch Nuke
 
4) Open the attached ConvolveVRAMBug.nk Nuke script
 
5) If you don't immediately see the issue, try changing frame, or change the project resolution, 
 
Expected behaviour
For the Convolve to gracefully fallback the to CPU, and still render
 
Actual behaviour
The Convolce node errors and renders black:

 
Older Nuke versions tend to get this error:

 
Workaround
Force the cache to refresh, by changing frame 
 
Reproduced by support
This bug has been reproduced in:
Nuke 15.1v3 - Rocky 9.3
Nuke 13.1v1 - Rocky 9.3
 
Unable to reproduce bug in:
Nuke 15.1v3 - Rocky 9.3
 
*Could not test on macOS or CentOS 7 as the Python script to fill VRAM didn't work
 
Earliest version tested
Nuke 13.1v1 - This issue appears to be in all tested versions of the product