Problem summary
CopyCat can't specify a GPU to use other than gpu 0 when training via command line/terminal
Customer reported version
nuke.15.1v4
Customer reported platform
rocky9
Steps to reproduce
1) Setup a basic CopyCat training Nuke script, like the one attached
2) On a multi-gpu machine, open terminal and run a command like the following:
<nukeInstall> --gpu 1 -X CopyCat1 -x <path/to/nuke/script>.nk
Expected behaviour
For the Nuke to start training the CopyCat node
Actual behaviour
Nuke fails to train and errors saying:
Exception caught processing model: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! (when checking argument for argument found_inf in method wrapper_CUDA___amp_foreach_non_finite_check_and_unscale_)
Workaround
Disable the CopyCat node's
useAutomaticMixedPrecision knob
This can be done by running the following code:
nuke.toNode('CopyCat1')['useAutomaticMixedPrecision'].setValue('Disabled')
Alternatively, changing how the machine sees the GPU can help with the following environment variable:
CUDA_VISIBLE_DEVICES Reproduced by support
This bug has been reproduced in:
Nuke 15.1v4 - Rocky 9.5
Nuke 15.1v1 - Rocky 9.5 -
regression
Unable to reproduce bug in:
Nuke 15.0v5 - Rocky 9.5
*haven't tested os platforms due to setup limitations
Earliest version tested
Nuke 15.0v5 - This issue doesn't appear in this version and has regressed