After a few days of bench testing across several different machines a few obscure bugs popped up. These cover race conditions for threading and possible driver leak mitigations for intel Gpu's.

Intel drivers seem to have an issue with reporting out of memory when there is clearly plenty left. Reducing down the draw distance and maybe switching thread mode to serene under the config options can help.

No issues with other driver manufacturers have been discovered.