After some more improvements to the system, it now runs incredibly smoothly in multiple different setups. This particular scene uses 16 dynamic fullscreen samples with ray jittering. The effect also has support for downscaling and blurring for soft lighting.
Let me know what you think, I'm happy to try out new things.
Namey5.