Divide and Conquer in Neural Style Transfer for Video
Abstract: Neural Style Transfer is a class of neural algorithms designed to redraw a given image in the style of another image, traditionally a famous painting, while preserving the underlying details. Applying this process to a video requires stylizing each of its component frames, and the stylized frames must have temporal consistency between them to prevent flickering and other undesirable features. Current algorithms accommodate these constraints at the expense of speed.
In this video, I introduce an algorithm called Distributed Artistic Videos and demonstrate its capacity to produce stylized videos over ten times faster than the current state-of-the-art with no reduction in output quality. Through the use of an 8-node computing cluster, DAV reduces the average time required to stylize a video by 92\%---from hours to minutes---compared to the most recent algorithm of this kind on the same equipment and input. This allows the stylization of videos that are longer and higher-resolution than previously feasible.
A link to the latest version of this repository: https://github.com/pgalatic/thesis
A link to the code used to produce these results: https://github.com/pgalatic/thesis/tr...
My personal Github: https://github.com/pgalatic
CREDITS AND SOURCES
Music:
[ No Copyright ] Coffee House Jazz | Calm Jazz Music | Relaxing Music | Relax Music Meditation
• Video
The program used to make the Neural Style Transfer examples was an implementation of:
G. Ghiasi, H. Lee, M. Kudlur, V. Dumoulin, and J. Shlens,
"Exploring the structure ofa real-time, arbitrary neural artistic stylization network,"
CoRR, vol. abs/1705.06830,2017.
The implementation is called Magenta. https://github.com/tensorflow/magenta
Credit for VGG images:
https://neurohive.io/en/popular-netwo...
Credit for Gram Matrix images:
/ 775298068099002369
Other papers that were referenced:
L. A. Gatys, A. S. Ecker, and M. Bethge,
"A neural algorithm of artistic style,"
CoRR, vol. abs/1508.06576, 2015."
J. Johnson, A. Alahi, and L. Fei-Fei,
“Perceptual losses for real-time style transfer and super-resolution,”
in European Conference on Computer Vision, 2016.
M. Ruder, A. Dosovitskiy, and T. Brox,
“Artistic style transfer for videos and spherical images,”
International Journal of Computer Vision, vol. 126, pp. 1199–1219, Nov 2018.
online first.
N. Sundaram, T. Brox, and K. Keutzer,
“Dense point trajectories by gpu-accelerated large displacement optical flow,”
in European conference on computer vision, pp. 438–451, Springer, 2010.
K. He, X. Zhang, S. Ren, and J. Sun,
“Deep residual learning for image recognition,”
CoRR, vol. abs/1512.03385, 2015.
S. Gross and M. Wilber,
“Training and investigating residual nets.”
http://torch.ch/blog/2016/02/04/resne..., 2016.
F. Shen, S. Yan, and G. Zeng,
“Neural style transfer via meta networks,”
in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
This video was made with Shotcut.