I’ve been a long time deadmau5 fan. Last January I had the chance to see him perform at the House of Blues in Boston as part of his cube v3 tour. There’s a great behind the scenes video where he talks about about the software that powers the show -

The software is called TouchDesigner and is developed by Derivative. In May I played with Style Transfer and 3d Scanning, and I was curious if it would be possible to implement a custom TouchDesigner operator to perform style transfer on video inputs in real time.

TouchDesigner

There’s a great introduction series by Ben Voigt that walks through TouchDesigner fundamentals. In short, TouchDesigner is a software built for processing signals, generally audio or video. TouchDesigner projects (or networks) consist of operators (nodes) that output data to other operators. Signals are transformed with operators such as TOP, CHOP, or SOP, and then sent to outputs, like the screen of your laptop or the surface of deadmau5’s cube.

touchdesigner

TouchDesigner supports Custom Operators and has a repository of examples. Before jumping into writing a custom operator, I thought it would be best to check out other TouchDesigner projects and see if real time style transfer is a viable goal.

Real Time Style Transfer

A quick search yields exsstas’s repository “StyleTransfer-in-TD”, and this post on the community forums. Both seem to have issues with framerate; exsstas uses this repository for style transfer that mentions videos can be composed from merging stills.

To make life easy I tried zhangzhang1989’s PyTorch-Multi-Style-Transfer which includes an out of the box real-time demo using a webcam. To get the demo running I needed the dependencies opencv-python, torch, torchvision and torchfile.

Unfortunately, even running on my GPU (GTX 1060) I’m only able to manage ~5 - 10 frames/sec. On CPU, it takes several seconds to produce a single frame. Unfortunately, this project will stay on the shelf for the time being - maybe some future advances in style transfer research will bring it back to life. Regardless, TouchDesigner is a great piece of software and I hope to have more excuses to poke around.