Working on a thing to sonify silent colour videos.
One [vcf~] per pixel with center frequency Hz determined by hue and Q determined by saturation (just using HSV for now, nothing fancy).
One 20ms delay line per column of pixels
Input to each column's filters is a weighted sum of nearby delay lines.
Output is panned to stereo according to column's position in space
Decidedly not realtime even with OpenMP parallelism on 16 CPU threads.
(though maybe if I ported it to OpenCL and used single precision on GPU and kept video resolution low...)
Smooth droney sounds.
Maybe I can make it richer with some [hilbert~] magic (given a sinusoid, hilbert~ outputs 2 sinusoids approximately 90degrees out of phase for "most" of the spectrum, allowing you to get a [phasor~] out of it, which you can then waveshape at will - I'm thinking Chebyshev polynomial recursion to get bandlimited sawtooth waves).
@mathr How do you have video information direct sound then exactly? Just by volume, xceeding certain thrshholds and such. Or a complete FFT analysis with an array of values and interpretations behind?
@jayrope currently it's downscaled to 160x90 pixels. then each pixel has a delay line, vcf, and sawtooth waveshaper in a feedback loop, with the delays being read from neighbouring pixels too (like a spatial blur). the colour of each pixel in HSV space controls the filter: hue controls pitch (scaled like mtof over two octave range), saturation*value controls Q. full details in the source code:
https://code.mathr.co.uk/rodney/tree/6faf3d9ebdb043c79ce38d90179fe8517bf6518a check sonify.cl for the DSP, sonify.c for the control code, and sonify.sh for how to use it
@mathr Ah, so that's not Puredata anymore. My C skills are next to zero. Shame on that. And thank you for the pointer.
@jayrope compare my https://code.mathr.co.uk/rodney/blob/6faf3d9ebdb043c79ce38d90179fe8517bf6518a:/sonify.cl#l28 vcf function with the inner loop of Pd's https://github.com/pure-data/pure-data/blob/eeef4ba9130d3182146927c37fa57d61bbff0f0b/src/d_osc.c#L386-L447 (Pd has a linear interpolated table lookup for cos() which is faster but less accurate; I just use the C math library which makes it simpler but slower - lines 415,419-433 of Pd's code are implementing something like `cos(cf)` and `sin(cf)`)
the process is extracting the core mathematics/algorithms, and hardcoding it in C without any of the wrapping API layers that make it useable from a patcher system
This is a brand new server run by the main developers of the project as a spin-off of mastodon.social It is not focused on any particular niche interest - everyone is welcome as long as you follow our code of conduct!