To register and login, use your Google, Twitter, Facebook, LinkedIn, or OpenID credentials.

This is allowing us to stop most spam registrations. We've deleted most of the spam accounts that got through, and we're closely watching for more.

Audio batch processing on Android

NitroooNitrooo Posts: 21
edited January 2013 in Pd Everywhere

Hi all,
I'm a general programmer, currently focusing on Android mobile development.
I'd like to realize an app that records a short voice message and lets you hear it back with different distorsion effects applied on.
I just need classic filters (reverb, multiband compression, pitch shifting...), but since I have no theory background on signal processing, I've been looking for a library with prebuilt effects or a way to design them in pure java in a relatively easy way (I mean, without reading 30 books on DSP).
Now, why I'm writing this post? Well, in my search for such library I came across pd and I found it amazing, especially because I'm fascinated by generative software and art and I see great potential in that sense, maybe in future projects. I've read most of this book http://www.pd-tutorial.com and other tutorials and played around with pd. So I decided to figure a solution for my problem through pd even if it doesn't fit my needs so well because I essentially need non-realtime batch processing. On the other hand it looks like I would be able to find all the filters I need, ready to use, apart from little adjustments.
So, that's what I need to do:

Android mic -> original sound in array -> filter chain 1..n -> "wet" sound 1..n in array_1..n

Then the user of the app would be able to play each of the processed sounds by tapping different buttons and save them to disk.

Currently, my only concern is about processing speed. Since, as far as I learnt in the past days, pd is intended for realtime processing, its speed is limited by the sampling rate, but I want to process the original sound as fast as possible, I want CPU speed to be the limit. I've read on another forum about upsampling to speed up things, but it looks like it could introduce glitches, jumps and may require tweaks. It seems a bit dirty, plus it's not completely clear to me yet.
What do you think about that? Do you think I can make an "instantaneous" processing in an effective way using libpd and pd patches on a mobile device... or maybe it's a naive idea?

I'm sorry for the length of my post and for my English.
Ideas/suggestions are welcome!
Thanks in advance!

Answers

  • pbrinkmannpbrinkmann Posts: 686 ✭✭

    Yes, you can use libpd for batch processing. It's actually simpler than the real-time approach because you don't need any of the Android-specific components. PdBase is all you need; look at the process family of methods in particular. Don't hesitate to ask if you have further questions.

  • NitroooNitrooo Posts: 21

    Thanks Peter for your answer and for your excellent job with libpd!

    So, I've briefly taken a look at the process methods in libpd.
    Let's say that I have a 10 seconds input sound loaded in an array and a process call in a loop that passes all the chunks of my input array. If I've understood correctly, the loop would end in less than 10 seconds (for example, half a second), assuming that I have enough CPU speed, with no upsampling or other tricks. In other words, there's no timing-awareness in the process methods.

    Is this correct?

  • pbrinkmannpbrinkmann Posts: 686 ✭✭

    That's right. As a matter of fact, the one and only design goal of libpd was to expose a process method without awareness of time. Everything else came later :)

  • NitroooNitrooo Posts: 21

    Encouraged by your words I've started to mess around with the PdBase.process method.
    And... it works! But now I'm having troubles with receiving messages from pd.

    This is the core of the code I'm using: http://pastebin.com/NDLcz9vS

    And here's a simple test patch: http://pastebin.com/DxLvP4ZL

    Using the same code and another patch I made, I've been able to load a .wav from disk, fill an array with it, copy that array in another one and save it to another .wav file. So, the sendBang and the process calls work, because at some point I get a .wav output file on my device. I just can't get the listener part to work, the callback is never called and then the loop never ends.

    What's wrong? Am I missing something?
    In general, are there better ways to synchronize pd and user code while using the PdBase.process methods?

  • pbrinkmannpbrinkmann Posts: 686 ✭✭

    You won't receive messages unless you poll the message queue. Just add PdBase.pollPdMessageQueue(); after each process call.

    Also, while your approach certainly works, your life may be easier if you do a little more work in Java and less work in Pd. Specifically, you may want to read your input file into an array in Java (make sure the array size is a multiple of 64), create an output array of the same size, process everything with one process call for the appropriate number of ticks, then write the output array to a file. The Pd patch will be much simpler, just using [adc~] for input and [dac~] for output.

  • pbrinkmannpbrinkmann Posts: 686 ✭✭

    Another thing, unrelated to libpd but it still caught my eye: If you end up doing a lot of batch processing in Pd, you may want to do that in a separate thread so your app won't become unresponsive.

  • NitroooNitrooo Posts: 21

    Thanks for your precious help, it worked! Yes, your approach is faster and more desirable. Actually I was just experimenting with these new things I'm learning these days, pd and libpd.

    Following your advice, I've come up with this code: http://pastebin.com/SDrZ9YcL

    And this patch: http://pastebin.com/Cq9Ws2eP

    So, the input buffer inFrames is correctly filled with the data read from the input .wav file (-1 to 1 double values), but after the PdBase.process call, the output buffer is still filled with zeroes. However, If I connect the noise~ object to the two inlets of the dac~, I correctly get an output .wav file with the noise, so the dac~ is working. The input .wav is stereo, so two channels, like the objects in the patch. Why the adc~ isn't receiving the input buffer?

  • pbrinkmannpbrinkmann Posts: 686 ✭✭

    You aren't getting any input because you've chosen zero input channels in the openAudio call. Also, if you're working in stereo, then you need to pay attention to the formatting of your buffers. The process methods (except processRaw) expect channels to be interleaved. I don't know what format you get from your wav reader class, but if the output sounds wrong, it'll probably be due to buffer formats.

  • NitroooNitrooo Posts: 21

    Whoops, I totally missed it! Yes the input buffer was correctly interleaved. Now it's working... I definitely need more sleep. :D

    Thanks again! :)

  • This is very useful as I'm trying to do something a little bit similar. Naively I just tried to connect the adc~ to a writesf~ to record a wav file.

    I was curious as to why my simplistic idea works fine on a Mac but not on Android? I expected writesf~ to automagically handle any buffering required.

    Also in the code, where is the class WavFile imported from? I found the following WavFile libary and wondered if its the one you used. http://www.labbookpages.co.uk/audio/javaWavFiles.html

  • NitroooNitrooo Posts: 21

    Yes, I'm using the WavFile class found on that website. It works fine.

    As for writesf~, I think it should work on your Android too. Can you post some code?
    This is a patch I tested working on my device that do the task I mentioned in a previous post: http://pastebin.com/22914mAD

    Nothing special, just playing around and experimenting. One thing I noticed: if you want to write a wave file using soundfiler and your input array is already filled, you don't even have to call the PdBase.process method, just send a message, it will work. It makes sense, since there's no data flow to push on through thick cables. Differently, if you're using writesf~ you need to advance the processing by the appropriate number of ticks.

  • Thanks for pointing that out, I haven't yet read about sound filer. It might be more useful for batch processing.

  • I've now got an interesting problem, I think I have a solution for it but it would be interesting to compare notes.

    I want to use the batch processing to take a series of wav files and cross-fade them together. Say each file is 10 seconds long, I'm going to cross fade the last second. I only need to batch process the overlapping parts of the files, the remainder I can just pass through to my file writer.

    I see that the batch api processes an input buffer to an output buffer. In this case I've got a bit of a design quandary because I'm reading from two files to create one output file.

    Am I right to assume that a PdBase instance when its loaded a patch, then the patch maintains a persistent state between process invocations?

    If this is the case. my patch could contain two buffers (tables) A and B that get filled on paired invocations of calls to the batch process.

    On the first invocation I fill the first buffer (table) and the outputBuffer is not changed

    On the second invocation I fill the second buffer (table) which then triggers the cross fade using a vline to control the amplitude of each 'track', with the outputBuffer being filled with the mixed tracks

    I then write the output buffer to the output file

    The main other option I thought of is to create a single buffer containing data interleaved from the A and B input files, then I'd have to demux it in my patch before mixing the data. This seems to be doing work with little value as it has to mux the data before calling process, then demux the data in the patch itself.

    Performance wise, if there is a big overhead on making more PdBase process calls it might make sense to do the interleaving, but I suspect the first option is simpler.

    Any thoughts?

  • Just to say, it took me a little bit of thinking but now things are working great. I ended up interleaving a stereo buffer and passing this into PD to mix down into a mono output file. After a bit of work and re-thinking its all working beautifully, a testament to this excellent library. Most of the compute time is spent in the file I/O the libpd/PD processing is very fast.

  • pbrinkmannpbrinkmann Posts: 686 ✭✭

    Glad to hear it works. Thanks for the update!

Sign In or Register to comment.