To register and login, use your Google, Twitter, Facebook, LinkedIn, or OpenID credentials.

This is allowing us to stop most spam registrations. We've deleted most of the spam accounts that got through, and we're closely watching for more.

Noticeable crackles in my app with Nexus 4 (but not with my other devices)

freewheelnatfreewheelnat Posts: 19
edited February 2013 in Pd Everywhere

I'm the developer for MovSynth, an Android app that uses libpd, and I am working on an update. It is using the current version of libpd (I did an update a few days ago) and I have noticed issues with my (new) Nexus 4, but not with Sony S tablet and Samsung Galaxy S2. The issue is some quite noticeable crackles in the sound.

From the logcat for the Nexus 4: OpenSL options: {opensl.output_buffer_size=256, opensl.input_buffer_size=512}

For my other devices: OpenSL options: {opensl.output_buffer_size=1024, opensl.input_buffer_size=1024}

I tried to use the default parameters by not calling PdAudio.init(this) and I got this OpenSL options: {opensl.output_buffer_size=512, opensl.input_buffer_size=512}

This sounded noticeably better. So I am thinking the issue may be with the buffer size that is selected for the Nexus 4? Has anyone else come across this issue?

The other option could have been the sample size (as N4 is 48000 and other devices are 44100) but I tested my patch on my computer setting 48000 for sample size and it sounded fine, though I can't eliminate this issue 100% because the test may not have reproduced the exact same conditions.

Other test I have done is with "Circle of Fifths". If I touch many notes to play a chord in a row (thus "overloading" the app), the sound quality deteriorates quicker on the Nexus 4 than on the Galaxy S2. But I am aware this isn't a very scientific/accurate test...

Answers

  • pbrinkmannpbrinkmann Posts: 686 ✭✭

    Can you post your code somewhere? I'd like to try this on my own Nexus 4.

  • Sure, I'll email it to you. Do you want the source code (so you can modify things to debug) or is the apk enough?

  • NitroooNitrooo Posts: 21

    Could you please keep us publicly updated on this issue? I'm getting sporadic clicks in my Android app and I'm still trying to figure out what's the cause. It may be unrelated to this, but it's still interesting to know about. Thank you!

  • Sure, I'll update the thread if/when I find the reason for those crackles I get on the Nexus 4.

  • pbrinkmannpbrinkmann Posts: 686 ✭✭

    @Nitroo, please provide more details. In particular, what device/Android version are you using? When did you last update your copy of libpd?

  • NitroooNitrooo Posts: 21
    edited February 2013

    I'm testing the app on my Samsung GT-I9100 running Android 4.0.3. I updated the libpd source like... one week ago? Anyway I'm using just the PdBase class, so I'm taking care myself of the recording and playing code. I'm interested anyway on this possible relationship between buffer size and crackles (well, actually they most likely are related if the buffer size is too small).

    Oh, by the way, even if it's a bit OT: I made a small change to the pure-data C code, just a little addition to suit my app's needs, nothing related to signal processing. Recompiling the source with the latest NDK (r8d) I get at runtime a "cannot locate '__gnu_thumb1_case_uqi'" error very similar to this: https://dev.guardianproject.info/issues/229

    Forcing the toolchain version to 4.4.3 works fine and fixes the issue. Peter, have you faced the same problem before?

    Thanks, and sorry for having hijacked the thread. :ar!

  • pbrinkmannpbrinkmann Posts: 686 ✭✭

    @freewheelnat The garbage collector kicks in about twice a second and frees about 2M of memory every time, which takes around 50ms. That's a lot, and it seems like a likely explanation for the glitches you're experiencing. I haven't yet had a chance to look at your code, but chances are that you can fix it by using memory more defensively, e.g., by reusing objects. (Java gurus will tell you that that's bad practice, but this is one of the cases where Android development is quite different from regular Java development.)

  • pbrinkmannpbrinkmann Posts: 686 ✭✭

    @Nitrooo I'd have to see your code to find out what's causing your glitches. Do you also get glitches when you're using PdAudio? If PdAudio works for you but you don't want to use it, then you should look at the way it chooses its buffer sizes and copy that.

    About the NDK trouble, I'm still on r8, so I haven't yet encountered this.

  • @pbrinkman: yes, the gc kicks in a lot because of the camera preview frame. But this isn't the cause of the problem because it kicks as often on other devices (I have checked this) and the other devices do not have those crackles, and the Samsung Galaxy S2 is a device less powerful than the Nexus 4 so if the S2 can deal with that amount of GC, so should the N4. The problem I have is specific to Nexus 4, the app sounds fine on other devices. The only differences between the way the app runs on Nexus 4 vs other devices are 1) sample rate (48000 vs 44100), and 2) buffer size (for all other devices, I get 1024/1024, but for Nexus 4, I get 256/512 if I use AudioParam.init()and 512/12 if I don't use init()).

  • pbrinkmannpbrinkmann Posts: 686 ✭✭

    Unfortunately, it's not that straightforward. Yes, the N4 is more powerful than the S2, but it also has a higher resolution screen and camera as well as tighter timing requirements on the audio thread. Also, my N4 reports gc times of around 70ms, which is about three times what I see on other devices. It would be interesting how long gc takes on the S2; I expect it's a lot less than 70ms.

    So, I still think that the gc activity is the cause of your troubles. The question is what to do about this. One possibility might be to rewrite the app in terms of TextureView/SurfaceTexture. That would avoid the wholesale creation and destruction of a large byte array. You can ask the TextureView object to create a bitmap for you, and if you choose a small size for that, there'll be less for the gc to do.

    Another possibility would be to make buffer sizes in PdAudio/PdService more configurable, but I'm reluctant to do that. OpenSL is extremely finicky that way, and manually choosing buffer sizes is probably not feasible for developers who don't have a dozen different devices to test on.

    Maybe there should be one parameter that you can set for performance (the current behavior) or for safety (one might punt and choose 1024-frame buffers throughout). Not sure how to handle this; suggestions are welcome.

  • I see your point re GC. I can change the code that gets the best preview size of the camera - I'm not sure it gets the smallest preview size at the moment (I wrote that more than a year ago...) so I can test the preview size, thus the byte array size, and its effect on the sound that way.

    I'm still not comfortable with the smaller buffer size on the N4. I think the option to be able to "fix" it to a given value would be useful for at least testing purposes (so one can experiment with values to see the effect on the app/specific device etc).

    How about adding one parameter (int bufferSize) to your PdAudio.initAudio and appropriate documentation (warning of risks - example below)? IMHO, if the documentation is clear then developers will know that the bufferSize option is risky. I don't want to twist your arm with this though, as I'm not sure how important it is for most developers and you don't want to complicate the API for no valid reason as it could lead to more misuse/misunderstanding from developers.

     * @param bufferSize - if the value is greater than 0, this value is used for Android 2.3+. Note: this means that the buffer size will not be optimised for each device the app runs on and so this may lead to performance issues on a given device. Use with caution!
    
  • I'll post again once I've tried to change the preview size of the camera, hopefully I can do that today or tomorrow.

  • pbrinkmannpbrinkmann Posts: 686 ✭✭

    Sounds good. Let's see whether we can resolve this through defensive memory usage. If that doesn't work, we can consider other options.

  • I've done some tests as follows:

    1) amended my code to pick camera preview size and reduced the camera preview from 720 to 320 in width (and whatever the equivalent height is) --> reduction of GC_FOR_ALLOC messages (it went from about 3 messages per second to 1 per second, which is what I am getting with the S2). Slight reduction of crackles but crackles still very noticeable.

    2) amended PdAudio.initAudio to use 1024 for in/out buffer (instead of 256/512) --> no crackles anymore.

    3) amended PdAudio.initAudio to use 512 for in/out buffer (instead of 256/512) --> fewer crackles than when the camera preview size was larger, still about one crackle every 30 seconds.

    So clearly, changing the camera preview size has improved the performance and has brought the GC in line with what I see on other devices but this hasn't been enough to completely remove crackles.

    So, I'm curious, what is the negative trade off for having a bigger buffer size? Increased latency? I haven't actually noticed any difference but I guess the crackles at lower latency on the N4 are getting in the way of comparing properly and all my other devices are picking 1024/1024 buffer anyway. The nature of my app is slightly different from most apps - most apps use touch and the latency is quite obvious but as MovSynth relies on movements picked up by the camera, it's not as easy to measure the latency (as it could be the movement wasn't picked up or was picked up not as soon as it started because the movement was too slight initially etc).

  • "reduction of GC_FOR_ALLOC messages (it went from about 3 messages per second to 1 per second, which is what I am getting with the S2)" --> also, pause length is similar to what I get on the S2 (30 to 60ms)

  • pbrinkmannpbrinkmann Posts: 686 ✭✭
    edited February 2013

    I suggest you reduce the preview size even further. If I'm not mistaken, then you map the image down to a grid of 2x12 or so, and I figure that the resolution of the image doesn't have to be much more than that.

    About the trade-offs of bigger buffer sizes, there are two, basically. The most important one is latency, as you suspected. But I also recall that in some of my tests, excessive buffer sizes caused glitches, so it's not necessarily true that bigger is safer.

    My main concern, however, is about the API and the way it is used. One of my goals is to protect developers from device-dependent low-level details like buffer sizes. That's not just a matter of convenience; it's also a matter of documentation, support, and reliability. I'm afraid that a configurable buffer size would invite people to tweak the buffer until works well on their device, causing breakage everywhere else.

    So, let's try an even more defensive approach to preview sizes. If that doesn't help, I suppose Plan B is to add a second init method that makes proven but conservative choices.

  • "I suggest you reduce the preview size even further. If I'm not mistaken, then you map the image down to a grid of 2x12 or so, and I figure that the resolution of the image doesn't have to be much more than that." --> Not exactly, no. This wouldn't work because a) the grid can vary (this is set by the user when defining a palette), and b) the algorithm is about detecting sufficient movement in the area, so this is done based on movement in more than x % of pixels in the area, with only one pixel per cell, you would be really subject to background lighting effect, that would be very difficult to control. Also, c) the image would look very pixelated and this wouldn't make for a good UX.

    The purpose of the app is to play a synth via movements of the body - there is absolutely nothing to be gained for a user if the preview frame becomes a 2x12 grid, I might as well just implement touch and make it a kind of on screen keyboard, like every synth out there. But this isn't what I am interested in doing, the purpose of the app is to experiment with creating music via movement (picked up by the camera), not an on screen keyboard.

    I don't really know whether you should change the API or not (I gave you a suggestion as you asked for it) - I have changed your code for my purposes and I can live with that, this is after all the advantage of using an open source library ;-) The reason I am not sure whether you should change the API is because I agree with you that there is a risk that developers misread, or don't read, the documentation and tweak an app to work on 1 device only (too many developers still rely on only 1 or 2 devices when testing - the company I work for only provide one device to their official Android tester, which is completely ridiculous).

    My other question re: buffer size was, is it better to have the same in/out size or is there a case for different ones? I'm asking out of curiosity because I have noticed the N4 picks different sizes (256 for out and 512 for in) but all the other devices I have tested on pick the same size.

  • pbrinkmannpbrinkmann Posts: 686 ✭✭

    Android treats audio input and output as totally separate streams, and so it's okay to choose different buffer sizes. Since there's no low latency track for audio input, I chose a more conservative setting for input.

  • For now, I have settled to a preview camera size of 320 * 240 (or nearest available on device) and fixed buffer of 1024/1024 (I have changed PdAudio code for this). It tested fine on all my devices (HTC One S, Nexus 4, Nexus 7, Samsung Galaxy S2 and Sony Tablet S) so this is the best I can do at this point for the app... I'll see how it goes - the app has over 2000 active users so I'm sure I'll hear back from them if this buffer size creates an issue ;-)

Sign In or Register to comment.