Integrating libpd with the Unity Game Engine

arcadiadivine

I'm still relatively new to this, so you'll have to forgive my ignorance, but how would I integrate libpd with the Unity Game Engine? More specifically the free version. Is there any specific process involved? Would it even be possible with the free version of Unity? Here is a link to the website. http://unity3d.com/unity/licenses

pbrinkmann

FRACT uses libpd with Unity 3.5 (fractgame.com). Henk Boom, one of the developers, says he'll write a blog post about it soon.

arcadiadivine

Cool, that's a relief knowing it can be integrated into Unity 3.5. Do you have any tips or suggestions on integrating libpd into the free version of Unity 3.5? I recently bought the Android and iOS basic licenses because I eventually want to develop for those platforms. Well actually I got them for free because of the sale Unity Technologies was having, but that's not the point of the conversation. I figured it was either libpd or Csound. I'd prefer to use libpd because it doesn't seem like it takes nearly as much effort as integrating Csound into Unity 3.5. Like I said before though, do you have any tips or suggestions on integrating libpd with Unity 3.5? Any amount of help would be greatly appreciated!

pbrinkmann

I have personally never looked at Unity. All I know about libpd and Unity is what I learned from Henk Boom's tweets (@henkboom). I'm looking forward to reading his blog post.

reakinator

Wow, I didn't realize FRACT used libpd, I've been following it's progress and wondered what they used for sound synthesis - cool!

@John Colgrove, can you find any documents at tying into Unity's audio layer through it's plug-in interface? I can only find this little bit of info from the Unity manual:
http://unity3d.com/support/documentation/Manual/Plugins.html

So, there is a C plug-in interface, and you need to find out how it provides and audio callback, where at you could tie in libpd with libpd_process_float() and friends.

Or, you can wait for this anticipated blog post. :)

pbrinkmann

Yes, FRACT uses libpd: http://createdigitalmusic.com/2012/02/fract-3d-adventure-game-played-with-synths-and-sequencers-myst-meets-music-making/

From what I gather from Henk's tweets, there are concurrency issues to consider when using the C plugin interface, and so it may be a bit tricky to get right.

pbrinkmann

Thanks for linking to the Unity manual, Rich! I took a look, and it appears that we really ought to add C# bindings for libpd. We already have a current pull request (https://github.com/libpd/libpd/pull/6) that adds support for DLLs to the build system, so that should help a bit. Any volunteers?

joewhite4

@John Colgove coincidentally i'm just on my way back from the Amsterdam MusicHackDay where I was working on integrating ZenGarden as a plugin in Unity. I've got it running but have some issues at the moment with opening patches on iOS.

As Peter pointed out, Henk has already integrated libpd in FRACT. Hopefully he'll get his blog post up soon and I'd also like to document the process I went through. In the meantime here's some info that you might find helpful.

- a C/++ library has to be integrated as a plugin to work in Unity
- The plugin is just a simple C#/JS script
- Using plugins requires Unity Pro UNLESS you're building for iOS or Android
- The iOS/Android add-ons are currently free (until April I think) so get them quick!
- The library just needs a C interface, I would say it doesn't seem neccessary for C# bindings in libpd, but I could be wrong.

curby

@John Colgrove I've also been tinkering with this and had moderate success using Unity to run patches using libpd as a plugin on Android.

I say moderate success in that I've gotten libpd to initialise, load a test patch without error and output audio on my android phone. The audio is mostly correct, however there are some artifacts and the audio seems to jump a bit.

For those wondering the method I've used to tie libpd into Unity's audio layer is using the AudioClip.PCMReaderCallback >> http://unity3d.com/support/documentation/ScriptReference/AudioClip.PCMReaderCallback.html
I'm using this callback ( which from the little to no documentation about it, have assumed is a buffer request from the audio card via FMOD ), to 'pump' the libpd process function. However I'm unsure of how 'in synch' this is with libpd (it seems not very) and whether it's being called at the correct times. Does a buffer request from the audio card seem like a viable method to call libpds process functions?

Also as an aside I just wanted to say a big thanks to all who've contributed to the libpd project, it's really inspiring work!

pbrinkmann

@Luke,
Yes, that's exactly what the process functions are for. There shouldn't be any timing issues because libpd's only measures time in terms of the number of samples requested via libpd_process*. The artifacts you're experiencing are probably due to mismatched buffers; make sure that Unity and libpd agree on buffer size, endianness, and the alignment of channels.

curby

@Peter OK, thanks for verifying. I'll take another look at those buffers.

berenger

actually Unity for android and iOs is free until 8th of april : https://store.unity3d.com/index.html

so get it while it's hot and follow this topic !

tebjan.halm

now, is there any c# file around? otherwise i would set one up...

pbrinkmann

Several people have asked about C# support, but so far nobody has done it, at least as far as I know. It would certainly be good to have.

tebjan.halm

yes, i have working c# bindings now... will fork the repo in github and open a pull request.

i just have a little problem. i am trying to set the printhook too but it looks like its not called by libpd. for that i have created a setter method and a test method in the C code because .Net/Pinvoke can't set dll fields. the test method works, but when i load a patch with prints i don't get anything...

tebjan.halm

pull request done.

pbrinkmann

Thanks, Tebjan! I left some preliminary feedback at GitHub.

pbrinkmann

By the way, about having libpd call methods in C#, it appears that COM Interop is the way to go: http://msdn.microsoft.com/en-us/library/zsfww439.aspx

filippo.peccoz

Very interested about this, too. This could open up heaps of new opportunities for in game music!

pscmtl

Hi!
1) I have compiled libpd using MinGW:
make csharplib = libs/libpdcsharp.dll!
2) Now in SharpDevelop I have compiled csharp/LibPDBinding.csproj = bin/Debug/LibPDBinding.dll!
3) In SharpDevelop I have compiled csharptest/LibPDBindingTest.csproj = LibPDBindingTest.dll!
Now I am looking for help: Anyone successfully use this in Unity or even a simple C# audio demo? I see that tebjan have a folder called csharp in samples: https://github.com/tebjan/libpd/tree/master/samples/csharp but it doesn't work with LibPDBinding.dll
Thanks!

pscmtl

Hi again!
I think the first step is a simple console application (no sound) then I will try using the C# PortAudio or NAudio. Right now I have an error:

System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
at LibPDBinding.LibPD.openfile(String basename, String dirname)
at LibPDBinding.LibPD.OpenPatch(String filepath) in C:\MinGW\msys\1.0\home\psc\libpd\csharp\Native\LibPDNativeMethods.cs:line 143

I am using SharpDevelop.
Thanks!

pscmtl

Resolved the read or write protected memory error: when using OpenPatch you need to add ./ if the patch is in the same directory of your executable:

OpenPatch(@".\test_csharp.pd");

Now trying to import those DLL in Unity...

metaphysician

hey listers, new guy here - greetings!

has anyone got libPD working with Unity3D Free? alpha build, beta build, whatever. i know tebjan created a version of libpd with C# bindings, but i've no idea as to what to do to make it work on a Mac, as there is no documentation.

i've heard precious little since then. patrick sebastien got it to barely work on PC i believe. anyone else have any idea? i've been in contact with someone who claims it's working fine in a prototype form, but hasn't released anything so far to check out, and i know Henk Boom uses it in Fract but i haven't heard anything there either. he was supposed to detail a blog about it 'soon' back in March when i last read about it.

anyway, any information appreciated. thanks for any help!

rrodkin

I too have been waiting for the much-anticipated post from Henk, but I suspect that with new fatherdom and game dev, it won't be coming any time soon.

Good news, though, I think. After yet one more google, I found this post, and looking around here, it doesn't look like anyone has picked it up. I haven't tried ANY of this out yet, so caveat emptor, but it does look promising, and at least offers some transparency into a workable solution:

http://blog.tridek.com/post/29834155461/using-libpd-with-unity3d-on-mobile-devices-part-1

I also noticed that a poster in this thread -- filippo.peccoz -- is involved with the tridek project, and has posted an article as well, which looks very interesting:

http://blog.tridek.com/post/29055895421/welcome-to-trideks-audio

@Filippo: Any news or insights re: the above? Tridek looks like a pretty cool project, would love to hear more.

Thanks!

rrodkin

Arrrgh. Looks like I was a bit over-eager -- according to the Tridek blog post, they've only got Unity and libpd running via TCP in Edit mode . . . which is essentially the same method as using OSC -- all well and good until runtime.

I suppose they'll need to figure it out sooner or later, and it will be fun and interesting to observe . . . and I wish them best of luck!

In the meantime, looks like we're back to waiting for Mr. Boom!

pbrinkmann

Patrick Sébastien to the rescue! https://github.com/patricksebastien/libpd4unity

rrodkin

WOW! that is quite a gift . . . thank you!!

pscmtl

np but it's still a work in progress! it would be very nice if people could fork and pull request so that we have a solid integration. i am also looking for a way to use audio input (soundcard input) and compiling for android and iOS. i have finally managed to use statically linked external (look at my fork of libpd).

the main issue is that pd / libpd can only be use in a single instance, meaning that we have to switch~ patch to avoid glitches and i am have issue with expr~ / sig~ timing.

bzgeb

I wrote a post about embedding libPD into a Unity iOS app http://bronsonzgeb.blogspot.ca/2012/08/embedding-pure-data-into-unity-ios-app.html which also covers how to embed libPd into an iOS app. I wasn't aware that Patrick Sébastien had created libpd4unity at the time.

metaphysician

the biggest thing about Patrick's approach is he compiled the wrapper/DLL for the libpd base in Windows and the Unity project he uses crashes hard and completely on a Mac. it looks like yours was made for a Mac, so i will definitely give this one a try when i have a moment.

i like filippo's approach in terms of designing or prototyping, but my idea would be to run a separate and more generally functional GUI sending commands to libpd to dynamically create Audio Sources and manipulate audio parameters, rather than using the PD canvas to do it. my idea is to be able to use this GUI with other game engines and APIs.

i am curious if at some point you'll be able to run multiple instances of libpd. i suspect it will be a processor hog for smaller devices, but i'd be happy to be proven wrong.

metaphysician

@rrodkin - i think you should take a look at the post again from filippos crew (it's sebastien who is programming it). it looks like they do have it working on Android and iOS, they simply prototype in PD/OSC. and yes there's only one instance of libpd. they have directions on what exact code to modify/adapt in iOS Xcode and Android.

rrodkin

@psmtl and bzgeb - looking forward to trying these out, most likely over the weekend, thank you again!

@metaphysician - i did see the instructions for droid and ios, my comment was more in reaction to my own assumption that they had an actual plugin, since that's what the Fract team allegedly have produced. Not sure to what the comment about 'only one instance of libpd' refers?

Anyway, it seems there are three approaches in the mix now, not including FRACT, very encouraging!

metaphysician

well, patrick can correct or fill in details, but if i remember, he wants to be able to run several instances of libpd at one time. each instance would presumably have a separate function, so let's say one for each Audio Source in the Scene and maybe another for the Audio Listener. i think this is a great idea but i suspect it would cause issues, performance wise, on a portable device. these are guesses - i am nowhere near a coder or even a PD/libpd expert, though i've done a lot of MaxMSP since the earliest days.

rrodkin

Ah, I see what you mean now, thanks for clarifying, and yes, would definitely like to hear more about his approach as well.

I'm coming up to speed on pd as well, though I do have js and c# skills. . . got into Max via Max4Live and the need to integrate with Unity3D. . . then drifted away from it as there was no way to produce a single runtime executable, for mobile or desktop. Which is why I'm intrigued and now drifting back (albeit to pd, not max).

In the meantime, for my purposes, it turns out that Unity's built in audio capabilities have been good enough for POC work -- it's at least possible to dependably sync multiple audio files, which is primarily what I need for the moment. pd obviously offers more flexibility, and is therefore the focus of my next round of R&D.

FBP

Hi all,

in case you're interested, here's a Unity-PD binding that uses libpd:

https://github.com/hagish/kalimba

Hope Kalimba can help you out in connecting the two together! Try it out and let me know :)

Cheers,

FBP

Howdy, Stranger!

Categories

In this Discussion