Thursday, January 22, 2009

Microphone access in Silverlight via Flash and JavaScript

Silverlight is unquestionably a powerful tool for .NET application developers; using almost of all the same WPF skills,.NET developers can create rich internet applications (RIA) right out of the gate. However, Silverlight has no microphone/webcam support, and if you need this functionality in a cross-browser RIA, the common thinking is cross out Silverlight and pencil in Flash or Java.

While it's true that you'll need to dip into Flash or Java for microphone/webcam support (at least until Silverlight 3, and possibly not even then), there's no reason you can't take advantage of Silverlight as your base application host and selectively use Flash as necessary for Silverlight's missing features. Then, using a JavaScript bridge, you can get these browser technologies communicating with each other fairly painlessly. This integrated solution is what we're considering at Tutor.com when we add voice communication support to the Tutor.com Classroom, and this post will detail how to use Flash for microphone support inside a Silverlight application, where the button controlling the microphone on/off switch is in the Silverlight application.

Flash: Microphone management
First, we'll create a simple Flash movie (I'm using ActionScript 2) that manages and outputs microphone input. Thankfully, there's really very little code you need to write to do this; just note that your Flash movie must be at least 250x150 to allow room for the microphone Allow/Decline dialog, and if you don't want the movie to be visible, you need to set its background color to transparent.

As an aside, note that there's similarly very little code you need to write to connect to a Flash Media Server (FMS) for real-time communication, although the example here omits the FMS part. As another aside, note that the code for Flash webcam support is almost identical, except instead of a Microphone object you're working with a Camera object.

OK, so in the first frame of your Flash movie, write a function to access the microphone and monitor microphone activity:

var mic:Microphone = null;

function toggleVoice(isOn:Boolean)
{
//setup the mic
if (mic == null)
{
//call Microphone.get() to access the microphone and prompt user with Allow/Decline dialog
mic = Microphone.get();

//Microphone.get() will return null if user declined access
if (mic == null)
return;

//setup onActivity handler to get notification of mic activity
mic.onActivity = function(active:Boolean)
{
//call out to JavaScript bridge via ExternalInterface
flash.external.ExternalInterface.call("IsTalking", active);
};

//create movie clip and attach mic to clip so we can hear output
this.createEmptyMovieClip("sound_mc", this.getNextHighestDepth());
sound_mc.attachAudio(active_mic);
}

//set the microphone gain as per the isOn input variable
mic.setGain(isOn ? 50 : 0);
}

Below the function, add a callback handler that makes the function callable via JavaScript:

flash.external.ExternalInterface.addCallback("ToggleVoice", this, toggleVoice);

Html: Application host page
When you add a Silverlight application to your solution in Visual Studio, the boilerplate html makes your application full-screen in the browser window (as in the Tutor.com Classroom); this is exactly what we ant. Next, we need to add our Flash html object tag. Silverlight is a bit picky about where you add this html tag since you need to preserve this 100% height style, so slot the Flash object tag immediately before the div tag, and style it with "position:absolute" so that it doesn't interfere in the html layout:

<form id="form1" runat="server" style="height:100%;">
<asp:ScriptManager ID="ScriptManager1" runat="server" />

<object
id="voice"
style="position:absolute;"
classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000"
codebase="http://fpdownload.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=8,0,0,0"
width="250"
height="150"
align="middle"
...
/>

<div style="height:100%;">
<asp:Silverlight
ID="Xaml1"
runat="server"
...
/>
</div>
</form>

Javascript: Flash/Silverlight bridge
Next, we'll create the JavaScript function that our Silverlight app will call, which will in turn call into our Flash movie to set the Microphone input status. Since our Flash movie registered the ToggleVoice() callback function using ExternalInterface, Flash added a JavaScript entry point to function when the Flash movie loaded:

<script language="javascript" type="text/javascript">
function toggleVoice(on)
{
var flashVoiceObject = thisMovie("voice");
if (!flashVoiceObject)
return;

flashVoiceObject.ToggleVoice(on);
}

function thisMovie(movieName) {
if (navigator.appName.indexOf("Microsoft") != -1) {
return document.getElementById(movieName);
}
else {
return document[movieName];
}
}
</script>

We can also very simply call from JavaScript into Silverlight, and we'll do this to alert our Silverlight application when Flash notifies us of microphone activity:

function IsTalking(active)
{
//access the Silverlight application
var control = document.getElementById("Xaml1");

control.Content.Page.ToggleIsTalking(active);
}

Silverlight – Microphone toggle button
Finally, in Page.xaml, add a toggle button with Checked and Unchecked handlers, and an indicator that we'll show when the microphone is active:

<ToggleButton Content="Toggle Microphone Input" Checked="ToggleVoice_CheckedToggled" Unchecked="ToggleVoice_CheckedToggled" />

<TextBlock Text="You are talking..." Visibility="Collapsed" x:Name="IsTalkingIndicator" />

Then in Page.xaml.cs, implement the handler and call into the JavaScript function:

private void ToggleVoice_CheckedToggle(object sender, RoutedEventArgs e)
{
//call js layer to set microphone state
ScriptObject voiceToggleScriptObject = (ScriptObject)HtmlPage.Window.GetProperty("toggleVoice");

ToggleButton tb = sender as ToggleButton;
voiceToggleScriptObject.InvokeSelf(tb.IsChecked.Value);
}

Lastly, implement the ToggleIsTalking() function that we'll call from JavaScript when the Flash movie responds to the microphone's onActivity() event. You just need to decorate this function with the [ScriptableMember] attribute so that Silverlight registers the function with the JavaScript runtime:

[ScriptableMember]
public void ToggleVoiceOn(bool On)
{
IsTalkingIndicator.Visibility = (On) ? Visibility.Visible : Visibility.Collapsed;
}

And that's it! You now have a relatively straightforward Flash <=>JavaScript <=> Silverlight solution that affords you the best of all worlds.

17 comments:

Anonymous said...

...and this post is pure gold. Thanks a lot.

Anonymous said...

can you post FMS part of it, so that we can learn how to use it.

FMS is crucial part of this because FMS is what it required to record and store voice.

orangejim@gmail.com

raulgspan said...

sure, the FMS part is very straightforward. when your movie loads, you just need to connect:

nc = new NetConnection();
nc.connect(FMS_URI);

set up a receiver:

//create stream
ns_r = new NetStream(nc);

//create clip and attach stream
this.createEmptyMovieClip("sound_mc_r", this.getNextHighestDepth()); sound_mc_r.attachAudio(ns_r);

//play stream
ns_r.play(STRING_IDENTIFIER);

and then setup a publisher:

ns = new NetStream(nc);
ns.publish(STRING_IDENTIFIER, "record");
ns.attachAudio(Microphone.get());

it's really that simple--

Anonymous said...

Do you happen to know how I can store the recorded data locally? In a memorystream or some sort of buffer? Then I guess it could be base64 encoced and passed to Silverlight..

Ken Smith said...

Just for what it's worth, although this technically works, I've spent a lot of time playing with this approach, and it leaves a lot to be desired. There's a lot more to managing a camera and a microphone in a production application than simply showing it and hoping that it works. You've got error handling, repositioning, and all sorts of other issues that you have to deal with, and none of them are straightforward when you have to go through two levels of indirection to get there.

To take one example: when you show a ChildWindow in Silverlight 3, the flash video window is positioned on top of the ChildWindow. Not the sort of experience you'd generally want to offer your users.

The more I work with this particular solution, the more annoyed I am that MS -- despite acknowledging that webcam support is the most requested feature -- decided not to put it into SL 3.0, and will apparently be delaying it until SL 4.0. Damn MS...

raulgspan said...

Anonymous said...
“Do you happen to know how I can store the recorded data locally? In a memorystream or some sort of buffer? Then I guess it could be base64 encoced and passed to Silverlight..”

Unfortunately, there's really no way to save the data locally. You have to run FMS or its equivalent and attach the microphone to a publish NetStream--

raulgspan said...

Ken Smith said...
“Just for what it's worth, although this technically works, I've spent a lot of time playing...”

Yes, this Javascript bridge between Flash and Silverlight is definitely of the "hack solution" variety, and I agree that MSFT should have gotten these rich features to us sooner, but in any app (especially one in the browser) you're going to have issues to deal with. In general, if you support IE 6, 7, and 8, Firefox 2 and 3, and Safari 4, nicely designed html browser apps degrade to CSS hack after hack in the most non-straightforward imaginable. At least you know your Silverlight app will run the same everywhere...

Regarding the overlay issue: I think you'll find that the Flash movie staying on top of a ChildWindow is really a limitation of certain browsers. Just make a simple html page with a flash movie and try to show a spotlight <div> on top of it. You'll see that some browsers are fine with this and others (not naming names, but if I remember correctly MSFT is in the wrong here) exhibit the behavior you describe where the movie stays on top of the spotlight. You'll just have to employ a relatively simple "hide Flash movie before showing spotlight" technique. Just don't actually "hide" the movie, since Firefox will reload it when you re-show it :); set its size to 1,1 or its position off-screen or some other such thing--

Anonymous said...

Thanks for the genuine solution. My Q. Is there a way to record the audio generated through the microphone without using Flash Media Server?

raulgspan said...

Nope, there's no way to record the audio stream locally. You have to publish it to FMS and record from there.

Anonymous said...

Good job! Can this solution be used for live video conferencing? Thanks

raulgspan said...

Sure, just use the attachVideo() method. I don't recall the exact syntax, but the concept should work fine--

Unknown said...

By any chance I can find a completed package to test it out? Tks.

raulgspan said...

No, sorry. Let me know if there's anything in particular that you're getting stuck on--

Unknown said...

Not sure what dll or lib are required as a whole?

Anonymous said...

What is the file format of the movie?

raulgspan said...

David said... "Not sure what dll or lib are required as a whole?"

You just want a barebones Silverlight project. The bridge to Flash is via Javascript.

raulgspan said...

Anonymous said... "What is the file format of the movie?"

It's just a regular SWF file.