How it worked; Hacking in Seattle

At NBCU's hackathon, the Advengers assembled to build adHarmony: a second screen app for advertising. You can read about what it does in my last week's blog post "What I built hacking in Seattle".

Revealing how the trick is done

This post is all about how adHarmony worked.  You will probably find this more technical than my last two posts, but I'll try to keep it accessible.

a VERY quick VERY high level recap of adHarmony

adHarmony is an app that runs on a phone and listens for ads being played on some other device (a TV, another computer, whatever).  When adHarmony hears an ad, it gives the viewer a prompt to provide feedback and engage with or skip the ad.  The app delivers information about the user's feedback to a backend service.  

adHarmony Project Scope

1. adHarmony can hear and recognize nearby playing ads
2. adHarmony gives the viewer options specific to that ad
3. If the user chooses to skip the ad, skip the ad
4. Whatever the user chooses, keep track of what was selected

Now that you're caught up, the rest of this post will make a lot more sense!

adHarmony can hear and recognize nearby ads

How does adHarmony know when an ad is playing, and not the content?  The secret is: the ads we used in the demo are special.  

LISNR: the real hero

Using LISNR, we put a high-pitched, inaudible tone into the ads we demoed.  Humans can't hear it, but adHarmony (and apparently dogs) can.  

Supposedly, dogs hear a low hum

In each tone is encoded a secret message that the app hears.  Our friend Jill from LISNR described as "an intelligent audio QR code".¹

adHarmony gives the viewer options specific to the ad

When the app hears the secret message, it tells LISNR the secret message.  LISNR then responds with a specific message we configured ahead of time. The message tells adHarmony what to show for this ad.  Namely...

1. what the swipeable image should looks like
2. what web address to load when the user swipes left
3. what web address to load when the user swipes right

You wouldn't notice, but the app loads a webview after the user swipes.

If the user chooses to skip the ad, skip the ad

WARNING: below is some real inside baseball.  The gist is: for the demo, we replaced the normal program that handles ad playback on NBC's website with a custom program that waits for our server to tell it to skip an ad.

For the demo, we changed how an NBC video player loads on our computer using a web proxy called Charles. We actually changed the normal functionality in two ways:  

1. First...ya know how a website might load any video ad when you load it?  For our purposes, we made sure a specific ad (the demo ad) was returned on each page load.  
2. Second, we loaded a special version of ad manager we tweaked for the hackathon.  The ad manager is the piece of code that coordinates video ad playback on a website.  

How to force the ad you want to return is pretty basic testing functionality that FreeWheel supports.
The functionality we snuck into the special ad manager is noteworthy!

What makes this ad manager special is that it continually pings our dev server asking if it can skip the ad. 

If the dev server ever says "ok", the ad manager stops playing the ad and does whatever its supposed to do next: play another ad or go back to content.  The dev server only says "ok" when the adHarmony app has received an ad skip request.

Tracking Usage

Whenever the user app hears an ad or has some interaction, the app sends that message to the dev server and it's logged in a database.  We built out the backend, but there's nothing too special to talk about.  Nothing flashy or over-designed.  Just a Proof of Concept with a pretty basic database schema.

Putting it all together

• 1&2. When the webpage loads, it get's the special ad manager and the forced ad response from "the FreeWheel ad server" (actually local via Charles web proxy).
• 3. While the ad plays, the adHarmony app hears the LISNR code
• 4. AdHarmony pings the LISNR service and renders what returns
• 5. When adHarmony hears an ad, or whenever a user interacts, out dev server logs that event
• 6. While playing an ad, the ad manager continuously checks with the dev server to see if it should skip the ad.  When the dev server says ok, the ad manager skips that ad.

Advengers! Assemble!

So that's how it works.  I was thinking of ending the blog post with where we could go from here or what I learned, but I already covered those things in my past two blog posts. I guess I sort of wrote this series of posts backwards, so let's end by introducing the team!

Ben Pelcyger aka The Face aka General Mischief
In charge of general mischief...and faces

Mengdi Chen aka Death Wish
Very nearly strangled the scrum master every 30 minutes

Xindong Wang aka the Pelican
Ate his weight in fish

Haijun Yang aka The Human Yawn
Apparently does not require sleep to code.  

Yan Sun aka Hawk-eye
He caught like a million Pokemon that weekend

Wei Wei aka Noble Bull the No Bull Bull
Drank more coffee and red bull this weekend that anyone on earth

---

[1] Footnote: How practical is this?

It may sound difficult to pull this off outside a lab environment because you would need to put a secret message into each ad.  However, embedding a the tone into each ad isn't much of a hurdle.  It could be added to the master (mezzanine) file before it gets transcoded into its various forms.   

Fun fact: Nielsen has done something similar in the past to measure TV and radio viewership.  As far as I know, Nielsen tagged ads AND content, and they used these nifty pagers called "Portable People Meters" to track listeners.

Invented by a company named Arbitron!