Introducing Marsbot for AirPods

Looking at headphones as widely-used augmented reality hardware

9 min readJun 28, 2021

(originally published on Foursquare’s “Intersections” blog — Oct 20, 2020)

Hello from Foursquare Labs! We’re Foursquare’s internal R&D team — the team tasked with building new, novel and weird stuff using Foursquare tools, technology and data. Today we’ve got a new experiment for you to play with: Marsbot for AirPods, a lightweight virtual assistant that proactively whispers local recommendations (and other fun snippets) into your headphones or earbuds as you’re walking around. It’s our take on what happens when Foursquare merges with Clippy (yes, from Microsoft Word!) with a dash of Samantha from the movie Her (ScarJo’s character!) and the realization that headphones may just be the world’s most ubiquitous Augmented Reality hardware.

Let me share some of the backstory on the project and the ways we think it foreshadows the convergence of virtual assistants, wearables, augmented reality, and the types of tools we use to explore the world (once it’s safe to do so again).

Recently the Foursquare Labs team been exploring a number of different themes including:

Apps You Don’t Have To Use: We don’t believe “the future of software” looks like someone endlessly flipping through Instagram or doomscrolling their way through Twitter. Nor is it 100 different apps continuously sending notifications begging for your attention. We believe the future of software is apps that understand your context and surroundings, and apps that smartly nudge you when there is something interesting or important at your periphery.
Proactive Virtual Assistants: We always think of Siri, Alexa and Google Assistant as the genies that live inside the magic lamps that are our phones. The problem is that the genie doesn’t come out unless you rub the magic lamp (“Hey Siri! Hey Alexa! Ok Google!”). But what if the genie was the one tapping you on the shoulder?
Audio as “Poor Man’s Augmented Reality”: If you only look at the AR opportunity as “vision,” the whole space can sometimes feel like it’s waiting for a breakthrough in hardware — specifically some form of AR glasses that people actually want to wear all day long. We decided to build for the “lowest common denominator” piece of Augmented Reality hardware that’s out in the wild today (and that people already wear all day long) — headphones. (Note: We’re not the first to flirt with this idea).
Apps for AirPods: While we have apps for our phones and watches and TVs and cars, we have yet to see apps designed specifically for AirPods, specifically all-day AirPod use. When we got started we asked ourselves, “what value would an app have to provide in order to convince someone to leave one AirPod in their ear all day long…. even if they weren’t actively listening to anything?” (Again, loosely inspired by Her.)
Software That Makes Cities Easier to Use: This was Foursquare’s original mission when we started the company some 10+ years ago. We dreamed big about maps that would be personalized, technology that would facilitate serendipity, and software that would encourage and reward you for exploration. This DNA still runs strong through much of the work Foursquare does today.

For more than a year, we’ve been working to jam all five of these themes into a single “app for AirPods” which we are calling Marsbot for AirPods. Some of you may have heard my talk about this at the Betaworks Audio Summit almost a year ago. And Foursquare Superfans may remember Marsbot 1.0 which was a proactive assistant that sent text messages (and you can think of this as a sequel to that).

Today we’re excited to push what we’ve built into the Apple App Store so you all can play with it. Currently, it’s only available on iPhone, but despite the name, it will work with any type of headphones, not just AirPods.

Download now for iOS

Android users: Sign up for the Android waitlist

So, what is Marsbot for AirPods?

Think of it as a “proactive walking assistant” (shoutout to Lindsay Zver who coined that phrase during a Q&A I did with Boston College students). It’s simple: put your AirPods in your ears and as you’re walking around, Marsbot will proactively whisper things to you that you may find interesting. The whisper could be about a place, an object, a building, some street art, or even a person (!). We’ll pause your podcast and lower your music while we whisper something (and we’ll never interrupt a voice or video call).

We built Marsbot for AirPods to feel like you are walking down the street with a friend who knows everything about the city and is constantly pointing out the most interesting things to you. Many of the notifications are designed to help you notice places and things you may have never noticed, even though you’ve walked by them 100 times before.

Here are a few examples of the types of things you could hear as you walk by places in NYC:

[PLAY] Blue Bottle Coffee is the #1 rated coffee shop in Greenwich Village according to Foursquare.
[PLAY] Los Tacos №1 is a taco shop that made Eater’s list of “38 Essential Restaurants”
[PLAY] Sotto 13, on the south side of W 13th Street, is an 8.6 rated Italian Restaurant according to Foursquare
[PLAY] Alex is at The Scratcher!

Marsbot for AirPods also lets you record your own audio snippets, using your own voice, which are then left at places for other people to discover as they walk by. We’ve seen people use this to leave tips (“Get the burger!”) or friendly advice (“Don’t get the burger!”).

While it’s been fun to see people do the obvious, it’s been fascinating to see people do the completely unexpected — like singing a song, telling a joke, or leaving a missed connection. After we noticed the first few unexpected uses, we were inspired to turn some of them into features:

McDonald’s Jingle: We wrote some code so that everytime you walk by a McDonald’s (and I mean any McDonald’s in the world), you’ll hear me whistle, which is weird but also oddly interesting. (Last week we were using the official McDonald’s jingle, but Foursquare Legal made us axe it due to ™ concerns. So, hey McDonald’s, if you’re reading this, can we please have your permission to replace my whistle with your jingle?)
Walk-Out Clips: Marsbot will let you know when you pass another Marsbot for AirPods user on the street (“Max is nearby!”), but why not make that moment feel like you’re David Ortiz walking up to the plate at Fenway Park? When you upload a few seconds of audio (your “walk-up clip”), other people will hear it as you pass by them on the street. (You can add your “walk-out clip” — or opt-out of these “nearby” messages entirely — from the Settings screen in the app).

In short, with Marsbot for AirPods, we created an open-ended platform that allows people to leave any type of audio snippet anywhere in the world, but an audio snippet that can only be unlocked, delivered, and heard once someone walks by the place where the snippet was planted. (We will be proactively monitoring uploaded clips for abusive or offensive content and have also given users the ability to “flag” offensive content they have heard for human review.)

Also worth noting that this is a v1 — and that there is so much more we can do with this platform, whether it’s adding different sources of content (travel guides? transit times? COVID updates? to-do list items?), different voices (celebrity voice-overs? synthetic celebrity voice-overs?), different types of audio (music? sounds effects?), different ranges of motion (spatial audio & head tracking?) Stay tuned!

A few odds and ends:

Right now, Marsbot for AirPods is only available for iPhone. Despite its name it will work with any type of headphones or earbuds, wireless or wired.
It’s designed to work best when you’re walking around, though it’ll also work if you’re running or riding your bike. It’ll even work when you’re driving, which was originally a bug but it was one of those “bugs that felt more like a feature,” so we left it in. We’d love to hear your feedback on this.
It’ll work anywhere in the world, but you’ll probably find the most pre-loaded content in big cities like NYC, San Francisco, Chicago, Seattle or Los Angeles (aka: places Foursquare has offices — ps: we’re hiring!)
It also works better outside than inside. Marsbot for AirPods is NOT a liner walking tour (“turn left, walk 100 steps, turn right”). It’s more of an “open-ended, audio-first, augmented reality experience triggered by whatever path you chose to travel as you go about your day”. That said, it’ll work better outside (think: big open spaces with street addresses and intersections) than inside (e.g. offices w/ conference rooms). You can leave audio snippets at a specific place (coffee shop, park) or at any latitude/longitude (hiking trail, bike path, random intersection, mountaintop) on earth.
The experience is designed to be light. You may only hear one audio snippet a day, or you may go days without hearing any audio snippets. In the spirit of “an app you don’t have to use,” we didn’t want you to have to fiddle with settings or “mute” buttons because the app was too chatty. It’s worth noting that the experience may be really chatty the first few days as it gets used to your surroundings.
That said, version 1.0 of this thing does not have a “follow” mode — meaning you can hear everything and anything created by someone else, and they can hear everything and anything created by you. This may create chaos, or it could create something interesting. If you hear anything offensive, please just use the app to flag it, and we’ll take care of it.

And finally…

We didn’t create this app in the hopes of getting 10,000 or 100,000 or 1 million users. Remember, it’s a prototype… and a beta… and an experiment rolled into one. We built it to show where we think the world is heading as well as to show how Foursquare technology can help developers get there faster. As we like to say: “It’s not just something that only Foursquare can build, but something that only Foursquare would build.” (Note: Marsbot for AirPods is powered-by our freely available Pilgrim SDK.

We are very much looking forward to seeing how people use the platform, so be adventurous and fun and playful with it. Please don’t hesitate to reach out to us with any questions or feedback!

Huge thanks to Max, Anish, Emma and Alicia (and DavidW & MattS), the rest of the Foursquare Labs crew, and all of our amazing beta testers for all their work in pushing this out into the real world!

///

Bonus Thoughts: Keep reading if you’re interested in learning our thoughts on best practices for fellow “audio app” designers

You’ll notice a few thoughtful features we added to *Marsbot for AirPods *which we think will become standard in audio-first apps. It took a lot of tinkering for us to land on these, so we figured we’d share:

Whenever you put your AirPods in, we play a special chime and say “Marsbot Activated!” to let the user know that this app is present and working. (We do this right after the standard AirPods chime plays.) We thought this was especially important because our app requires background location permission to understand context, and we wanted the user to be aware that the app is always looking for things it can whisper to you (because it’s always looking for geofences). Read more about Foursquare’s commitment to privacy.
If you happen to be listening to something else while Marsbot is active, we’ll keenly adjust — we’ll pause (and unpause) your podcast; we’ll temporarily lower the volume of any music; and we’ll never interrupt a phone call or video chat. We also made sure to test these rules on a number of 3rd party podcast, music, and voice/video apps like Breaker, Spotify, Zoom, Gmail, etc.
We try to keep our messages lightweight (no more than 10 spoken syllables), and we try not to be too chatty (you really shouldn’t hear more than a few messages per day). Our intention was to make Marsbot for AirPods more of a “lightweight companion” and less of an in-your-face, always-on presence.
We used audio for our onboarding, and we think it works pretty well! This was a trick we learned from the folks at TTYL. There is so much room for innovation and invention here. (Worth noting that both many forms of VR and AR onboarding are going through a similar renaissance)
In a similar spirit, we decided to do our Weekly Feature Updates not as emails or push notifications or pop-ups in the app, but instead in the form of an “audio newsletter.” So if you’re using Marsbot for AirPods, you may hear my voice pop up every now and then giving you an overview of new features or bug fixes (and I’ll try not to talk for more than 30 seconds!)