How Audio Beacons Monitor You Via Smartphone
Table of contents
Has this ever happened to you? You’re talking to someone about something obscure and then the next time you open your phone, you see an ad for what you were just talking about. You’re pretty sure that this was the first time that this obscure topic came up and you get a strange deja vu feeling that perhaps your smartphone might be listening to you. Guess what? Your smartphone may very well be listening to every word you say, and there is quite a bit of anecdotal evidence that would lead us to believe that, including our own personal experiences.
The fact that electronic devices are listening to what you say shouldn’t come as a huge surprise. Around 4 years ago LG admitted that they were actually looking through the filenames on USB devices attached to smart televisions like the one seen below:
Just a few years ago, Samsung was called out for capturing private conversations and then passing them on to Nuance Communications. If you have an Amazon Echo in your house or any other sort of “smart assistant” then you’ve pretty much just resigned yourself to the fact that every single conversation you have is being listened to. While the topic of smart devices that listen to what you say is controversial and a bit scary, the real power comes from smart devices that can listen to hidden messages being broadcasted via “audio beacons” that are injected into content like movies, songs, or commercials.
Reading Audio Beacons Using Apps
Founded in 2015, Silicon Valley startup SilverPush has taken in $2.8 million in funding so far for their “mobile adtech startup that has created a single identity of users across all of their digitally connected devices“. While that description sounds innocuous enough, what SilverPush does is far from it. A warning letter from the Federal Trade Commission (FTC) describes the functionality of SilverPush platform as follows:
Silverpush makes available for application developers a “Unique Audio Beacon” technology that enables mobile applications to listen for unique codes embedded into television audio signals in order to determine what television shows or advertisements are playing on a nearby television. This functionality is designed to run silently in the background, even while the user is not actively using the application. Using this technology, Silverpush could generate a detailed log of the television content viewed while a user’s mobile phone was turned on.
That warning letter was sent by the FTC to application developers that were using SilverPush technology in the U.S. and as a result, SilverPush stated that they do not use that technology in the U.S. anymore, only in foreign countries (where people’s privacy doesn’t really matter.. or??). This made us wonder how these “unique audio beacons” mentioned in the above description actually get inserted into content? As it turns out, there are a number of companies that offer this as a service, companies like LISNR.
Inserting Audio Beacons Into Content
Founded in 2012, Ohio startup LISNR has taken in $14.35 million in funding from investors that included Intel (NASDAQ:INTC) to develop “a high frequency, inaudible technology; a communication protocol that sends data over audio“. For all you nerds out there, this means they’re using an audio frequency range of ~ 18.7 – 19.5 kHz which can transport encrypted information like time and location. Yes people, we’ve now moved into the “Internet of Sound” age. While the simplest application is to determine what the smartphone carrier is listening to or watching, the communication method can be used in any number of ways as seen below:
LISNR doesn’t require a server call to identify a Smart Tone, which means you can create connections without wifi, data, or general device connectivity. All smart devices are built with a microphone, no matter the make or model. LISNR Smart Tones are detected by the device’s existing microphone without server calls or recording audio like other services. One question remains though. How can we make this technology more palatable so people allow their behavior to be monitored so closely? As it turns out, just tell people that if they install the monitoring app you’ll give them free isht. This brings us to a company called Shopkick.
Getting People to Use Audio Beacons
Founded in 2009, Shopkick started out as a Silicon Valley startup but in 2014 was acquired by an $18.5 billion Korean telecommunications firm called SK Telecom (NYSE:SKM). The way this works is that people are rewarded for engaging with retailers, but not just by purchasing products. Simply walking into a store can give you rewards because the technology uses audio beacons to determine proximity since GPS just isn’t accurate enough. For example, there are over 230,000 retail locations that will give you “kicks” just for walking into a “bricks and mortar” retail location as seen below:
According to a recent article in Tech Crunch, the app has driven over $2.5 billion in sales and their 20 million active users spend 2.5 hours a month on the app. In exchange for using the Shopkick app, the company has distributed $60 million in gift cards to date. It appears that Shopkick is now getting into grocery so all those active users are likely to be controlled even more.
So what ever happened to SilverPush?
When all the privacy concerns around the SilverPush business model were voiced, the company promised that they were “killing the software”. Forbes put out an article that titled “Silverpush Quits Creeping World Out, Ceases Tracking TV Habits Via Inaudible Beacons” and everyone sighed in relief and went back to their daily routines. As it turns out though, that may not be the case.
Earlier this year, a German university published a paper that claims the SilverPush technology is alive and well. Here’s an excerpt from that detailed paper:
At the time of writing, we are aware of 234 Silverpush Android applications that are listening in the background for inaudible beacons in TV without the user’s knowledge. Several among them have millions of downloads or are part of reputable companies, such as McDonald’s and Krispy Kreme.
An article published this month by Ars Technica provides more details about this finding and also states that “SilverPush founder Hitesh Chawla said the finding surprised him because his company abandoned the ad-tracking business in late 2015“. A look at the SilverPush website shows that they are quite obscure about describing their technology but it does in fact look like they’re still monitoring audio signals from televisions.
While people are criticizing SilverPush, you can be assured that these “audio beacons” being emitted from any movie, TV show, or song you’re listening to can now be tracked by any device that can listen and plenty of other startups are getting in on the action. While we talked about your smartphone tracking such info, you may want to think about how always-on devices like Alexa are being used to track what songs you listen to and what movies you watch. Then, Amazon can try to sell you more stuff that’s similar to what you already listen to and watch. Your entire online experience is going to become one giant echo chamber that constantly tries to get you to open up your wallet. Isn’t the future great?