One-Way-Mirror Society: High Privacy Risk Consumer Tracking Technologies

Report home | Read the report (PDF) | Previous section | Next section

Facial Recognition

Facial recognition technology was initially developed for security purposes, but it has found a new use in digital signage for marketing and ad targeting purposes. Essentially, the process is that a camera captures an individual’s image, then checks it against algorithms that analyze at least 80 facial characteristics, such as distance between eyes, length of the face, width of the face, depth of eye sockets, and so forth. [48] Layers of algorithms are used to crunch the facial information into determinations about a person’s age bracket, gender, and ethnicity. The next efforts are going toward coding the facial expressions of shoppers to “capture their emotional reactions to in-store environments.” [49]

The video stream from the camera capturing the facial data is sent to a computer with a face-tracking engine that registers the number of viewers in front of the screen and can even determine whose eyes actually looked at the screen. Some software packages can also determine the gender, age, and ethnicity of the viewers. [50]

Audience Surveillance and Measurement for Marketing

One of the primary selling points for those wanting to deploy digital signage is that the screens are not just a one-way technology going from screen to consumer. The most advanced digital signage installations have screens concealing a host of technologies that gather information from the rooms they are placed in and the people who come within view of the screens, and then respond accordingly, often instantly. Digital signs can record the customers near them, monitor room temperature, check carbon dioxide levels, and more. For example, it is now an unremarkable feature for a digital signage installation to show ads targeted to the specific gender or age of a person looking at the screen as the person is standing in front of it.

To accomplish this, digital signs are equipped with sensors and/or cameras or webcams built directly into the screen, [51] that can capture and record large amounts of information about who is looking at the sign, for how long, and at what time of the day. Then sophisticated video analytics create a demographic profile of the gender, age, and ethnicity among other characteristics. In some cases, multiple cameras are used, including cameras outside the screens. As seen in Figure 3, cameras can be tucked inconspicuously into end cap displays, on ceilings, and elsewhere.

Figure 3
Video analysis technologies exist in many retail and other environments. People looking at digital screens can have their images captured by a sensor or camera in or near the screen, then be analyzed by facial recognition technology. The cameras may be miniature and difficult to detect.

It is important to remember that digital signage networks can involve an entire video architecture, one that includes existing security cameras. The audience measurement ecosystem may also use other shopper measurement systems in addition to the digital signage. [52]

Technologies that Measure Ethnicity, Age, and Gender

While it may come as a substantial surprise to consumers, it is a current business practice to use advanced video analysis technologies to determine a consumer’s age, gender, and sometimes ethnicity to target ads and marketing directly to a particular customer. This technology is not new, but it did reach a maturation point in 2008/2009. Often called advanced audience measurement features, or advanced video analytics, the technology is used to determine a customer’s ethnicity, gender, and age using facial recognition software and other techniques. [53] The technology has reportedly reached about a 90 percent accuracy rate.

Initially, the technology began as simple gaze tracking, but expanded into the demographic uses. [54] Cognovision, one company selling this technology, states in its materials that it measures five areas of consumer behavior and characteristics:

• Actual Impressions – The number of people who look at your displays
• Length of Impressions – How long people look for
• Potential Audience Size – The number of people who walk by
• Dwell Time – How long people stay near your displays
• Anonymous Demographics – Demographics of your audience (gender and age bracket) [55]

The point of creating demographic profiles is twofold: one, to determine how many people are watching the ad on the digital signage, and what ages, genders, and ethnicities they are; and two, to target the advertising based on that information. [56]

The ultimate goal is to have digital signs that change content based on the characteristics of the people standing right in front of the display:

“[A]dvanced consumer demographics…will enable dynamic message selection on the digital sign and the ability to vary content based on viewer characteristics including gender, ethnicity, and banded age group.” [57]

One example of this nexus can be seen in TargetScent, a kiosk-style “gender aware” fragrance dispenser. The kiosk/dispenser uses a small computer running Quividi facial recognition software. A camera in the display detects human faces in the vicinity of the display, estimates the corresponding gender of prospective customers and sends that information to the fragrance dispenser, choosing one of four fragrances based on the facial analysis. The units were introduced in 2009 in Europe. [58]

A more generalized example of this can be seen in the Whole Foods installation of the Marketplace Station digital signage network in Chicago in the U.S. and in Canada. A 2008 press release about the program, which rolled out first in Canada, described how the digital signage stations would benefit consumers with product information and food and lifestyle ideas. There is also a one-sentence description in the press release that hints at the fact the signs are equipped with advanced video analytics:

A software application will also be in place to comprehend viewer metrics of each digital station, for hands-on tactical management of campaigns from start to finish. [59]

In May of 2009, the Marketplace Station digital signage network was deployed in the Chicago Whole Foods store. Beginning in March 2010, a press release notes that a new kiosk system will be added and consumer analytics will be captured. The new digital program is described as being capable of deriving data from actual audience viewership captured through an anonymous analytics sensor. The press release goes on to state that they will be reporting on the “gender of impressions.” [60] Past analytics reports on the company’s web site reveal that gender is indeed being analyzed at the Whole Foods stores through the facial recognition capabilities of its digital signage network.

The Whole Food’s privacy policy makes no mention of its digital signage network. The Marketplace Station made no mention of its facial recognition software in its privacy policy. There is though, a YouTube video about Intel chips that highlights the Whole Foods/Marketplace Station digital signage installation, complete with an explanation of how the advanced analytics captures the gender of people looking at the signs. The video even shows a person shopping at the Whole Foods store looking up at the screens and being analyzed for demographic characteristics. [61] (Figure 3). Only the individual who looks at the video will have any real idea what is happening with Whole Foods digital signage behind the scenes.

Figure 4.
A screen shot of a YouTube video showing how the Whole Food’s Marketplace Station digital signs are using facial recognition technology to analyze the gender of customers shopping at certain stores.

One of the issues this digital signage installation brings up is that of the digital signage industry’s view of privacy and image capture and storage. One company that sells advanced video analytics, [62] TruMedia, has adopted a self-imposed standard that no images or “personally identifying information” will be stored without consumer consent. This is a frequently encountered refrain; that consumer privacy is protected because images are not stored by a particular digital signage system. [63]

TruMedia states in its privacy policy:

Images from our sensors are processed and converted in real-time into counts (how many) and durations (how long). Using complex proprietary algorithms these counts are further assigned to specific demographic categories such as gender and age-group. No images are ever and will ever be stored for use, review or sharing with any private or governmental body. [64]

Here, the line is drawn at the retention or storage of the data. But the data is still captured, analyzed, and used without consumer consent and very likely without meaningful con sumer knowledge. Another argument often encountered is that images are not recorded, therefore privacy is protected:

CognoVision’s Anonymous Impression Metric (AIM) technology uses face-detection and people counting technology to measure the effectiveness of digital signage, and enables real- time content targeting based on audience characteristics, allowing for truly measured and targeted delivery of media. The system has been designed to completely respect privacy – no personally identifiable information is ever collected, and no images are ever recorded. [65]

The Marketplace Station is using CognoVision’s AIM technology, which means that the images of shoppers are not supposed to be recorded. However, just because the companies have decided that the lack of storage or recording of the data is equivalent to privacy does not mean that consumers should be left in the dark about such technologies. And it does not mean that customers in these stores should be subject to this activity without consenting to it. There is tremendous uncertainty about where these cameras are deployed in screens, if the images are being recorded, what information is actually kept, and how the consumer consent process is supposed to work. Of course, current limits on data collection and retention are subject to change without notice to the public. Indeed, entire systems operate without any notice to the public.

Some in the industry have raised privacy concerns about the deployment of these technologies, noting that simple gaze tracking was not as much of an issue as the demographic profiling and targeting.

The technologies that enable this are originally intended for shopper gaze tracking, allowing retailers to understand how many people walked by a screen or display, how many looked, at what and for how long. This is exciting, as it can open the door to real-time analytics that allow us to respond according to what works — and what doesn’t.

The issue at hand is that some of the firms behind this technology can also “flip the switch” to track shopper demographics such as age, ethnicity and sex. Conceptually, the idea is to “auto serve” content geared towards the type of shopper walking by and ensure that it’s as relevant as possible. [66]

Mobile Marketing and Customer Loyalty programs Linked to Digital Signage

Advanced digital signage networks can be tied to loyalty programs. One early method of tracking customer behavior in stores was to use tracking devices attached to shopping carts and then linked to customer loyalty programs.

Several companies have developed tracking systems that use RFID, GPS, or infrared sensors attached to shopping carts, hand baskets, or hand-held shopping devices to track the customer’s path through the store. These systems can provide reliable information on the shopping process, and the data are easily linked to individual-level customer transaction and loyalty information. [67]

But this customer tracking model has a significant drawback: if a shopper does not use a cart or a tracking device, then the consumer tracking fails. A more modern approach is to use digital signage as a bridge between the retailer and consumer in an opt-in program. One example of this model is Hot Topic, a retailer, that has deployed 1,500 in-store kiosks and digital signs which are linked with the store’s customer loyalty program. [68]

Figure 5
Hot Topic’s digital signage kiosks that link to its customer loyalty program. The kiosks contain a lens that looks back at customers.

To sign up for the Hot Topic program, customers interact with the kiosks and type in their name, date of birth, email address, mobile phone provider, mobile phone number, address, gender, and other details. The kiosk screens themselves do not appear to link to a privacy policy or the Terms and Conditions of the loyalty program. Instead, the kiosks have a FAQ section, but not a detailed privacy notice. The kiosks have a camera and lens embedded in them, (Figure 3) but it is not disclosed in any notice, nor is what the cameras are potentially capturing, recording and/or analyzing discussed in a written policy.

Nevertheless, the Hot Topic web site Terms and Conditions contains the following paragraph:

REGISTRATION: To participate in, you must create a member account (“account”) by registering your information with Hot Topic in a Hot Topic store, on our kiosks, or on the Web Site. You must have a valid email address to receive offers and other Program benefits. One email address per account. It is your responsibility to read the full Terms and Conditions at the time that you register. By providing the required information to Hot Topic and creating an account, you’re confirming that you’ve read and agreed to the Terms and Conditions. (emphasis added)

One of the substantive issues with the majority of digital signage in place today is a lack of meaningful notice. Hot Topic terms and conditions “require” a consumer to read the full Terms and Conditions at the time of registration at an in-store kiosk, but it is well known that consumers rarely read notices. Indeed, a kiosk operator can easily check to see if the notice was read, but operators are not likely to do so because they would rather rely on the fiction that consumers have knowledgably consented.

As seen in the Hot Topic loyalty program, a substantial linkage can exist between the digital signage industry and mobile advertising for cell phones. Current examples are primarily opt-in, with customers taking the first step to give a retailer or business a mobile number or an email.

Another example of how this can work is Hungry Howie’s pizza in Clearwater Florida. Digital signs sit in the restaurant location. The Hungry Howie’s digital signage does not report back via cameras, instead, the signage focus is on interactively acquiring customer’s mobile phone numbers via a touch screen. After a customer enters a mobile phone number on the screen, customers then receive a text message on their mobile phones. Upon a second opt-in, customers then receive coupons and other SMS texts via their mobile phones. [69] Customers are given the opportunity to opt out of the program.

Another digital signage-mobile example may be found in the campaigns of the company MegaPhone. MegaPhone uses trucks to host large portable digital billboards. Various interactive games run on the digital signage, which require mobile phone interaction to play. [70]

The company states in a brochure discussing case studies:

Megaphone tracks all interactions and outcomes while aggregating data for each unique caller. Based on GPS call location, time stamping, cal length , buttons pressed, bounces, sharing, word of mouth, drop triggers, and mobile channel content engagement we can define a psychographic profile of your consumer. [71]

At the 2008 NBA All-Star Game in New Orleans, Adidas ran a campaign with MegaPhone. A portable digital sign on a large truck hosted a game called “3 stripe throw down.” To play, people called a phone number shown on the game billboard. According to the case study written by Megaphone, the objective of the campaign was to add the players to a mobile mailing list, and to drive foot traffic to local Adidas stores. Callers to the Adidas game receive a message with walking instruction to the Adidas store closest to the game location. The callers also have the opportunity to opt in to an Adidas SMS mailing list for special events during the all-star weekend. The Philadelphia 76ers ran a similar campaign.

While the campaign allowed for choice, a significant question to ask is if the people who called the phone number in order to play the digital billboard game had any idea that a third party was tracking their gaming interactions, aggregating data for each unique caller, and “defining a psychographic profile” of them. It is unclear how long consumer data was stored, and it is also unclear if the data was mingled with other identifiable consumer information.

MegaPhone did not have a privacy policy posted that described its privacy and data practices. The Adidas privacy policy addressed SMS programs in broad terms. [72] From a consumer perspective, it would have been a challenge for a consumer to meaningfully understand from the information available to them how their data would be handled and passed along.

________________________________________

Endnotes

[48] Manolo Almagro, Quividi’s Digital Sex Change Feature, DailyDOOH , March 2, 2009 <http://www.dailydooh.com/archives/8887/print/>.

[49] Raymond R. Burke, Retail Shoppability: A Measure of the Worlds Best Stores, 2005. p. 13. Originally published in Future Retail Now: 40 of the World’s Best Stores, Retail Industry Leaders Association. <http://kelley.iu.edu/features/archive/fall_2007/shoppability2.html>.

[50] Bill Yackey, The Push for Digital Signage Metrics, June 9, 2008, Digital Signage Today, <http://www.digitalsigngaetoday.com/article.php?id=20004>.

[51] Vendco Introduces Screens with Integrated Audience Measurement, July 30, 2008, <http://www.digitalsignagetoday.com/article.php?id=20284>.

[52] See for example ShopperGauge. “ShopperGauge is an in-store monitoring system that delivers continuous reporting of REAL shopper behavior.” Its website states: “24/7 digital monitoring by a strategically installed camera measures body language. Interpretive software reports traffic, dwell time, and shopper engagement with the display or shelf.” The web site notes that the shopper data is live. ShopperGauge, < http://www.shoppergauge.com/how-shoppergauge-works>.

[53] See for example audience measurement products by Quividi <http://www.quividi.com/>, Cognovision <http://www.cognovision.com/solutions.php>, Tru-Media <http://www.tru-media.com/>, and Wututu <http://www.wututu.com/en/>.

[54] Laura Davis-Taylor, The In Store Shopper Profiling Debate, May 20, 2008, POPAI <http://www.popaidigitalblog.com/blog/articles/The_in_store_shopper_profiling_debate-439.html>.

[55] Cognovision Solutions page, <http://www.cognovision.com/solutions.php>.

[56] Steve Arel, Video Analytics for Digital Signage Deployments, White Paper, <http://DigitalSignageToday.com>, 2009. White Paper sponsored by Intel. “Various forms of analytics exist, breaking down consumer activity and behavior. The video version gives businesses a more detailed look at individuals who come in contact with digital signage. Through cameras installed on and integrated into monitors, software can show everything from the length of time someone watches an ad or message to exactly who watches, and correlate the effectiveness of those spots.”

[57] NRF: STRATACACHE debuting audience measurement tech, Jan. 13, 2009, <digitalsignagetoday.con/article.php?id=21409>.

[58] Quividi Press Release, Presensia and Quividi Invent the First Gender-Aware Fragrance Dispenser, January 7, 2009. <http://www.quuividi.com/news/090107/pressreleases>.

[59] The Marketplace Station Introduces In-Store Digital Stations at Whole Foods Market to help Marketers and Consumers, November 17th, 2008. <http://www.themarketplacestation.com/files/news/51.pdf>.

[60] 30 Retailers One Digital Network, January 21, 2010. <http://www.themarketplacestation.com/files/news/55.pdf>.

[61] Digital Signage Innovations- Intel Technology in Digital Signage. July 22, 2009. YouTube, <http://www.youtube.com/watch?v=ibAFvT_Vr7s>.

[62] <http://www.themarketplacestation.com/files/analytics/analytics-january-2009.pdf>.

[63] See for example: NRF: STRATACACHE debuting audience measurement tech, Jan. 13, 2009, <digitalsignagetoday.con/article.php?id=21409>

[64] TruMedia Audience Measurement Systems Privacy Policy, <http://www.tru-media.com/inside.asp?ID=18>.

[65] Dynasign Integrates CognoVision Audience Measurement Technology, Feb. 24, 2009. <http://Digital SignageToday.com/article.php?id=21733>.

[66] Laura Davis-Taylor, The In-Store Profiling Debate, May 20, 2008. <http://www.popaidigitalblog.com/blog/articles/The_in_store_shopper_profiling_debate-439.html>.

[67] Raymond R. Burke, The Third Wave of Marketing Intelligence, Kelley School of Business, Indiana University, p. 112.

[68] Joab Jackson, IT Firms Promote Interactive Digital Signs at Retail Show, PC World, Jan. 14, 2010. <http://www.pcworld.com/printable/article/id,186959/printable.html>. See also Hot Topic Implements NCR Netkey Self- Service , Jan. 12, 2010. <http://www.digitialsignageexpo.net/DNNArticleView/tab…ents-NCR-Netkey-Self-Service-Kiosks- and Digital-Signage/Default.aspx>.

[69] Case study: Hungry Howie’s Pizza, Inc., July 31, 2009, Digital Signage Expo.net, < http://www.digitalsignageexpo.net/DNNArticleMaster/DNNArticleView/tabid/78/smid/1043/ArticleID/1667/reftab/70/Defa ult.aspx>. For a video example of how the opt in and mobile response works and looks, see < http://www.sundropsystems.com/Products/loyaltxt.aspx#>.

[70] MegaPhone describes itself as “Making Digital Signage Interactive” on its home page. “MegaPhone is a Phonecall- Controlled, Real-Time, Multi-Player Collaborative Gaming Platform for Big Screens in Public Spaces.” MegaPhone home page, <http://www.playmegaphone.com/>.

[71] MegaPhone Case Studies, <http://www.playmegaphone.com/documents/MegaPhone_InfoDocs.pdf>. See also video of the Adidas game, YouTube, MegaPhone, Sept. 8, 2008 <http://www.youtube.com/watch?v=29RSh_4A16s>.

[72] Adidas privacy policy, <http://www.adidas.com/us/shared/legal.asp#Link8>.

Roadmap: The One-Way-Mirror Society – Privacy Implications of the new Digital Signage Networks: IV. High Privacy Risk Consumer Tracking Technologies

Report home | Read the report (PDF) | Previous section | Next section