How Online Shopping Makes Suckers of Us All

As Christmas approached in 2015, the price of pumpkin-pie spice went wild. It didn’t soar, as an economics textbook might suggest. Nor did it crash. It just started vibrating between two quantum states. Amazon’s price for a one-ounce jar was either $4.49 or $8.99, depending on when you looked. Nearly a year later, as Thanksgiving 2016 approached, the price again began whipsawing between two different points, this time $3.36 and $4.69.

We live in the age of the variable airfare, the surge-priced ride, the pay-what-you-want Radiohead album, and other novel price developments. But what was this? Some weird computer glitch? More like a deliberate glitch, it seems. “It’s most likely a strategy to get more data and test the right price,” Guru Hariharan explained, after I had sketched the pattern on a whiteboard.

The right price—the one that will extract the most profit from consumers’ wallets—has become the fixation of a large and growing number of quantitative types, many of them economists who have left academia for Silicon Valley. It’s also the preoccupation of Boomerang Commerce, a five-year-old start-up founded by Hariharan, an Amazon alum. He says these sorts of price experiments have become a routine part of finding that right price—and refinding it, because the right price can change by the day or even by the hour. (Amazon says its price changes are not attempts to gather data on customers’ spending habits, but rather to give shoppers the lowest price out there.)

It may come as a surprise that, in buying a seasonal pie ingredient, you might be participating in a carefully designed social-science experiment. But this is what online comparison shopping hath wrought. Simply put: Our ability to know the price of anything, anytime, anywhere, has given us, the consumers, so much power that retailers—in a desperate effort to regain the upper hand, or at least avoid extinction—are now staring back through the screen. They are comparison shopping us.

They have ample means to do so: the immense data trail you leave behind whenever you place something in your online shopping cart or swipe your rewards card at a store register, top economists and data scientists capable of turning this information into useful price strategies, and what one tech economist calls “the ability to experiment on a scale that’s unparalleled in the history of economics.” In mid-March, Amazon alone had 59 listings for economists on its job site, and a website dedicated to recruiting them.

Not coincidentally, quaint pricing practices—an advertised discount off the “list price,” two for the price of one, or simply “everyday low prices”—are yielding to far more exotic strategies.

“I don’t think anyone could have predicted how sophisticated these algorithms have become,” says Robert Dolan, a marketing professor at Harvard. “I certainly didn’t.” The price of a can of soda in a vending machine can now vary with the temperature outside. The price of the headphones Google recommends may depend on how budget-conscious your web history shows you to be, one study found. For shoppers, that means price—not the one offered to you right now, but the one offered to you 20 minutes from now, or the one offered to me, or to your neighbor—may become an increasingly unknowable thing. “Many moons ago, there used to be one price for something,” Dolan notes. Now the simplest of questions—what’s the true price of pumpkin-pie spice?—is subject to a Heisenberg level of uncertainty.

Which raises a bigger question: Could the internet, whose transparency was supposed to empower consumers, be doing the opposite?

If the marketplace was a war between buyers and sellers, the 19th-century French sociologist Gabriel Tarde wrote, then price was a truce. And the practice of setting a fixed price for a good or a service—which took hold in the 1860s—meant, in effect, a cessation of the perpetual state of hostility known as haggling.

As in any truce, each party surrendered something in this bargain. Buyers were forced to accept, or not accept, the one price imposed by the price tag (an invention credited to the retail pioneer John Wanamaker). What retailers ceded—the ability to exploit customers’ varying willingness to pay—was arguably greater, as the extra money some people would have paid could no longer be captured as profit. But they made the bargain anyway, for a combination of moral and practical reasons.

The Quakers—including a New York merchant named Rowland H. Macy—had never believed in setting different prices for different people. Wanamaker, a Presbyterian operating in Quaker Philadelphia, opened his Grand Depot under the principle of “One price to all; no favoritism.” Other merchants saw the practical benefits of Macy’s and Wanamaker’s prix fixe policies. As they staffed up their new department stores, it was expensive to train hundreds of clerks in the art of haggling. Fixed prices offered a measure of predictability to bookkeeping, sped up the sales process, and made possible the proliferation of printed retail ads highlighting a given price for a given good.

Companies like General Motors found an up-front way of recovering some of the lost profit. In the 1920s, GM aligned its various car brands into a finely graduated price hierarchy: “Chevrolet for the hoi polloi,” Fortune magazine put it, “Pontiac … for the poor but proud, Oldsmobile for the comfortable but discreet, Buick for the striving, Cadillac for the rich.” The policy—“a car for every purse and purpose,” GM called it—was a means of customer sorting, but the customers did the sorting themselves. It kept the truce.

Customers, meanwhile, could recover some of their lost agency by clipping coupons—their chance to get a deal denied to casual shoppers. The new supermarket chains of the 1940s made coupons a staple of American life. What the big grocers knew—and what behavioral economists would later prove in detail—is that while consumers liked the assurance the truce afforded (that they would not be fleeced), they also retained the instinct to best their neighbors. They loved deals so much that, to make sense of their behavior, economists were forced to distinguish between two types of value: acquisition value (the perceived worth of a new car to the buyer) and transaction value (the feeling that one lost or won the negotiation at the dealership).

The idea that there was a legitimate “list price,” and that consumers would occasionally be offered a discount on this price—these were the terms of the truce. And the truce remained largely intact up to the turn of the present century. The reigning retail superpower, Walmart, enforced “everyday low prices” that did not shift around.

But in the 1990s, the internet began to erode the terms of the long peace. Savvy consumers could visit a Best Buy to eyeball merchandise they intended to buy elsewhere for a cheaper price, an exercise that became known as “showrooming.” In 1999, a Seattle-based digital bookseller called Amazon.com started expanding into a Grand Depot of its own.

The era of internet retailing had arrived, and with it, the resumption of hostilities.

In retrospect, retailers were slow to mobilize. Even as other corporate functions—logistics, sales-force management—were being given the “moneyball” treatment in the early 2000s with powerful predictive software (and even as airlines had fully weaponized airfares), retail pricing remained more art than science. In part, this was a function of internal company hierarchy. Prices were traditionally the purview of the second-most-powerful figure in a retail organization: the head merchant, whose intuitive knack for knowing what to sell, and for how much, was the source of a deep-seated mythos that she was not keen to dispel.

Two developments, though, loosened the head merchant’s hold.

The first was the arrival of data. Thomas Nagle was teaching economics at the University of Chicago in the early 1980s when, he recalls, the university acquired the data from the grocery chain Jewel’s newly installed checkout scanners. “Everyone was thrilled,” says Nagle, now a senior adviser specializing in pricing at Deloitte. “We’d been relying on all these contrived surveys: ‘Given these options at these prices, what would you do?’ But the real world is not a controlled experiment.”

The Jewel data overturned a lot of what he’d been teaching. For instance, he’d professed that ending prices with .99 or .98, instead of just rounding up to the next dollar, did not boost sales. The practice was merely an artifact, the existing literature said, of an age when owners wanted to force cashiers to open the register to make change, in order to prevent them from pocketing the money from a sale. “It turned out,” Nagle recollects, “that ending prices in .99 wasn’t big for cars and other big-ticket items where you pay a lot of attention. But in the grocery store, the effect was huge!”

The effect, now known as “left-digit bias,” had not shown up in lab experiments, because participants, presented with a limited number of decisions, were able to approach every hypothetical purchase like a math problem. But of course in real life, Nagle admits, “if you did that, it would take you all day to go to the grocery store.” Disregarding the digits to the right side of the decimal point lets you get home and make dinner.

By the early 2000s, the amount of data collected on retailers’ internet servers had become so massive that it started exerting a gravitational pull. That’s what triggered the second development: the arrival, en masse, of the practitioners of the dismal science.

This was, in some ways, a curious stampede. For decades, academic economists had generally been as indifferent to corporations as corporations were to them. (Indeed, most of their models barely acknowledged the existence of corporations at all.)

But that began to change in 2001, when the Berkeley economist Hal Varian—highly regarded for the 1999 book Information Rules—ran into Eric Schmidt. Varian knew him but, he says, was unaware that Schmidt had become the CEO of a little company called Google. Varian agreed to spend a sabbatical year at Google, figuring he’d write a book about the start-up experience.

At the time, the few serious economists who worked in industry focused on macroeconomic issues like, say, how demand for consumer durables might change in the next year. Varian, however, was immediately invited to look at a Google project that (he recalls Schmidt telling him) “might make us a little money”: the auction system that became Google AdWords. Varian never left.

Others followed. “eBay was Disneyland,” says Steve Tadelis, a Berkeley economist who went to work there for a time in 2011 and is currently on leave at Amazon. “You know, pricing, people, behavior, reputation”—the things that have always set economists aglow—plus the chance “to experiment at a scale that’s unparalleled.”

At first, the newcomers were mostly mining existing data for insights. At eBay, for instance, Tadelis used a log of buyer clicks to estimate how much money one hour of bargain-hunting saved shoppers. (Roughly $15 was the answer.)

Then economists realized that they could go a step further and design experiments that produced data. Carefully controlled experiments not only attempted to divine the shape of a demand curve—which shows just how much of a product people will buy as you keep raising the price, allowing retailers to find the optimal, profit-maximizing figure. They tried to map how the curve changed hour to hour. (Online purchases peak during weekday office hours, so retailers are commonly advised to raise prices in the morning and lower them in the early evening.)

By the mid-2000s, some economists began wondering whether Big Data could discern every individual’s own personal demand curve—thereby turning the classroom hypothetical of “perfect price discrimination” (a price that’s calibrated precisely to the maximum that you will pay) into an actual possibility.

As this new world began to take shape, the initial consumer experience of online shopping—so simple! and such deals!—was losing some of its sheen.

It’s not that consumers hadn’t benefited from the lower prices available online. They had. But some of the deals weren’t nearly as good as they seemed to be. And for some people, glee began to give way to a vague suspicion that maybe they were getting ripped off. In 2007, a California man named Marc Ecenbarger thought he had scored when he found a patio set—list price $999—selling on Overstock.com for $449.99. He bought two, unpacked them, then discovered—courtesy of a price tag left on the packaging—that Walmart’s normal price for the set was $247. His fury was profound. He complained to Overstock, which offered to refund him the cost of the furniture.

But his experience was later used as evidence in a case brought by consumer-protection attorneys against Overstock for false advertising, along with internal emails in which an Overstock employee claimed it was commonly known that list prices were “egregiously overstated.”

In 2014, a California judge ordered Overstock to pay $6.8 million in civil penalties. (Overstock has appealed the decision.) The past year has seen a wave of similar lawsuits over phony list prices, reports Bonnie Patten, the executive director of TruthinAdvertising.org. In 2016, Amazon began to drop most mentions of “list price,” and in some cases added a new reference point: its own past price.

This could be seen as the final stage of decay of the old one-price system. What’s replacing it is something that most closely resembles high-frequency trading on Wall Street. Prices are never “set” to begin with in this new world. They can fluctuate hour to hour and even minute to minute—a phenomenon familiar to anyone who has put something in his Amazon cart and been alerted to price changes while it sat there. A website called camelcamelcamel.com even tracks Amazon prices for specific products and alerts consumers when a price drops below a preset threshold. The price history for any given item—Classic Twister, for example—looks almost exactly like a stock chart. And as with financial markets, flash glitches happen. In 2011, Peter A. Lawrence’s The Making of a Fly (paperback edition) was briefly available on Amazon for $23,698,655.93, thanks to an algorithmic price war between two third-party sellers that had run amok. To understand what happened, it seemed sensible to talk to the man who helped develop the software they were using.

Guru Hariharan uncapped a dry-erase marker in a conference room at Boomerang’s headquarters in Mountain View, California. He was talking about what had led retailers to this desperate place where it’s necessary to change prices multiple times a day. On a whiteboard, he drew a series of lines representing the rising share of online sales for various kinds of products (books, DVDs, electronics) over time, then marked the years that major brick-and-mortar players (Borders, Blockbuster, Circuit City and RadioShack) went bankrupt. At first the years looked random. But the bankruptcies all clustered within a band where online sales hit between 20 and 25 percent. “In this range, there’s a crushing point,” Hariharan said, clapping his hands together for emphasis. “There’s a bloodbath happening.”

Beyond this crushing point, traditional retailers with both a brick-and-mortar and an online presence feel compelled to compete purely on price. Hariharan talked wistfully of the days when he’d walk into RadioShack and have a salesperson direct him to the exact connector cable he needed. But once retailers enter the crushing zone, expenses like staff, training, and customer support typically are slashed. Profit margins keep falling nonetheless—why go to the store at all if no one there can help you?—and a death spiral ensues. (RadioShack traced just this path before filing for bankruptcy in 2015.)

“It didn’t have to be that way,” Hariharan said. Now he’s helping retailers fight back.

We can’t process every piece of price information thrown our way. So we judge a store’s prices based on a handful of products we know well. Grocers have recognized this for decades, which is why they keep the price of eggs and milk consistently low, making their profits on other goods whose markups we don’t notice as easily.

When he was at Amazon, Hariharan, who has a degree in machine learning, helped invent and patent the Amazon Selling Coach, a system that helps third-party vendors optimize their inventory and prices. He and his team at Boomerang have built a massive system that tracks prices and has informed billions of pricing decisions for clients ranging from Office Depot to GNC to U.S. Auto Parts. But its software engine isn’t built to match the lowest price out there. (That, Hariharan notes, would be a simple algorithm.) It’s built to manage consumers’ perception of price. The software identifies the goods that loom largest in consumers’ perception and keeps their prices carefully in line with competitors’ prices, if not lower. The price of everything else is allowed to drift upward.

Amazon long ago mastered this tactic, Hariharan says. In one instance, Boomerang monitored the pricing shifts of a popular Samsung television on Amazon over the six-month period before Black Friday. Then, on Black Friday itself, Amazon dropped the TV’s price from $350 all the way to $250, undercutting competitors by a country mile. Boomerang’s bots also noticed that in October, Amazon had hiked the price of some HDMI cables needed to connect the TV by about 60 percent, likely armed with the knowledge, Hariharan says, that online consumers do not comparison shop as zealously for cheaper items as they do for expensive ones.

What’s interesting is how other retailers are now beginning to adapt. To show me this, a Boomerang employee opened up the dashboard seen by the firm’s clients. Scrolling through a menu of premade algorithms, he selected a rule, “Beat Competitor by 10%,” for certain items meeting the following criteria:

That is: If the competitor’s price is greater than the cost of making the item, and the competitor isn’t running a onetime promotion, then undercut the competitor by 10 percent. The rule was implemented with a click, and onscreen, I could see a healthy drop in the client’s Price Perception Index.

But that’s not the end of the story. The price cuts will register on competitors’ pricing sonars. Whether or not to respond in kind depends, in part, on how their algorithms interpret the signal. Is this the first shot in a pricing war? Or is the retailer just trying to clear inventory from its warehouse? In practice, it’s hard to tell. So an innocuous, temporary price cut may set off a machine-against-machine price war that, if left unchecked, could quickly devastate a retailer’s bottom line. Boomerang clients are prompted to select “Guardrails”—further rules that provide a check on the initial set of rules—and establish a certain amount of human oversight. Faisal Masud, the chief technology officer at Staples, one of Boomerang’s first customers, thinks human involvement makes sense only in rare cases. “We want to make sure the software makes the decisions, not the human being,” he says. “It’s all automatic. Otherwise you’re losing.”

The complexity of retail pricing today has driven at least one of Boomerang’s clients into game theory—a branch of mathematics that, it’s safe to say, has seldom found practical use in shopping aisles. Hariharan says, with a smile: “It lets you say, ‘What is the dominant competitor’s reaction to me? And if I know the reaction to me, what is my first, best move?’ Which is the Nash equilibrium.” Yes, that’s John Nash, the eponymous Beautiful Mind, whose brilliant contributions to mathematics now extend to the setting of mop prices.

Where does all this end?

One scenario is: in simplicity.

The apparel start-up Everlane, for instance, is betting that it can capitalize on consumer backlash to retailers’ ever more vaguely underhanded tactics. The company spells out the cost of making each of its products and the profit it earns on each. Recently it informed customers that the cost of cashmere from Inner Mongolia had dropped. It was dropping the price of its cashmere sweaters by $25, because they now cost less to make. “Radical transparency,” Everlane founder and CEO Michael Preysman calls the approach.

On another occasion, Everlane decided to clear clothing and shoe inventory by giving customers three choices of what to pay. The lowest price covered the cost of making and shipping the items. The middle price also covered the overhead of selling them. And the highest provided Everlane a profit.

Lest someone wonder, Would framing price as a moral dilemma be the ultimate pricing ploy?, the answer is no: 87 percent of customers chose the lowest price, Preysman reports. (Eight percent picked the middle price; 5 percent chose the highest.) The point, Preysman stresses, was to give customers a glimpse of how stuff gets made, how workers get paid, and other things not typically visible on a shoebox or a sweater tag.

“The theory of Everlane, I think, is still a theory we have to prove,” Preysman says. Companies have “trained customers in the U.S. to be as addicted to sales as possible. It has become a core piece of the retail-industrial complex and it is very, very difficult to unwind. So reeducation is hard when you play in a market where people play these games on a daily basis.”

But a different scenario follows from the possibility that consumers don’t really want clarity. They are content to be fooled into paying more if they can keep the belief that they’re paying less; that they have the agency and agility to find special, unbeatable deals, only for them. This would amount to a rejection of the new truce that Everlane is extending. And it would open the way for retailers and economists to grab their holy grail.

Perfect price discrimination was, again, supposed to exist only as a classroom thought experiment. But it posits that a seller knows the walk-away price of every single buyer and hence, by offering a price just barely below it, can extract every last farthing of potential profit from each of them.

In the past, retailers have used demographic data to try to deduce walk-away price. In 2000, some people thought Amazon was doing this when customers noticed they were being charged different prices for the same DVDs. Amazon denied it. This was the result of a random price test, CEO Jeff Bezos explained in a news release. “We’ve never tested and we never will test prices based on customer demographics.”

But demographics are actually a crude way of personalizing prices, the Brandeis economist Benjamin Shiller argued in a recent paper, “First-Degree Price Discrimination Using Big Data.” If Netflix were to use only demographic factors, such as people’s race, household income, and zip code, to personalize subscription prices, his model predicted, it could boost its profits by 0.3 percent. But if Netflix also used people’s web-browsing history—the percentage of web use on Tuesdays, the number of visits to RottenTomatoes.com, and some 5,000 other variables—it could boost its profits by 14.6 percent.

Netflix was not doing any of this; it hadn’t even provided Shiller with the data he used (which he obtained from a third party). But Shiller demonstrated that personalized pricing was feasible.

Are other companies doing this? Four researchers in Catalonia tried to answer the question with dummy computers that mimicked the web-browsing patterns of either “affluent” or “budget conscious” customers for a week. When the personae went “shopping,” they weren’t shown different prices for the same goods. They were shown different goods. The average price of the headphones suggested for the affluent personae was four times the price of those suggested for the budget-conscious personae. Another experiment demonstrated a more direct form of price discrimination: Computers with addresses in greater Boston were shown lower prices than those in more-remote parts of Massachusetts on identical goods.

In their paper, “Detecting Price and Search Discrimination on the Internet,” the researchers suggested that consumers could benefit from a price-discrimination watchdog system that would continuously monitor for customized prices (although it’s unclear who would build or operate this). Another paper—this one co-authored by Google’s Hal Varian—argues that if personalized pricing becomes too aggressive, shoppers will become more “strategic,” selectively withholding or disclosing information in order to obtain the best price.

Which, to Bonnie Patten of TruthinAdvertising.org, seems like a whole lot of work. It’s already “so complicated,” she told me. “Everything is 50 percent off, but they have all these exclusions where it doesn’t count, and then everyone is trying to calculate 20 percent of 50 percent in their heads.” She already has a full-time job, was her point. And three kids.

“As a general matter,” she went on, “I find it so difficult to determine the actual price of the product that when I’m shopping for my kids, my new technique is to make all my decisions at the cashier. I pick up lots of clothes. I completely ignore all pricing until I get to the register. And then if something is too much, I say, ‘I don’t want it.’ ”

This struck me as sensible in the extreme. And how did she shop for herself?

“I do not shop,” Patten said.

In what sense?, I asked, confused.

“I just gave up,” she said. “I just stopped shopping.”

I thought about this after we hung up. Maybe it was a function of her job, which let her see too much. Maybe she was a certain type—“survival shopper” was the label she used—who simply didn’t experience the thrill of finding a pair of $30 moccasins for $8. Such thoughts helped stay the alternative explanation, the one Gabriel Tarde called “the madness of doubt”: that there’s a finite amount of uncertainty we can absorb, a limit to how much we can check the ticker to see whether the Swiffer’s price is up or down this morning; that somewhere in us is a shut-off point, and that Patten had hit it.