Choosing Innovation Metrics for Success in the Digital Era

He set the stage with a basic question: Why measure innovation? The answer is that if a company wants to put money into developing new products, services, or business models, its executives need to assess the effectiveness of its current approach to innovation. Moreover, such metrics yield valuable information that can be used to incentivize managers and to hold them accountable. Above all, said Chaudhuri, having reliable innovation metrics helps us set strategic direction, including determining how to evaluate new ideas and allocate resources.

But measuring innovation can be anything but straightforward, he said. For instance, many companies rely on easily-obtained financial metrics such as revenue growth or percentage of sales from new products, but it’s difficult to tie those exclusively to specific innovations or innovative processes. “We often measure what’s available [to measure,] as opposed to what’s needed,” he observed. Chaudhuri noted that the Mack Institute is actively engaged in developing new, more effective approaches to measuring innovation, which will be published in a future report.

Co-director Harbir Singh emphasized the key role of innovation in today’s business environment. “If you look at the top ten companies in market cap today, six of them did not exist in the mid-eighties. Many big ones have fallen out of the [ranking] and been replaced by firms that have tremendous income coming from essentially intellectual property. So the stakes are extraordinarily high.”

Anne Marie Knott on How to Measure Innovation

As a project engineer at Hughes Aircraft in the 1990s, Anne Marie Knott was dismayed when the firm was acquired by General Motors and the new owners started reorganizing Hughes’ R&D functions. “I said, ‘this is terrible, it’s going to permanently degrade our R&D capability,” Knott recalls. But she couldn’t convince anyone else. Why? Hughes had no reliable, agreed-upon way to measure its R&D success, hence no way to demonstrate it was deteriorating.

Fast forward to today, and businesses are still seeking the holy grail of meaningful innovation metrics. Knott, now a business professor and corporate advisor, presented the attendees with new and existing measures of R&D productivity.

Knott observed that there are dozens of known ways to measure innovation. However, “there aren’t uniform standards; there’s concern that you can game the measures so one group gets more resources than another; and perhaps most importantly, they’re not meaningful to investors.”

A convenient measure to which many firms turn is the number of patents issued, Knott said. However, she pointed out, only 50% of firms that engage in R&D file any patents, and of those that do, only 40% file in a given year. So the usefulness of that metric is limited.

Knott led the participants through several commonly used innovation measures. One is the Vitality Index, which tallies the percentage of revenues from new products launched in the past three years. But the definition of “new” can vary, and this approach also fails to measure process innovation and incremental innovation. Another one is the McKinsey R&D Productivity Measure, which projects the expected value of a company’s R&D initiatives, and tracks progress to date divided by the amount of R&D spending. “What’s the big problem with that one?” asked Knott. “It’s a forecast, [not a measure], and R&D forecasts are notoriously unreliable.”
Knott also discussed the Holt Innovation Premium method, the basis for Forbes’ well-known “The World’s Most Innovative Companies.” A firm’s innovation is measured by its market cap minus the net present value of forecasted cash flows. She noted that a major caveat is stated by the authors themselves: there’s no correlation between their method and actual returns to investors.

The “RQ” or Research Quotient measurement is Knott’s contribution to the field, based in part upon a Nobel Prize-winning theory by economist Paul Romer. “It’s intended to be a company equivalent of individual IQ… Just as smart individuals solve more problems per minute, smart firms solve more technical problems per dollar of R&D.” Knott stated that the RQ approach possesses the most important properties for an innovation measure: it’s reliable, universal, and uniform. She said that based on nearly 50 years of research, increasing RQ increases optimal R&D investment, market value, and overall firm growth.

Outlining a larger economic vision, Knott described a scenario in which investors would use the RQ method to value stocks, which would shift their investment toward high-RQ companies. That would spur companies to try to boost their RQ, generating higher returns. Ultimately, said Knott, we would see increases in business’s RQ — and a corresponding boost in America’s GDP.

Sharing Perspectives from Industry

Panelists from the pharmaceutical industry and transportation sector discussed their efforts to innovate, and measure innovation, against the backdrop of digital and other massive changes in their sectors.

Yvonne Phan, a global senior manager at AstraZeneca, described the firm’s recently-launched digital platform designed to reach health care practitioners and insurance companies. The platform sits on the medical affairs side of the business, however, which Phan said subjected it to certain FDA regulations. “There’s a very small sandbox we’re allowed to play in, and we’ll get fined it we go beyond it.”

On the consumer side of AstraZeneca’s business is James Smith, a director of marketing. His team, too, is looking toward digital. He explained that a traditional TV campaign can reach millions of people, but they may not be potential users of a particular medication. Digital changes the game. “Pharma’s starting to get into that space where we can use information about your digital footprint… about your medical history, and we can start to target patients more effectively.”

Andre Costa, the chief information officer of CCR S.A. in Brazil, a major infrastructure services company in Latin America handling the operation of motorways, subways, and airports, discussed his use of open innovation: collaborating with a hub of players to solve business challenges. He noted that one recent success involved redistributing the number of passengers in subway cars in order to improve passenger experience and reduce electricity costs. This was a major win, he explained, because electricity constitutes 70% of the subways’ operating expenses.

Singh noted that all three panelists were tackling new business models, such as the introduction of digital platforms and open innovation. It’s difficult to measure innovation in these cases, he said, because the process is still evolving. He asked the panelists about their baselines and how they determined what to measure.

Costa noted that it’s difficult to set up a baseline in the infrastructure services industry because airports and roads, for instance, are dramatically different projects. In addition, “in infrastructure, you build something to last 100 years,” which makes some measurement timeframes problematic. He said CCR builds business cases using a metric known as total cost of ownership (TCO), and also assesses how innovation is improving customers’ everyday lives.

Phan noted that on the medical affairs side she’s prohibited from tying dollar values to innovation activities. “So it’s tricky for us to find very directed metrics to ensure we’re putting money into where it’s worth it.” For the new website, she does track site visits and return visits, and resources downloaded or emailed to others. Hence she is able to calculate something similar to CLV, or customer lifetime value. She has also purchased data from the site metrics of competitors in an effort to set baselines.

Smith commented that for his commercial digital solution, his team brought in patients and physicians to help co-create the site. He reflected, “Having customer input, and on an ongoing basis to refine it, has made [the site] better and helped us understand where we should invest the money.”

Singh acknowledged the complexity of the innovation measurement problem, wrapping up by saying, “This is a moving target; we’re not going to have closure.” Yet we can continue to apply analytics, he said, to come up with ever-more effective measurement models down the road.

Getting Your People to Think Innovatively

Introducing the Designing Incentives panel discussion, Chaudhuri said that besides measuring innovation in terms of “what we have achieved,” organizations must “promote the right behavior in order to further innovation, both radical and incremental.”

Venkateswara Duggirala, a digital innovation director for Tata Consultancy Services, talked about how businesses need to make sure employees’ incentives to innovate are aligned with the firm’s overall metrics. For example, someone’s KPIs might focus on generating thought leadership and ideation, yet the metric the company really values is revenue from digital sales. A situation like that, he said, causes tension: it’s a zero-sum game. Ryan Sjostrom, a senior director of marketing at Novo Nordisk, agreed that for example, “if my incentives are based solely on stock price or corporate value but I’m looking to radically innovate, this conflict can stress leadership tolerance, acceptance, or initiation of significant change.”

Duggirala added that incentives to innovate need to be diffused through every level of the organization, so that “fresh eyes and fresh minds” — not to mention subject matter expertise — can be applied to achieve innovation.

A managing director at JPMorgan Chase, Kunal Vaed spoke about metrics which he said help attract and retain talent and promote an innovative mindset. One is JPMorgan Chase’s App Store rating, which Vaed stated is among the highest of all banks. Another is the firm’s active mobile users, where JP Morgan Chase leads the industry with 37.3 million as of Q4’19. These two indicators suggest that the digital side of the company is “satisfying and delighting customers.”

Another metric Vaed focuses on is the extent to which senior management spotlights technology in public statements. Such statements attract talent by expressing what the company values, he said. He noted that JPMorgan Chase’s CEO Jamie Dimon’s 2008 shareholder letter only mentioned technology once, but the most recent letter mentioned it 21 times. “So as a bank, we have recognized, and the leadership has bought into the idea, that technology is the beginning of innovation.”

Financial compensation has long been an incentive for employees, of course, both to stay with the firm and to engage in the innovative behaviors the firm seeks. But the panel agreed that younger workers in particular are incentivized by more than money. Employees care about company culture, workplace environment, and the firm’s social impact. Duggirala and Sjostrom named employee happiness and talent retention as metrics that should be included in any incentive system.

“Clearly, incentives matter,” Chaudhuri summed up. “We know that they drive behavior. They also matter because of the misalignment problem we’ve been talking about: They can actually steer you away from innovation and other growth goals… So you want to make sure to get that right.”

Workshop: Developing Metrics for Innovation

In the workshop, participants were tasked with assessing the state of measuring innovation in their firms or divisions.

Chaudhuri offered a framework for thinking about innovation metrics, dividing 23 quantifiable metrics into three categories. First were inputs, such as R&D spending, the idea pipeline, and human resources devoted to innovation. Next were process effectiveness measures, such as patents commercialized, and average time to market for new products. The third category was performance outcomes, or outputs, such as ROI on investment in innovation, percent of sales from new products in a given number of years, and customer satisfaction.