Class themes were Nonprofit role in public governance, domestic and international perspectives in MGNPO and Big Data, Leveraging the Power of Analytics in IDSC

Table of contents

MGNPO

• The Privatization of Political Representation, Levine (2016)

IDSC

• Diamonds in the Data Mine, Loveman (2003)
• A Process of Continuous Innovation: Centralizing Analytics at Caesars (2013)
• A Step by Step Guide to Smart Business Experiments. Anderson and Simester (2011)
• The Business of Artificial Intelligence (2017)
• How to Design Smart Business Experiments, Davenport (2009)
• How to Build an Insights Engine, Van den Driest and Sthanunathan (2016)
• Selling into Micromarkets, Goyal, Hancock, and Hatami (2012)

MGNPO

The Privatization of Political Representation, Levine (2016)

• In an era of public-private partnerships, what role do nonprofit community-based organizations (CBOs) play in urban governance?
• new way to understand CBOs’ political role in poor neighborhoods: CBOs as nonelected neighborhood representatives.
• By reconsidering CBOs’ political role in urban neighborhoods, this study uncovers a consequential realignment of urban political representation, and identifies an important tradeoff between the urban poor’s access to resources and the ability to hold their leaders democratically accountable
• important tradeoff between resource allocation and democratic accountability: neighborhoods represented by CBOs gain greater access to resources, but they may sacrifice the ability to hold their leaders democratically accountable.
• Existing theories focus narrowly on public funding, but CBOs also receive grants from private funders—an arena where elected officials have little oversight or influence. Theoretical models of CBOs and urban governance are incomplete if they do not account for the increased presence of private funding in cities.
• macro economic conditions in the 1970s initiated a shift from partisanship to “partnership” in urban politics. Economic decline dismantled institutions with ties to partisan politics like labor unions and locally-owned firms
• structural arrangements present a distinct context for scholars to understand CBOs’ role in urban governance
• Economic shifts carry additional political consequences as new actors (funding organizations and their nonprofit grantees) cement their place as “partners” in urban governance, while traditional actors (like elected politicians) can fall out of favor
• conditions structurally position CBOs to assume new political roles in urban neighborhood
• Reflecting structural changes to community development funding, CBO leaders viewed government executives and private funders as the key holders of resources, not state legislators or city councilors
• Accordingly, their political strategy focused not on persuading district-based elected officials, but on engaging directly with appointed government officials and foundation program officers.
• These were not one-off, superficial hearings, but ongoing relationships cemented through meetings, phone calls, e-mails, neighborhood tours, and ribbon cuttings
• Instead of direct involvement in electoral politics, the CBOs indirectly channeled their legislative interests through regional associations and intermediary organizations
• To be sure, CBO leaders expressed reverence for their elected representatives. They frequently listed them alongside other “allies” or “supporters” of their work, and they invited them to speak during ribbon cuttings as signals of deference. Even though elected officials played little role in planning or proposing projects, support was nevertheless useful because their last-minute obstruction could impede implementation.
• structural shifts— government’s continued reliance on CBOs amid declines in public funding, the growth of private funders, and the move toward partnership—rearrange the puzzle pieces of governance, creating the space for CBOs to wedge out district-based politicians as the assumed representatives of disadvantaged neighborhoods
• When we reorient our thinking this way, it complicates how we understand the political representation of the urban poor
• The sociology of urban neighborhoods and local political life would grow more complex and comprehensive if researchers devoted more attention to the role of private organizations in public governance
• tradeoff between the urban poor’s access to resources and the ability to hold their leaders democratically accountable
• CBOs have become necessary to bring resources to poor neighborhoods, but leaders are not elected and can remain in their positions for decades
• When CBOs operate as neighborhood representatives, poor neighborhoods gain access to resources but sacrifice the ability to elect, appoint, or impeach their representative
• Limited research also suggests that CBOs, operating outside electoral accountability, can facilitate gentrification and contribute to affordable housing crises when they adopt market-based logics
• Be careful not to fetishize democratic accountability. Stable CBO leadership may be more desirable than the uncertainty of administrative and electoral turnover,
• CBOs may be critical organizations willing to maintain affordable housing and advocate for the urban poor in gentrifying neighborhoods
• we should be careful not to assume uniform resident interests. Popular opinion in disadvantaged neighborhoods is diverse and often contradictory.
• CBOs, as nonelected neighborhood representatives, are not subject to the same system of accountability as electeds. Remains an open empirical question if the lack of democratic accountability helps or harms poor neighborhoods

IDSC

Diamonds in the Data Mine, Loveman (2003)

Idea in brief

• Harrah’s secret? Employees dazzle customers with exceptional service. Valets greet them by name, hosts ensure they’re happy, and the company rewards them handsomely for choosing Harrah’s.
• it uses sophisticated, proprietary technology to deeply mine its customer database
• by slicing information into ever-finer segments, Harrah’s gets to know its customers better
• continually enhances the benefits of choosing its casinos over flashier rivals.

The idea in practice

1. Acquire a rich repository of customer information
2. Slice and dice data finely to develop marketing strategies
3. Identify core customers by predicting their lifetime value
4. Gather increasingly specific information about customers’ preferences—then appeal to those interests
5. Reward employees for prioritizing customer service

Article

• Harrah’s Entertainment has the most devoted clientele in the casino industry—a business notorious for fickle customer
• They increased customer loyalty in two ways
  • database marketing and decision-science-based analytical tools
  • deliver the great service that consumers demand
• In short, by mining customer data deeply, running marketing experiments, and using the results to develop and implement finely tuned marketing and service-delivery strategies
• opted to invest in development of the intellectual and technological capabilities needed to assemble and analyze data about those customers.
• wanted to change Harrah’s from an operations-driven company that viewed each casino as a standalone business into a marketing-driven company that built customer loyalty
• Common practice calls for defining marketing strategies apart from database strategies—that is, the company comes up with a grand marketing scheme and then tries to adjust the database to its strategies. They decided instead to let the data suggest the specific marketing ideas to use
• Understanding the lifetime value of our customers would be critical to our marketing strategy.
• The best way to engage in this kind of data-driven marketing is to gather more and more specific information about customer preferences, run experiments and analyses on the new data, and determine ways of appealing to players’ interests.
• decided to act on a radical idea: reward customers for spending in ways that added to their value
• split customers into three tiers: Gold, Platinum, and Diamond cardholders, based on their annual theoretical value. Platinum and Diamond cardholders receive greater levels of service
• essential for customers to see the perks that others were getting.
• Every experience in the casino was redesigned to drive customers to want to earn a higher-level card.
• turns out, marketing that appeals to customer aspiration works wonderfully
• Set up a series of triggers in the database and analyzed customer responses to those triggers
• Once entered into the database, those responses provided fodder for more slicing, dicing, and experimentation
• database strategy hinged on our ability to combine data from all of our properties, so customers could use their reward cards in multiple locations
• combining transactional data from all out sites was so important they developed and ultimately patented the technology to do it
• Deep data mining and decision-science marketing would be worth little in driving same-store sales growth were it not for another simultaneously applied and extremely critical ingredient—an absolute focus on customer satisfaction.
• data indicated that customers want friendly and helpful attention in addition to fast service. decided to link employee rewards to customer satisfaction.
• The better the experience the guest had, the more money employees stood to make
• implemented a bonus plan to reward hourly workers with extra cash for achieving improved customer satisfaction scores which they culled from very detailed customer surveys
• important to note that we chose to measure customer satisfaction scores independent of a property’s financial performance.
• This score-driven customer satisfaction measure allowed properties—even those in troubled markets—to continue to grow.
• meeting budget at the expense of service is a very bad idea. if you’re not making your numbers, you don’t cut back on staff
• just the reverse: the better the experience the guest has and the more attentive you are to them, the more money you’ll make
• good service is not a matter of an isolated incident or two but of daily routine
• We maintain our competitive advantage by using our human capital and technology systems to get to know customers better

A Process of Continuous Innovation: Centralizing Analytics at Caesars (2013)

• This unit is the consolidated and centralized analytics department for the entire enterprise. We provide analytical support for every aspect of the operations in every jurisdiction.
• CEO is a classically trained economist. His trade really is analytics and numbers. He has, I think, instilled a pretty unique culture across the enterprise in terms of how analytics gets woven into the fabric of virtually everything that we do.
• operators that work at Caesars Entertainment are tremendously astute at leveraging and consuming analytics.
• the communication necessary to do this well cannot be understated
• tremendous amount of time identifying stakeholders and doing our best to give them full transparency into how we were progressing, how things would be affecting their operations, and also, incorporating the feedback that they were providing to us continuously.
• exhausting exercise, but an absolutely essential one.
• really critical for us to have a very visible and meaningful wins throughout the process to be able to proceed in an unencumbered manner.
• giving all of the operations somewhat of a uniform view of their business was not a small thing, and getting to a common language around how we were going to be measuring the business from one location to the next was also pretty meaningful
• having that exercise in getting everyone on board, and giving them a uniform platform against that was something that added immediate value
• That’s a pretty low bar, at least in concept - but in actuality, that does take a good amount of time
• making sure that we preserve nuance when it made sense and then eliminating nuance that was not as meaningful and resulted in inefficiencies, does take some time. Part of that is process-oriented. Part of that is technical expertise, but a lot of that is the governance
• that exercise never goes away. The business evolves, and [it] requires that you’re continuously examining this;
• Within every area of the org - all have innovation outlines attached to their strategic plan for the coming year.
• One of the other benefits the centralized structure provides us is [that] it’s a great platform for partnership because it provides meaningful scale and gives an overview of a number of great live experiments across the enterprise, as well as various operating concerns that we’re providing support against.
• good deal of our innovation happens through partnerships outside of our organization or across industries
• We have experimentation going on every day, across every aspect of the operation. There are tests literally occurring all the time
• the explosion in data and the use of recursive algorithms offers us a great deal of flexibility
• Changes in data structure, architecture, and cost are also contributing to a wide range of exploratory analytics.
• there are a number of organizations that we work with to help us accelerate our capabilities in this space. Our centralized platform enables us to leverage these types of resources and absorb their capabilities in ways that add tremendous value
• also provides a meaningful platform for immediate consumption of the analytics and streamlined integration into the operations
• There are a number of talented providers of analytical thought leadership across industry, but we’ve found that effective partnership is greatly facilitated by having a strong internal capacity and infrastructure. Ensures that value is sustained over time
• Our charge is to provide the analytics’ view on any given question. It is not our charge to necessarily have decision rights against every decision
• What we are also charged with doing is being transparent around degrees of uncertainty associated with every analysis that we do
• Ambiguity is something that can be difficult for any data-driven organization to deal with, but it is a point that we try to address directly
• nuance is often the distinctive trait of our best analyses

A Step by Step Guide to Smart Business Experiments. Anderson and Simester (2011)

• Over the past decade, managers have awakened to the power of analytics
• avalanche of data presents companies with big opportunities to increase profits-if they can find a way to use it effectively.
• Even companies that make big investments in analytics often find the results difficult to interpret, subject to limitations, or difficult to use to immediately improve the bottom line
• Most companies will get more value from simple business experiment
• Managers need to become adept at using basic research techniques
• embrace the “test and learn” approach
• Feedback from even a handful of experiments can yield immediate and dramatic improvements
• The ease with which companies can experiment depends on how easily they can observe outcomes
• Without an effective feedback mechanism, the basis for decision making reverts to intuition
• In general, it’s easier to experiment with pricing and product decisions than with channel management or advertising decisions
• Think like a scientist
  • Running a business experiment requires two things: a control group and a feedback mechanism
  • Ideally, control groups are selected through randomization
  • The key to success with treatment and control groups is to ensure separation between them so that the actions taken with one group do not spill over to the other.
  • can be difficult to achieve in an online setting
  • can also be hard to achieve in traditional settings, where varying treatments across stores may lead to spillovers for customers who visit multiple stores
  • If you cannot achieve geographic separation, one solution may be to vary your actions over time.
  • consider repeating the different actions in multiple short time period
  • The second requirement is a feedback mechanism that allows you to observe how customers respond
  • two types of feedback metrics:
    • behavioral
      • Behavioral metrics measure actions-ideally, actual purchases.
    • perceptual
      • Perceptual measures indicate how customers think they will respond to your actions
  • Given that the goal of most firms is to influence customers’ behavior rather than just their perceptions, experiments that measure behavior provide a more direct link to profit, particularly when they measure purchasing behavior
• Overcoming reluctance to experiment
  • Organizational recalcitrance is one of the key hurdles companies encounter when trying to create a culture of experimentation
  • main obstacle to establishing the new usual is the old usual
  • Experiments are designed to improve decision making, and so responsibility for them must occur where those decisions are made - in the business units themselves.
  • also important to set the right expectations. It’s a mistake to expect every experiment to discover a more profitable approach
  • Productive experimentation requires an infrastructure to support dozens of small-scale experiments
  • Your goal, at least initially, is to find the golden ticket-you’re not looking for lots of small wins.
• the best experimentation programs start with the low-hanging fruit - experiments that are easy to implement and yield quick, clear insights
• You can identify opportunities for quick-hit experiments at your company using these seven rules for running experiments
  1. Focus on individuals and think short term
  2. Keep it simple
  3. Start with a proof-of-concept test
  4. When the results come in, slice the data
     • look for subgroups within your control and treatment group
  5. Try out-of-the-box thinking
  6. Measure everything that matters
  7. Look for natural experiments
• Avoid obstacles
  • Companies that want to tap into the power of experimentation need to be aware of the obstacles both external and internal ones
    • External
      • legal obstacles: Firms must be careful when charging different prices to distributors and retailers, particularly firms competing with one another
      • threat of adverse customer reaction to charging customers different prices can be a sufficient deterrent - widespread availability of pricing information on the internet
    • Internal
      • shifting to an experimentation culture requires a fundamental change in management outlook
      • Management-by-intuition is often rooted in an individual’s desire to make decisions quickly and a culture that frowns upon failure
      • experimentation requires a more measured decision-making style and a willingness to try many approaches, some of which will not succeed
  • the goal is not to conduct perfect experiments; rather, the goal is to learn and make better decisions than you are making right now
  • Organizations that cultivate a culture of experimentation are often led by senior managers who have a clear understanding of the opportunities and include experimentation as a strategic goal of the firm
• There is generally a practical limit on the number of experiments managers can run. Because of that, analytics can play an important role
• other companies’ business models may allow for only a few experiments
• conducting experiments in channel settings is difficult because changes involve confrontation and disruption of existing relationships.
• In these situations, analyzing historic data, including competitors’ actions and outcomes in related industries, can offer valuable initial insights
• companies that truly embrace this data-driven approach will be able to delegate authority to run small-scale experiments to even low levels of management. This will encourage the out-of-the-box innovations that lead to real transformation

The Business of Artificial Intelligence (2017)

• 250 years the fundamental drivers of economic growth have been technological innovations
• most important of these are what economists call general-purpose technologies
• The most important general-purpose technology of our era is artificial intelligence, particularly machine learning
• Within just the past few years machine learning has become far more effective and widely available
• Why is this such a big deal? Two reasons.
  • Prior to ML, this inability to articulate our own knowledge meant that we couldn’t automate many tasks. Now we can.
  • Excellent digital learners are being deployed across the economy, and their impact will be profound
• In the sphere of business, AI is poised have a transformational impact, on the scale of earlier general-purpose technologies
• The bottleneck now is in management, implementation, and business imagination
• Like so many other new technologies, however, AI has generated lots of unrealistic expectations.
• What can AI do today?
  • The biggest advances have been in two broad areas: perception and cognition
  • The speed of improvement has accelerated rapidly in recent years as a new approach, based on very large or “deep” neural nets, was adopted
  • second type of major improvement has been in cognition and problem solving
  • Machine learning systems are not only replacing older algorithms in many applications, but are now superior at many tasks that were once done best by humans
  • Once AI-based systems surpass human performance at a given task, they are much likelier to spread quickly.
  • impressive achievements, but the applicability of AI-based systems is still quite narrow.
  • ML systems are trained to do specific tasks, and typically their knowledge does not generalize
• Understanding machine learning
  • The most important thing to understand about ML is that it represents a fundamentally different approach to creating software: The machine learns from examples, rather than being explicitly programmed for a particular outcome.
  • In this second wave of the second machine age, machines built by humans are learning from examples and using structured feedback to solve on their own problems such as Polanyi’s classic one of recognizing a face
• Different flavors of machine learning
  • Artificial intelligence and machine learning come in many flavors, but most of the successes in recent years have been in one category: supervised learning systems, in which the machine is given lots of examples of the correct answer to a particular problem.
    • process almost always involves mapping from a set of inputs, X, to a set of outputs, Y.
  • Successful systems often use a training set of data with thousands or even millions of examples, each of which has been labeled with the correct answer. system can then be let loose to look at new examples.
  • algorithms that have driven much of this success depend on an approach called deep learning, which uses neural networks
    • Deep learning algorithms have a significant advantage over earlier generations of ML algorithms: They can make better use of much larger data sets
    • old systems would improve as the number of examples in the training data grew, but only up to a point, after which additional data didn’t lead to better predictions. deep neural nets don’t seem to level off in this way: More data leads to better and better predictions.
    • It’s comparatively straightforward to label a body of data and use it to train a supervised learner; that’s why supervised ML systems are more common than unsupervised ones, at least for now. Unsupervised learning systems seek to learn on their own.
    • exceedingly difficult to develop a successful machine learning system that works this way.
    • If and when we learn to build robust unsupervised learners, exciting possibilities will open up. These machines could look at complex problems in fresh ways to help us discover patterns
    • Such possibilities lead Yann LeCun, the head of AI research at Facebook and a professor at NYU, to compare supervised learning systems to the frosting on the cake and unsupervised learning to the cake itself.
  • Another small but growing area within the field is reinforcement learning
    • In reinforcement learning systems the programmer specifies the current state of the system and the goal, lists allowable actions, and describes the elements of the environment that constrain the outcomes for each of those actions.
    • Using the allowable actions, the system has to figure out how to get as close to the goal as possible.
    • systems work well when humans can specify the goal but not necessarily how to get there
    • a reinforcement learning system will optimize for the goal you explicitly reward, not necessarily the goal you really care about, so specifying the goal correctly and clearly is critical
• Putting machine learning to work
  • There are three pieces of good news for organizations looking to put ML to use today
    • AI skills are spreading quickly
    • necessary algorithms and hardware for modern AI can be bought or rented as needed
    • it is often surprisingly easy to obtain sufficient data to start making productive use of ML
  • Machine learning is driving changes at three levels: tasks and occupations, business processes, and business models
  • example of task-and-occupation redesign is the use of machine vision systems to identify potential cancer cells - freeing up radiologists
  • machine learning systems hardly ever replace the entire job, process, or business model. they compliment human activities, which can make their work ever more valuable.
  • if the successful completion of a process requires 10 steps, one or two of them may become automated while the rest become more valuable for humans to do
  • This approach is usually much more feasible than trying to design machines that can do everything humans can do
  • leads to better, more satisfying work for the people involved and ultimately to a better outcome for customers.
  • Designing and implementing new combinations of technologies, human skills, and capital assets to meet customers’ needs requires large-scale creativity and planning.
• Risks and limits
  • machine learning systems often have low “interpretability;” meaning that humans have difficulty figuring out how the systems reached their decisions
  • Deep neural networks may have hundreds of millions of connections, each of which contributes a small amount to the ultimate decision
  • predictions tend to resist simple, clear explanation
  • machines are not (yet!) good storytellers
  • Machines know more than they can tell us
  • creates three risks
    • hidden biases
      • machine have hidden biases, derived not from any intent of the designer but from the data provided to train the system
      • may inadvertently learn to perpetuate their racial, gender, ethnic, or other biases.
      • biases may not appear as an explicit rule but, rather, be embedded in subtle interactions among the thousands of factors considered
    • statistical truths
      • unlike traditional systems built on explicit logic rules, neural network systems deal with statistical truths rather than literal truths
      • can make it difficult, if not impossible, to prove with complete certainty that the system will work in all cases
      • Lack of verifiability can be a concern in mission-critical applications
    • difficulty to diagnose
      • when the ML system does make errors, as it almost inevitably will, diagnosing and correcting exactly what’s going wrong can be difficult.
      • underlying structure that led to the solution can be unimaginably complex,
      • solution may be far from optimal if the conditions under which the system was trained change.
• the appropriate benchmark is not perfection but the best available alternative
• The advantage of machine-based systems is that they can be improved over time and will give consistent answers when presented with the same data
• Perception and cognition cover a great deal of territory - from driving a car to forecasting sales to deciding whom to hire or promote
• chances are excellent that AI will soon reach superhuman levels of performance in most or all of these areas. So what won’t AI and ML be able to do?
  • Computers are devices for answering questions, not for posing them
  • entrepreneurs, innovators, scientists, creators, and other kinds of people who figure out what problem or opportunity to tackle next will continue to be successful
  • other humans, not machines, are best at tapping into social drives such as compassion, pride, solidarity, and shame in order to persuade, motivate, and inspire
  • the biggest and most important opportunities for human smarts in this new age of superpowerful ML lie at the intersection of two areas:
    • figuring out what problems to work on next
    • persuading a lot of people to tackle them and go along with the solutions
  • This is a decent definition of leadership, which is becoming much more important in the second machine age
• The status quo of dividing up work between minds and machines is falling apart very quickly.
• As was the case with steam power and electricity, success came not from access to the new technologies themselves, or even to the best technologists, but from innovators who are open-minded enough to see past the status quo and envision very different approaches, and savvy enough to put them into place
• artificial intelligence, especially machine learning, is the most important general-purpose technology of our era
• impact of these innovations will be reflected not only in their direct contributions but in their ability to enable and inspire complementary innovations
• a general principle is clear: The most nimble and adaptable companies and executives will thrive
• Organizations that can rapidly sense and respond to opportunities will seize the advantage in the AI-enabled landscape
• successful strategy is to be willing to experiment and learn quickly
• Over the next decade, AI won’t replace managers, but managers who use AI will replace those who don’t

How to Design Smart Business Experiments, Davenport (2009)

Idea in brief

• Too many business innovations are launched on a wing and a prayer
• With a small investment in training, readily available software, and the right encouragement, an organization can build a “test and learn” capability
• Equip managers to perform small-scale yet rigorous experiments
• Mistakes make it more likely that great ideas will see the light of day

Idea in practice

• the best way to support decision making on potential innovations is to…
  • Design an experiment
    • Put it to the test by measuring what happens in a test group versus a control group
    • be clear on what you need to measure to produce a decisive result
  • Act on the facts
    • Nothing but a success in a testing environment should be rolled out more broadly
    • neither should failures simply be scrapped.
    • Refine the hypothesis on the basis of the results, and consider testing a variation
    • capture what’s been learned, and make it available to others in the organization through a “learning library,”
  • Make testing the norm
    • enable nonexperts in statistics to oversee rigorous experiments
    • A core group of experts can lend resources and expertise and maintain the learning library
    • Leadership must cultivate a test-and-learn culture
• As your managers become more comfortable with testing, they’ll discover that it paves the way for, rather than throwing up barriers to, promising new ideas.

Article

• Thanks to new, broadly available soft ware and given some straightforward investments to build capabilities, managers can now base consequential decisions on scientifically valid experiments.
• Consumer-facing companies rich in transaction data are already routinely testing innovations well outside the realm of product R&D.
• As randomized testing becomes standard procedure in certain settings – website analysis, for instance – firms build the capabilities to apply it in other circumstances as well.
• while the “test and learn” approach might not always be appropriate (no management method is), it will doubtless gain ground over time
• an investment in software and training will yield quick returns of the low-hanging-fruit variety
• The real payoff, however, will happen when the organization as a whole shift s to a test-and-learn mind-set
• When testing makes sense
  • In theory, it makes sense for any part of the business in which variation can lead to diff erential results.
  • In practice, however, there are times when a test is impossible or unnecessary.
  • Generally speaking, the triumphs of testing occur in strategy execution, not strategy formulation
  • Scientific method is not well suited to assessing a major change in business models, a large merger or acquisition, or some other game-changing decision.
  • Beyond using the tactical-versus-strategic criterion, there are other ways to decide whether formal testing makes sense.
  • Sales and conversion-rate changes are frequently used as dependent variables in tests and are reliably measured
  • Other outcomes, such as customer satisfaction and employee engagement, may require more effort and invasiveness to measure.
  • Tests are most reliable where many roughly equivalent settings can be observed.
  • formal testing makes sense only if a logical hypothesis has been formulated
• The process of testing
  • you’ll need to acquaint managers at all levels with your organization’s process of testing.
  • Having a shared understanding of what constitutes a valid test enables the innovators to deliver on it and the senior executives to demand it.
  • process always begins with the creation of a testable hypothesis, then the details of the test are designed
  • After the test is carried out for the specified period, data are analyzed to determine results and appropriate actions.
  • Results are ideally put into some sort of “learning library”
  • Might lead to wider rollout of the experiment or further testing of a revised hypothesis
  • managers must understand how the testing process fits in with other business processes
  • testing feeds into various subprocesses.
• Building a testing capacity
  • Establishing a standard process is the first step toward building an organizational test-and-learn capability
  • need to create an infrastructure to make that happen
  • training programs to hone competencies, soft ware to structure and analyze the tests, a means of capturing learning, a process for deciding when to repeat tests, and a central organization to provide expert support
  • Managerial training
    • managers should learn what constitutes a randomized test and when to employ it.
    • benefit of hosting a program like this, rather than sending managers outside for training, is the greater emphasis on how the testing connects to upstream and downstream activities in the business.
  • Test-and-learn software
    • several off-the-shelf options exist – the most common ones being broad statistical packages and analytical tools like SAS
    • Some software tools are tailored to particular problems or industries
    • Likewise, highly specialized tools exist for online-usage testing, such as the web analytics software
  • Learning capture
    • If a firm does a substantial amount of testing, it will generate a substantial amount of learning
    • share that knowledge and use it to guide future initiatives
  • Regular revisiting
    • One tricky aspect of establishing a long-term testing approach is determining when to retest
    • Ironically, it is human intuition, not testing or analytics, that must be applied to determine the need for retesting
  • Core resource group
    • Most of the firms that do extensive testing have established a small, somewhat centralized organization to supervise it.
    • Without a central coordination point, testing methods may not be sufficiently rigorous, and test and control groups across multiple experiments may confound one another
    • it’s not always easy to influence or coordinate testing even when a central group exists
• Creating a testing mindset
  • organizations also need to establish a testing culture
  • Senior managers have to become accustomed to, and even passionate about, the idea that no major change in tactics should be adopted without being tested
  • Ask for evidence
    • When people claim that testing has confirmed the wisdom of their idea, have them walk you through the process they used
  • Give it teeth
    • for example, “not using a control group” as sufficient rationale for termination
    • create a culture in which managers insist on tests for every major initiative
  • Sponsor tests yourself
    • best management teams in this regard have institutionalized the process of doing and reviewing tests
• Testing may not be appropriate for every business initiative, but it works for most tactical endeavors
• needs to come out of the laboratory and into the boardroom
• key challenges are no longer technological or analytical; they have more to do with simply making managers familiar with the concepts and the process
• learning from testing, should become central to any organization’s decision making
• It’s time to replace “I’ll bet” with “I know.”

How to Build an Insights Engine, Van den Driest and Sthanunathan (2016)

• Operational skill used to confer long-term advantage. But today those capabilities are just table stakes.
• The new source of competitive advantage is customer centricity: deeply understanding your customers’ needs and fulfilling them better than anyone else.
• You need data to accomplish this. Yet having troves of data is of little value in and of itself.
• ability to transform data into insights about consumers’ motivations and to turn those insights into strategy.
• requires innovative organizational capabilities that, collectively, we call the “insights engine.”
• market research departments have been shifting from merely supplying data to interpreting it— distilling insights about consumers’ motivations and needs on the basis of their behavior.
• Driven by the imperative to become customer-centric, leading firms are now completing the transformation of market research groups into true in- sights engines with a fundamentally strategic role.
• 10 characteristics of superior insights engines, divided into two broad groups:
  • operational characteristics, such as functional independence and experimental orientation
    • Data synthesis
    • Independence
    • Integrated planning
    • Collaboration
    • Experimentation
    • Forward-looking orientation
    • Affinity for action
  • people characteristics, such as business acumen and well-balanced analytic and creative thinking styles
    • Whole-brain mindset
    • Business focus
    • Storytelling
• by itself, even the most advanced insights engine can’t make a firm customer-centric. That requires leadership from the top to ensure that every function
• maintains a singular focus on understanding and meeting consumers’ fundamental needs

Selling into Micromarkets, Goyal, Hancock, and Hatami (2012)

• Using its emerging analytics capability, a global firm took a more granular look at its business
• Diced seven U.S. regions into 70 “micromarkets” and zeroed in on those with the greatest potential
• created sales “plays” for the newly identified hot spots, and redeployed the sales force
• Within a year the sales rate doubled - without an increase in marketing or sales costs
• Key to the firm’s remarkable turnaround was its ability to combine, sift, and sort vast troves of data
• micromarket strategy is perhaps the most potent new application of big-data analytics in B2B sales
• most often understood as physical regions, they needn’t always be
• Discovering and exploiting new-growth hot spots involves three steps:
  • Defining your micromarkets and determining their growth potential
  • using these findings to distribute resources and guide the sales force
  • incorporating the big-data mind-set into operations and organizational culture
• Find new pockets of growth
• The first step in pursuing a micromarket strategy is to create an “opportunity map” of potentially lucrative hot spot
• taps internal and external data sets from a variety of sources and uses sophisticated analytics to build a picture of the future opportunity, not the historical reality
• Next, managers examine what drives customers’ purchasing in each market, determine the firm’s current market share in each, and look for causes of the variance
• identifies which markets represent the greatest growth opportunities
• The goal is to define the problem, the methods for solving it, and, crucially, how to translate the resulting insights into tools
• Make it easy for the sales team
  • management must have the courage and imagination to act on the insights revealed by the analysis.
  • sales team needs to understand the rationale behind the micromarket strategy and have simple tools that make it easy to implement
• Align sales coverage with opportunity
  • The first step is to overlay the rough allocation of resources across markets on the basis of their overall potential
• Create sales plays for each type of opportunity
  • the challenge for companies is how to help a generalist sales force effectively tailor messaging and materials to the opportunity
  • Companies should identify groups of micromarkets—or “peer groups”—that share certain characteristics.
  • a set of four to 10 peer groups is a manageable number
  • For each peer group, marketing managers develop the strategy and “play”—the best way to sell into that set of customers or market.
  • devise and perfect plays either by adapting approaches that have been successful in similar settings or by testing new plays in pilot markets.
• Support the sales force in executing the plays
  • For a micromarket strategy to succeed, the sales training has to be experiential.
  • engage with the opportunity maps that reveal hot (and cool) micromarkets in a given geography and test their intuition against hard data.
  • data analysis is often superior to anecdote in this realm
• An opportunity map is the foundation of a micromarket strategy. Each step in the process:
  1. Define micromarket size
  2. Determine growth potential
  3. Gauge market share
  4. Identify the causes of differences in market share
  5. Prioritize growth pockets
• Put data at the heart of sales
  • To sustain the early wins from a micromarket strategy, companies need to change their approach to sales force management in three ways:
    • rethink performance management
    • open new channels between sales and marketing
    • invest in talent development
  • Performance management
    • Few managerial moves will kill new initiatives faster than continuing to reward old behaviors
    • managers must shift from assessing reps’ performance relative to the entire sales force to assessing it relative to the opportunity.
    • Performance management in a data-rich sales environment can get closer than ever before to measuring true performance of a sales force
    • By sorting micromarkets or customer sets into peer groups according to the future sales opportunity they represent, companies can create better-informed sales plans and target
  • Cross-functional collaboration
    • marketing often takes on an expanded role, particularly in providing sales with data analytics and supporting the development and testing of sales play for a specific micromarket or customer peer group
    • management must establish clear, standardized processes at key bridge points
    • arms-length interaction doesn’t maximize the potential of true collaboration
  • Talent development
    • both marketing and sales teams will need to step up their capabilities, particularly with analytic talent
    • the most effective sales organizations will be those that put data analytics at the center of their strategies.
    • critical component in this talent equation is the bridge between analysis and action
    • analytic talent is important, but put equal emphasis on translating the analysts’ insights into guidance that the field can act on
    • frontline talent and capability building is essential and will produce the kind of inventive thinkers that are critical to creating successful micromarket strategies.
• Finding growth with big data is more than an add-on; it affects every aspect of a business, requiring a change in mind-set from leadership down to the front lines front lines
• “This granular view is really a new way of thinking…and it takes time for it to become part of the company’s DNA.”
• Micromarket strategies are demanding, but they consistently give sales a competitive edge