April 1, 2019 · By Larry Lapide · 

Face Validity

Upon reviewing a model with my first manager at Arthur D. Little, I learned an important lesson. He taught me that while it is important to demonstrate that a model has scientific validity, it is equally important to have “face validity.”

Translation: To be useful, a model needs to be statistically sound in representing the real world. But, it must also make sense to managers by incorporating factors that make sense to them.

I am astounded to find that many of the Internet-based Big Data models today only have face validity; apparently, developers don’t care about scientific validity. They believe that if a model’s factors seem to relate to dependent factors, those are sufficient.

For business models, face validity is necessary, but not sufficient. Indeed, face validity alone might be potentially dangerous. I gained this insight by reading “Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy,” a book by Dr. Cathy O’Neil.

What is a weapon of math destruction?

Dr. O’Neil, a former quantitative analyst at a financial hedge fund, experienced first-hand the damage financial models did leading up to and following the mortgage market meltdown of the past decade.

They helped the collapse of financial institutions. She became disillusioned with mathematical models that affect society. Or, as one description of the book puts it: “A former Wall Street quant sounds an alarm on the mathematical models that pervade modern life – and threaten to rip apart our social fabric.”

Her premise is that the vast amount of Big Data is used in ways that are: “1) opaque; 2) unquestioned; and 3) unaccountable.” In simpler terms: 1) the detailed data is not transparent to a person affected by its decision-making; 2) the use of the data is beyond reproach in the mind of the modeler; and 3) modelers refuse to defend the model other than to say “it is what it says.” It’s akin to using the court of public opinion rather than the law to assess whether someone has committed a crime.

In addition, she states that models often result in behavior that has “vicious, self-reinforcing feedback loops,” whereby things get worse for those affected—especially minorities and the poor. While there are many examples in the book, I’d like to focus on two.

Illustrative WMDs

U.S. News & World Report (USNWR) was founded in 1933. Fifty years later, in 1983, it was a second-rate publication, lagging Time and Newsweek, which were then the industry leaders. To set itself apart, the magazine decided to start a service ranking colleges and universities with the intent to help young people make their first big decision.

This was a game-changer for USNWR because today it markets itself on its website as “a multi-platform publisher of news and information, which includes usnews.com and annual print and e-book versions of its authoritative rankings of Best Colleges, Best Graduate Schools and Best Hospitals.”

The initial rankings involved weighted factors that journalists (not educators) felt were reasonable and could be quantified. Essentially, all were face-valid factors, and not necessarily ones proven to be related to educational quality. Dr. O’Neill contends the rankings were too successful, and its model over time was a WMD to education.

The rankings started a “race-to-the-top” among universities, whereby they did everything to raise rankings. Because costs were not included among the ranking factors, this incentivized colleges to hire expensive faculty, beef up athletic programs, construct new luxurious dorms and enhance dining menus.

Some colleges even resorted to cheating their way to the top, fabricating the numbers they reported to USNWR. Parents and students spent a lot of money on college admission planners to get into the top-ranked colleges. In addition, some international students cheated on standardized exams. The author contends that this competition contributed to today’s exorbitant tuition costs and student loans, leaving too many minority and poor students saddled with debt they will never pay back.

Another illustration has to do with credit scoring activities of marketers and others using Big Data.

The author contends that the well-known FICO scores, used by credit card and other loan providers, are not WMDs. FICO is regulated and transparent to borrowers. A FICO score is based on the financial history of a borrower – not those similar to them. However, these scores, while valid for creditworthiness, are often used for hiring; they wrongly assume that a high score means a better worker. This can lead to wealthy applicants getting jobs over poor applicants (who arguably need them more).

Meanwhile, “e-scores” developed for marketing purposes include factors in addition to the FICO ones. The biggest offender is a borrower’s zip code because average loan-default rates vary significantly by zip code. An e-score is a WMD because it assumes that if my neighbors default on loans, I have a high chance of doing so. Thus, poorer loan applicants may not get loans, or if they do, they are subject to paying higher interest rates. A self-fulfilling prophecy, this increases the chance of these applicants defaulting because payments were set too high.

WMD models in business

The examples I just discussed can wreak havoc on societies, and in particular on minorities and the poor. But, do we have WMD models in the business world? I would say there aren’t many, because in business the focus is on attracting and retaining loyal customers and working with the best suppliers—certainly no harm intended. Some business models that appear to be WMDs, might at worst be models of moderate distortion. Below are three examples.

Certainly, Gartner’s Top 25 Supply Chains ranking looks like USNWR’s Best College rankings. Gartner uses six face-valid weighted factors in rating supply chains: peer opinions, Gartner opinions, return on assets (ROAs), inventory turns, revenue growth and a newly added corporate social responsibility score.

While Gartner’s intent is to recognize best supply chain practices with regard to its demand-driven value network model, the supply chain community initially believed it to be the 25 best/excellent supply chains.

Like the Best Colleges report, it started a race to the top. Managers sought access to Gartner analysts and focused on attaining high scores. As I’ve written in past columns*, it cannot identify excellent supply chains. It just includes big companies, is reliant on opinions based on limited knowledge of supply chain operations and gives too much credit to a supply chain organization for revenue growth to which supply chain managers are not held responsible.

However, this does not make the Top 25 a WMD. After all, it was created by supply chain experts with an intent to stimulate healthy dialogue about what is possible. It would only hurt a company if an organization gamed its way to the top. This, however, would be fully transparent to its company because the only valid judge of whether a supply chain is excellent is the company in which it resides – not a third party.

Starting in late 2004, I wrote a three-part series on sales and operations planning (S&OP) because practitioners were asking for advice. It provided managers with a four-stage S&OP process maturity model that I believed was needed in support of the resurgence in S&OP.** Similar to the maturity models developed by consultants, my S&OP model gauged how developed a company’s process was relative to an ideal.

Consultants often use these to recommend a journey to get to an ideal over time. The S&OP model assessed a company’s stage in terms of factors including the meetings conducted; how integrated and extended were the processes; and the extent to which enabling software technologies were integrated. Because the S&OP process involves collaboration and consensus building among the supply, demand and financial components of a company, I assumed getting to Stage 4 would offer the greatest benefit without any data to support this.

Was the model a WMD? I don’t think so because I believed, as did the industry, that internal and external collaboration would help companies achieve financial InSIGHTSobjectives. So, a strong S&OP process would cause no harm, as long as a company installed a truly collaborative process instead of an extremely contentious one that turned out to be detrimental to corporate culture.

Lastly, I wrote about unconstrained vs. constrained demand forecasts, arguing that supply chain planning should be based on true supply-neutral demand forecasts.*** These are forecasts that reflect demand devoid of distortions related to supply surpluses, shortages and other supply factors. Over time, forecasting demand that is not supply-neutral might condition customers to demand products based on availability, rather than on true needs.

I’ve seen a number of examples of demand distortions. For example, many companies forecast customer demand from shipment data. In some cases, shipments are not the same as true demand, such as when supply factors cause orders to be filled imperfectly (not delivered on-time or as split shipments). When this is the case, a shipment forecast is not the best representation of true demand.

Thus, while this type of forecasting model is not a WMD, it is a model of moderate distortion; especially when customers become conditioned to accept imperfect order fulfillment. However, a shipment forecast can become a WMD when customers get tired of poor delivery performance and buy from a competitor.

In summary, while there is not much evidence to support the fact that the business community develops WMDs, it might develop models of moderate distortion if they rely too heavily on face-valid factors, without checking for validity. However, if a model leads to decision-making that results in the loss of a customer, I’d say it is destructive.

More from:  MIT Center for Transportation and Logistics Visit the Supply Chain 24/7 Education Resource Center

Related Article: Making Sales & Operations Planning a Calendar Highlight

Making Sales & Operations Planning a Calendar Highlight

Article Topics Best Practices  Business  Education  University/College Big Data Data Optimization Sales & Operations Planning All topics

Comments Be the first to post a comment. 
You must be logged in to post a comment. Login

About the Author

Larry Lapide
Dr. Lapide is a lecturer at the University of Massachusetts’ Boston Campus and is an MIT Research Affiliate. He received the inaugural Lifetime Achievement in Business Forecasting & Planning Award from the Institute of Business Forecasting & Planning. Dr. Lapide can be reached at: llapide@mit.edu. Latest Business NewsDon’t Build Weapons of Math DestructionMarch 27, 2019 ·           The business community might not develop Weapons of Math Destruction, but it might develop models of moderate distortion if they rely too heavily on face-valid factors, without checking for validity.
Association For Supply Chain Management ASCM – Much More Than a Re-BrandingMarch 17, 2019 ·           APICS, the association for supply chain management, new name, Association For Supply Chain Management ASCM, reflects an association focused on the end-to-end supply chain.
Leadership and the Democratization of Supply ChainsFebruary 27, 2019 ·         The Fourth Industrial Revolution will continue the democratization of technology.
FedEx Surprises Wall Street with Resignation of President & COO David J. BronczekFebruary 15, 2019 ·         FedEx Corp. threw Wall Street into a loop by announcing the resignation of Fred Smith’s top deputy David J. Bronczek just weeks after he joined the company’s board, Raj Subramaniam will take over as…
 More Business

24|7 Pro Team Picks

McDonald’s Acquires AI-Powered Omnichannel Personalization Company Dynamic YieldUPS in the Process of Preparing to Enter United States Home Health Services MarketDHL Resilience360: Top 10 Supply Chain Risk Predictions 20195G & the Future of Connectivity: 7 Industries the New Wireless Technology Could TransformPreviousNext

NextGen Supply Chain Conference

Attend the premier educational conference that answers the question “What’s next in supply chain management?” NGSC is dedicated to preparing executives for the coming technologies that will have the most transformative effects on business. Learn more or register.This year’s
conference focuses onRoboticsArtificial
IntelligenceBig DataSave
the date!April 16-17at the landmark
Chicago Athletic Associationin Chicago

24|7 Company Profiles

As the largest nonprofit association for supply chain, we are an unbiased partner, connecting companies around the world to the newest insights on all aspects…

Managing the Supply Chain Complexity ParadigmThis paper offers insight into managing supply chain and product complexity by providing background into the complexity’s sources, a framework…PreviousNext

24|7 Resources

Research & DownloadsHow a Multiparty Platform Speeds Digital Transformation and Optimizes…If your business depends on complex relationships with hundreds or thousands of suppliers…

Special CoverageSpecial:
Autonomous Vehicles
Autonomous vehicles have great potential for improving existing, high-demand transportation services, moving everything from passengers to packages. While most attention is focused on cars and…

Company ProfilesCompany Profile:
DMLogic

DMLogic helps clients reach the highest levels of productivity and efficiency…

Photos & MediaSATO Launches Next-gen IoT Mobile PrinterNew rugged mobile combines high speed and expandability optimized for retail, logistics and more.