SEMS Says | How your expert thinks can determine methodology

The advent of inexpensive storage and accessible computing power has helped fuel the popularity of big data to support decision-making. Collecting, storing and mining data are now routine, and analysts at all levels of expertise can build data-driven models.

However, experience shows the perils of allowing purely data-driven models and artificial intelligence to totally drive the decision-making process. For example, Google's Flu Tracker missed the peak of flu season by 140 percent. Qualitative components of models cannot be ignored, and subject matter expertise continues to be needed to build models that support holistic decision-making. 

The literature and practice support two major methods or approaches to subjective decision-making: utility-based approaches, such as multiple objective decision-making (MODA), and the analytic hierarchy process (AHP). Both approaches require subject matter experts (SMEs). In a utility-based approach, SMEs identify a value function for every attribute, or characteristic to be evaluated, of the model. These value functions can be risk averse (concave), risk seeking (convex), risk neutral or S-shaped. The SME then identifies 0-100 values for every point on the function. The alternatives are then scored against the attributes.   

On the other hand, the AHP has the SMEs make pairwise comparisons on a scale of one to nine between the attributes (with respect to the goal) and alternatives (with respect to the attributes) The scale corresponds to qualitative assessments, such as equal, moderate or strong importance between the two attributes or alternatives being compared. Critics have argued the concern for potential rank reversal in the AHP and that the amount of pairwise comparisons needed in large models is overwhelming. A lesser argued concern of utility-based models is the danger of using a subject matter expert who doesn't have a decision analysis background or does not fully understand risk attitudes. In those scenarios, the 0-100 values may not be truly accurate or correctly reflect the attitude toward risk, sacrificing the integrity of the data being collected. 

Both approaches have pros and cons. I have found appropriate uses for both methods in my work; I typically meet with the SME before the elicitation session to get an understanding of his or her decision analysis background. Typically, the SMEs have limited backgrounds in this area, and that is OK. Honestly, if they were decision experts, they wouldn't need a facilitator to build a model. However, the key, and this is not always made explicit in the literature, is to truly understand how a decision maker thinks about and approaches a problem. If he or she can think in terms of risk and value, then a utility-based approach may be best. If he or she thinks more qualitatively, the AHP may be best. The goal is to elicit data that is reliable and can be used in the model. All models have limitations, so the challenges of each method are inherent to the process. 

Data elicited from SMEs can quantify qualitative aspects of the decision process and support a human element to the model. Big data and artificial intelligence are clear paths to the future, but subjective components are often needed to ensure the models and algorithms work within the organization's culture and make reliable predictions. Utility approaches and the AHP can assist with this, and it is up to us, as engineering managers, to select the method that supports our SMEs and accurately incorporates their data into the model. 

Natalie Scala is an assistant professor in the Department of e-Business and Technology Management in the College of Business and Economics at Towson University.