Walker Management Library
Walker Management *Library
Contact:
Walker Management Library
Owen Graduate School of Management
401 21st. Avenue South
Nashville, TN 37203
Website
Owen Graduate School of Management
401 21st. Avenue South
Nashville, TN 37203
Assessing Generative AI
Since we are at the forefront of Gen AI and continual innovation in this space, there is much to learn about this technology and what it offers and what it doesn't. It is helpful to seek information, evaluate what you learn, and experiment by testing out Gen AI. This will help you explore what Ethan Mollick calls the "Jagged Frontier," or the testing of AI's outer limits.
This page of the guide will explore some of the value and limitations as we currently know them. To help you start evaluating Gen AI, consider some of the questions below.
Value
- What can Gen AI do that is useful or helpful? What does it add?
- How is it better than other ways of doing something?
- How can it be used to improve life and work?
- What does the existing evidence show?
- Who will benefit?
Limitations
- What is Gen AI's limitations? What can’t it do?
- What does it take away? What does it cost?
- How does it negatively affect life and work?
- What does the existing evidence show?
- Who or what will be harmed?
Exploring Gen AI's Value/Limitations
Below are an assortment of articles exploring value and limitations. A good place to start is with Ethan Mollick's "15 Times to use AI, and 5 not to."
-
15 Times to use AI, and 5 Not toA useful discussion of when AI can help and when it can hinder from thought-leader, Ethan Mollick's blog.
-
Recalculating the Costs and Benefits of Gen AI Gen AI has the potential to create a tremendous amount of value for organizations and individual employees. By focusing on the value of the output that AI produces, however, we tend not to think much about other sources of value. We rarely ask: In what way does each activity we do create true and unique value? It’s time for leaders to take a cold, hard look at their organizations and ask that question so they don’t destroy more value than they create in adopting gen AI. An AI value audit can help leaders make an objective and holistic assessment of benefits and costs
-
The Crowdless Future? Generative AI and Creative Problem-Solving The rapid advances in generative artificial intelligence (AI) open up attractive opportunities for creative problem-solving through human-guided AI partnerships. To explore this potential, we initiated a crowdsourcing challenge focused on sustainable, circular economy business ideas generated by the human crowd (HC) and collaborative human-AI efforts using two alternative forms of solution search. The challenge attracted 125 global solvers from various industries, and we used strategic prompt engineering to generate the human-AI solutions. We recruited 300 external human evaluators to judge a randomized selection of 13 out of 234 solutions, totaling 3,900 evaluator-solution pairs. Our results indicate that while human crowd solutions exhibited higher novelty—both on average and for highly novel outcomes—human-AI solutions demonstrated superior strategic viability, financial and environmental value, and overall quality. Notably, human-AI solutions cocreated through differentiated search, where human-guided prompts instructed the large language model to sequentially generate outputs distinct from previous iterations, outperformed solutions generated through independent search. By incorporating “AI in the loop” into human-centered creative problem-solving, our study demonstrates a scalable, cost-effective approach to augment the early innovation phases and lays the groundwork for investigating how integrating human-AI solution search processes can drive more impactful innovations. Funding: This work was supported by Harvard Business School (Division of Research and Faculty Development) and the Laboratory for Innovation Science at Harvard (LISH) at the Digital Data and Design (D 3 ) Institute at Harvard. Supplemental Material: The online appendix is available at https://doi.org/10.1287/orsc.2023.18430 .
-
Machine learning and generative AI: What are they good for in 2025? While generative AI is widely accessible and useful, businesses need to know when to use other AI tools, like traditional machine learning.
-
How Can Teams Benefit From AI Team Members? Exploring the Effect of Generative AI on Decision-Making Processes and Decision Quality in Team-AI Collaboration Human teams with distributed knowledge can make high-quality decisions but often fail due to decision-making asymmetries. As AI team members become integrated collaborators, understanding how AI can reduce these decision-making asymmetries is essential. However, little is known about how AI team members can reduce these asymmetries and whether new AI-specific asymmetries emerge from team-AI collaboration. Building on the information asymmetries model, we conducted an exploratory experiment with 215 individuals across 81 teams performing a hidden profile task under three knowledge configurations: (1) human teams with asymmetric knowledge, (2) teams collaborating with AI with centralized knowledge, and (3) teams collaborating with AI with asymmetric knowledge. Our results show that teams with centralized AI knowledge make more accurate decisions than human teams due to reduced decision-making asymmetries, trust in AI, beneficial AI information processing, and a balanced AI collaboration focus. In contrast, teams with asymmetric AI knowledge show only moderate reductions in decision-making asymmetries. Moreover, due to emerging AI-specific asymmetries-such as mistrust, nonbeneficial AI information processing, and a critical AI collaboration focus-these teams fail to outperform human teams. We integrated our findings into process models that illustrate how successful team-AI collaboration depends on effective teamwork between human and AI members.
-
A Reality check of the benefits of LLM in business This paper thoroughly examines the usefulness and readiness of LLMs for business processes. The limitations and capacities of LLMs are evaluated through experiments conducted on four accessible LLMs using real-world data. The findings have significant implications for organizations seeking to leverage generative AI and provide valuable insights into future research directions.
-
Don't Let Gen AI Limit Your Team's Creativity No one doubts ChatGPT's ability to generate lots of ideas. But are those ideas any good? A recent real-world experiment showed that teams engaged in a creative problem-solving task saw only modest gains from AI assistance for the most part-and some underperformed. Surveys conducted before and after the exercise showed that the teams using AI gained far more confidence in their problem-solving abilities than the others did, but that much of their confidence was misplaced. But don't blame the technology, says Kian Gohar, CEO of the leadership-development firm GeoLab and one of the study's authors. "Brainstorming with generative AI requires rethinking your ideation workflow and learning new skills," Gohar says. This article offers guidance for approaching the exercise as a structured, ongoing conversation, opening up a staggering capacity to develop better and more-creative ideas faster.
-
How People Can Create—and Destroy—Value with Generative AI A first-of-its-kind scientific experiment finds that people mistrust generative AI in areas where it can contribute tremendous value and trust it too much where the technology isn’t competent.
-
Competencies Needed by Business Professionals in the AI Age: Character and Communication Lead the Way A study by Cardon et al. (2024) exploring the uptake of Gen AI by business practitioners and views about Gen AI's impact on business skills.
Information Limitations of Generative AI
Limitations
Generative AI tools do have several limitations. It is important to be aware of these limitations so that you can ensure you get the most out of your interaction with the Gen AI.
Training Data Limitations
Training data for Gen AI LLMs have the following limitations:
- Knowledge cutoff date – the date developers use to indicate the recency of training data. This may be mitigated to an extent by retrieval augmented generation (RAG), which can search supplied documents or the open web.
- Cultural bias – negative stereotypes or attitudes on the basis of gender, race, ability, sexual orientation, etc.
- Information quality – the extent to which information is credible
Read more about these limitations below:
-
Dated Data: Tracing Knowledge Cutoffs in Large Language ModelsIn this work, we define the notion of an effective cutoff. This is distinct from the LLM designer reported cutoff and applies separately to sub-resources and topics. We propose a simple approach to estimate effective cutoffs on the resource-level temporal alignment of an LLM by probing across versions of the data. Using this analysis, we find that effective cutoffs often differ from reported cutoffs.
-
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?This paper examines the risks of large training data, which includes biases inherent in using all of the data available on the internet.
Hallucinations
Another limitation with LLMs is that they are known to hallucinate or make up factually incorrect information. Check out the hallucination rates for various LLMs:
Also, check out the hallucination example below showing how ChatGPT made up a resource. You can also view the full transcript of the chat interaction.
Navigating Limitations
Despite the limitations of LLMs, you can mitigate the risks involved. The key ingredients to mitigating risks are
- awareness
- strategic use of Gen AI tools
Below are some resources to help. Also, be sure to check out this guide's sections on Prompt Patterns and Gen AI Upskilling, which will help you up your prompting game.
-
Beware of BotshitThis article discusses the problem of "botshit" (inaccurate content generated by AI chatbots and uncritically used by humans) and how to avoid it through the use of a guiding framework.
-
Nudge Users to Catch Generative AI ErrorsThis article discusses an experimental tool to highlight possible errors from LLMs so that businesses can help employees be more critical users of AI tools.
