Have an LLM play Devil’s Advocate to improve your chance of success

Sarah Packowski
4 min readOct 2, 2023

A large language model (LLM) can be an effective devil’s advocate: listing arguments against a proposed plan as a way to explore the plan’s weaknesses and risks.

Samples: https://github.com/spackows/MURAL-API-Samples

An LLM plays the role of devil’s advocate, posting arguments against proposed plans for growing a business.

Large language models

Large language models (LLMs) are neural networks with billions of nodes. Using massive amounts of text (eg. pages scraped from the internet) as training data, these models build up a vocabulary of words and internalize the statistical relationships between those words.

See: Large language models (Wikipedia)

Devil’s advocate

From wikipedia, playing the role of devil’s advocate is:

A situation where someone takes a position they do not necessarily agree with (or simply an alternative position from the accepted norm) for the sake of debate or to explore the thought further

In the context of business, team members might feel they cannot argue against a proposed plan because they’ll be seen as difficult, not team players, or even unloyal or lacking in dedication. Asking people to play devil’s advocate gives them permission — indeed encourages them — to disagree. Empowering people to safely disagree can prevent groupthink and identify potential problems with a plan.

However, being the first person to speak up can be tough. So in this post, we’ll explore using a large language model to play devil’s advocate — to start the conversation off.

The following image demonstrates prompting an LLM to play the role of devil’s advocate and argue against a proposed plan.

*Black text is what I typed, highlighted text is output from the model.

A large language model plays the role of devil’s advocate and argues against a proposed plan.
A large language model plays the role of devil’s advocate and argues against a proposed plan.

IBM watsonx.ai

The image above shows a new tool for experimenting with prompting LLMs, called Prompt Lab, in watsonx.ai.

MURAL

Mural is an online tool that is like a virtual whiteboard: you can draw shapes, stick notes, and move things around. It’s a fabulous tool for visually organizing ideas, designing solutions, and collaborating with teammates — in real time or asynchronously. I love using MURAL for team collaboration!

Exercise

The team is considering some ideas to grow our business. We want to use the devil’s advocate approach to identify potential problems and risks with our ideas. We want to perform this activity collaboratively in MURAL. We want to seed our mural with sample arguments against our plans generated by a large language model.

Prerequisites

Step 1: Create the sample notebook in watsonx.ai

In watsonx.ai, create a Python notebook from this url:
Devil’s advocate notebook

See: Creating a notebook

Python notebook in IBM watsonx.ai
Python notebook in watsonx.ai

Step 2: Section A of the notebook

Follow the instructions and run through the cells in Section A to discover the best prompt.

A prompt is the text and parameters you send to a large language model to cause it to generate the desired output.

The easiest way to figure out the most appropriate model and the most effective prompt text and parameters is to experiment in a tool like the Prompt Lab in watsonx.ai. After you have discovered the best combination, then you can move on to using the foundation models Python library to build your solution.

See:

Prompting a model to list arguments against a plan
Prompting a model to list arguments against a plan.

Step 3: Section B of the notebook

Follow the instructions and run through the cells in Section B to prepare a mural to work with.

A mural containing 4 proposed plans for growing a business
A mural containing 4 proposed plans for growing a business.

Note: To use the MURAL API, you need to generate an OAuth 2.0 token. You can do that by setting up OAuth 2.0 in Postman or by deploying an OAuth 2.0 application.

Step 4: Section C of the notebook

Follow the instructions and run through the cells in Section C to generate arguments against each of the four proposed plans and then post the arguments on sticky notes in the appropriate quadrant of the mural.

Arguments against proposed plans
A mural containing 4 proposed plans for growing a business. The mural also contains arguments against each of those plans, generated by a large language model, posted in the appropriate quadrant.

Conclusion

What can you do with a tool that generates text as coherent as a human might write, but that doesn’t understand what it’s writing? One that doesn’t know true from false, fact from fiction, or right from wrong? You could fine-tune the model, build guardrails to avoid egregious errors, and add fact-checking functionality to wrangle output. But the simplest use cases would be ones that require creative story-telling that is informed by common knowledge, where factual accuracy isn’t required, and where indeterminate or wacky content is a feature not a bug.

--

--

Sarah Packowski

Design, build AI solutions by day. Experiment with input devices, drones, IoT, smart farming by night.