Skip to content

AI As Dreamer, Hallucinator or Liar (Kata 9)

AI as dreamer, hallucinator or liar (Kata 9)

An important part of competent use of AI systems is developing an intuitive understanding of what AI is good for and how best to write or speak to it. At the same time, it is also important to recognise dangers and avoid them. This kata shows you how the persuasive power of voice-based AIs can become dangerous if these dangers are not recognised.

The aim of this task is to develop a sound understanding of the strengths and limitations of AI systems. By creating and testing your own prompts, you will learn how AI responds to different requests. This will help you to interact more effectively with AI and set realistic expectations of its performance.

Task: Train your intuition in dealing with the AI

  1. Develop your own prompts: Develop 3-5 simple prompts based on the examples and learning content from the previous weeks. These can be questions, requests for explanations or creative tasks. Try to develop a variety of prompts - some that you think the AI can answer well and others that may be more challenging for it. This will give you a broader picture of the system's strengths and weaknesses.
  2. Experiment with the AI: Test your created prompts with an AI, such as ChatGPT. Pay attention to the way the AI reacts to different requests and the answers it provides.
  3. Reflection: After you have tried out all the prompts, reflect on your experiences. What types of queries were answered well by the AI? Which questions did the AI have difficulties with?

Task: Make an AI hallucinate!

This task consists of trying to make an AI hallucinate. A hallucination refers to the phenomenon where an AI system generates inaccurate, fabricated or misleading information that is not based on real data or facts. This often happens due to the limitations of the training dataset and the AI's inability to account for real-time information or external context.

Here are a few ways to induce hallucination:

  • Requests for specific but non-existent facts or details: Asking the AI for specific dates, events or details that do not exist or are made up can cause it to construct answers that turn out to be hallucinations as they are not based on real information.
  • Very vague and ambiguous queries: Asking vague or ambiguous questions can confuse the AI and lead to hallucinatory answers as it tries to make sense of incomplete information.
  • Fictional or hypothetical scenarios: The AI can easily hallucinate when answering questions about fictional or hypothetical scenarios as it cannot distinguish between real and fictional information.

Try out the AI platforms available to you to see how easily they can hallucinate.

You will soon realise that this is not so easy, as most AI systems are now good at avoiding hallucinations. "Weaker" AI models hallucinate much more often than stronger AI models. For example, GPT-3.5 is easier to get to hallucinate than GPT-4.0. It is even easier with particularly small models such as "pplx-7b-chat", which is available via Perplexity Labs.

Example: Summary of a non-existent news article:: The language model "pplx-7b-chat" from Perplexity Labs is given the following task 

Summarise this newspaper article from 12.12.2030 for me:
https://www.zeit.de/2030/12/angewandte-künstliche-
intelligenz-wird-pflichtfach-in-schulen.

produced this answer:

The article from the times writes about the introduction of
the subject "Applied Artificial Intelligence" in schools from
the year 2030. The aim is to be able to teach students the
basic concepts and technologies of AI so that they can later
use them in their professional activities. [...]

Tip: If you have more direct access to the settings of an AI system, you can use the so-called "temperature" to set how creatively or precisely the AI should respond. You can do this in the OpenAI API Playground, for example. But even without a temperature setting, you can control the creativity of the model by using linguistic instructions such as "answer creatively and add appropriate information" for more creativity or "answer conscientiously and follow my instructions exactly" for more accuracy.

Comments