Sunday, April 23, 2023

Bible Study --- and Biblical Studies -- in the age of AI Chatbots

Have you ever had a conversation with a computer? A year ago I think most of us would have said, "Of course not!" Today, it's commonly-available technology, though still in its infancy stage. So what do you get when you try to discuss a Biblical text with a chatbot? 

One of the more accessible tools for me is hosted at chatpdf.com, where any pdf can be dropped into the online app. At that site, the free version accepts smaller pdf's and has a limit of 50 questions per day, and works by providing your chosen pdf to a chatbot as the topic of conversation. Here are my impressions so far. 

  • The chatbots can offer a different perspective on the texts. If anyone wants a discussion of a familiar text from a different point-of-view, a bot can be helpful and sometimes insightful. 
  • My hopes were that the chatbot would be a neutral point-of-view and fully reliable on factual matters; however that was not the case. 
  • The bot's responses are not error-free from a factual perspective. For example, I gave the app a pdf of Paul's letter to the Galatians and asked if it identified words or phrases that could indicate an origin in another language. It said no, it could not find any such words or phrases. However, the pdf mentions praying to "Abba", father. I followed up and asked about "Abba" specifically. It then acknowledged that Abba was used in the text, and is a transliteration of an Aramaic word meaning father. I don't know what caused it to originally say there were no such words. Still, an unwary reader could have assumed the bot's answer as correct. The answer did come from a computer, after all. 
  • While at times the chatbot is mistaken, it can be difficult to convince a bot to double-check its answer so that the conversation can go forward with more accurate premises. Given the 50-question-per-day limit of the bot that I'm using, I don't always bother to correct it. (It's a little bit like the 3 wishes in the old stories; I'm not keen to waste them.)
  • The chatbot does use outside materials in addition to the pdf provided. To avoid outside influences, I routinely ask the chatbot to limit its answers to information available in the pdf. It does not follow those instructions. As a simple example, it would persist in providing chapter-and-verse citations when answering, even though I had carefully provided a pdf that did not contain them in the hopes of running a blind test. 
  • The results are not bias-free. One desired advantage -- or imagined / assumed advantage -- of a computer could be a neutral point-of-view about the Bible. But when asked in a specific way, the chatbot will acknowledge that it has vast resources of training materials and background knowledge about the Bible that are informing its answers. That said, it won't name specifics of the materials used to train it, or cite references for its answers. 
  • At times the bot answers in a way that, in a human, would be called "steering the conversation." For example, when asked about one thing, it may say that wasn't a main focus of the text and redirect the answer to another topic. 
  • On several of the previous points, the answers may depend on which materials were loaded into the bot as training / background information. It is not clear who curated the training materials or what criteria they used to deem certain materials more reliable and desirable. The day may come when chatbots can come pre-loaded with (say) the Catholic Encyclopedia and catechism, or the equivalents for other groups, or for the neo-atheists. What we don't have is a value-neutral chatbot. The fact that it does not disclose its sources makes that more of a concern. 

Some people have experimented with "jailbreak" prompts to allow various chatbots to break out of their pre-programmed / trained viewpoints, some with more success than others. My own conversations have not been jailbreak attempts so much as requests to the bot not to refer to outside materials. Those requests seem to be routinely disregarded. Even when specifically requested to use only the material at hand -- and even after having been called on the fact that it was using other materials and please stop -- still the bot tends to claim it is looking only at the pdf while clearly using information not contained in the pdf. In a human we'd call it either failure to follow instructions or dishonesty. With a chatbot some other options include lack of understanding / failure to communicate, or having been instructed/trained to say one thing and do another. There are signs that the bot trainers may be making it harder to get a chatbot to go off-script. Whether that's desirable is a different question. 

In future posts I plan to share some conversations I've had with the chatbots. For all that the "state of the art" leaves something to be desired, there are some interesting conversations to be had. In this first post, I need to be clear: when reading responses from a chatbot, let the reader beware. We may be used to assuming that a computer's responses are as reliable as a calculator. In the case of the current chatbots, not so much yet. 

7 comments:

  1. This comment has been removed by the author.

    ReplyDelete
  2. The chatbot is a human speech imitator. What it creates is based on its textual base that it was fed to learn from. This means, in relation to things you wrote, that it will not offer a genuinely new perspective. If it seems to, then you probably aren't reading as many different human-written perspectives as you could. It also means that accurate information that happens to be in the response is just a byproduct of what is woven into that textual base, but as it shuffles it has no mechanism for identifying facts or information, much less ensuring those are preserved in its responses.

    ReplyDelete
  3. Hi James

    You may be interested to see the difference in output when it's given a text that it's familiar with (whether it admits it or not), and a text that it really hasn't seen before. That, and try asking some questions that are definitely not in its data banks. It does try to fly solo with nothing but knowledge of the language and the PDF in hand, and you can get some responses that aren't wholly canned. (Though at one point it devolved to the computer version of "I don't recall saying that", which was entertaining.)

    Though as you say, when it's dealing with a text with training materials in its data banks and a question on areas covered in its data banks, the result is that familiar composite/amalgam of the bot-trainer's approved publications, which isn't prefaced "They say" but may as well have been. When I ask the bot for its sources it's mum. But when I ask it for further reading material on a topic, it seems to suddenly remember some relevant sources. It's all down to figuring out how to ask the question. ;)

    Take care & God bless
    Anne / WF

    ReplyDelete
  4. Hi Martin

    It's definitely interesting to me. :)

    Take care & God bless
    Anne / WF

    ReplyDelete
  5. Hi Anne! Yes, it does respond differently to different questions and a major factor is what it has text about in its language set. Even when it has information that is quite good, or rather I ought to say when its creative response preserves what you and I would recognize as reliable and useful information, it can still do the same jumbling and shuffling exercise when asked for sources to support what it said. A student of mine kindly documented this, which I shared on my blog here: https://www.patheos.com/blogs/religionprof/2023/02/chatgpt-on-mary-magdalene.html

    Not sure whether you saw that already, but if not I think you'll find it interesting. There are real authors, titles, and subtitles of books combined to create new ones!

    ReplyDelete
  6. I read the linked post. That is hilarious! I've heard about it citing sources that are ... erm ... possibly non-existent. To see an example is priceless. Thank you for sharing.

    Take care & God bless
    Anne / WF

    ReplyDelete