Bible Study --- and Biblical Studies -- in the age of AI Chatbots

Have you ever had a conversation with a computer? A year ago I think most of us would have said, "Of course not!" Today, it's commonly-available technology, though still in its infancy stage. So what do you get when you try to discuss a Biblical text with a chatbot? 

One of the more accessible tools for me is hosted at chatpdf.com, where any pdf can be dropped into the online app. At that site, the free version accepts smaller pdf's and has a limit of 50 questions per day, and works by providing your chosen pdf to a chatbot as the topic of conversation. Here are my impressions so far. 

• The chatbots can offer a different perspective on the texts. If anyone wants a discussion of a familiar text from a different point-of-view, a bot can be helpful and sometimes insightful. 
• My hopes were that the chatbot would be a neutral point-of-view and fully reliable on factual matters; however that was not the case. 
• The bot's responses are not error-free from a factual perspective. For example, I gave the app a pdf of Paul's letter to the Galatians and asked if it identified words or phrases that could indicate an origin in another language. It said no, it could not find any such words or phrases. However, the pdf mentions praying to "Abba", father. I followed up and asked about "Abba" specifically. It then acknowledged that Abba was used in the text, and is a transliteration of an Aramaic word meaning father. I don't know what caused it to originally say there were no such words. Still, an unwary reader could have assumed the bot's answer as correct. The answer did come from a computer, after all. 
• While at times the chatbot is mistaken, it can be difficult to convince a bot to double-check its answer so that the conversation can go forward with more accurate premises. Given the 50-question-per-day limit of the bot that I'm using, I don't always bother to correct it. (It's a little bit like the 3 wishes in the old stories; I'm not keen to waste them.)
• The chatbot does use outside materials in addition to the pdf provided. To avoid outside influences, I routinely ask the chatbot to limit its answers to information available in the pdf. It does not follow those instructions. As a simple example, it would persist in providing chapter-and-verse citations when answering, even though I had carefully provided a pdf that did not contain them in the hopes of running a blind test. 
• The results are not bias-free. One desired advantage -- or imagined / assumed advantage -- of a computer could be a neutral point-of-view about the Bible. But when asked in a specific way, the chatbot will acknowledge that it has vast resources of training materials and background knowledge about the Bible that are informing its answers. That said, it won't name specifics of the materials used to train it, or cite references for its answers. 
• At times the bot answers in a way that, in a human, would be called "steering the conversation." For example, when asked about one thing, it may say that wasn't a main focus of the text and redirect the answer to another topic. 
• On several of the previous points, the answers may depend on which materials were loaded into the bot as training / background information. It is not clear who curated the training materials or what criteria they used to deem certain materials more reliable and desirable. The day may come when chatbots can come pre-loaded with (say) the Catholic Encyclopedia and catechism, or the equivalents for other groups, or for the neo-atheists. What we don't have is a value-neutral chatbot. The fact that it does not disclose its sources makes that more of a concern. 

Some people have experimented with "jailbreak" prompts to allow various chatbots to break out of their pre-programmed / trained viewpoints, some with more success than others. My own conversations have not been jailbreak attempts so much as requests to the bot not to refer to outside materials. Those requests seem to be routinely disregarded. Even when specifically requested to use only the material at hand -- and even after having been called on the fact that it was using other materials and please stop -- still the bot tends to claim it is looking only at the pdf while clearly using information not contained in the pdf. In a human we'd call it either failure to follow instructions or dishonesty. With a chatbot some other options include lack of understanding / failure to communicate, or having been instructed/trained to say one thing and do another. There are signs that the bot trainers may be making it harder to get a chatbot to go off-script. Whether that's desirable is a different question. 

In future posts I plan to share some conversations I've had with the chatbots. For all that the "state of the art" leaves something to be desired, there are some interesting conversations to be had. In this first post, I need to be clear: when reading responses from a chatbot, let the reader beware. We may be used to assuming that a computer's responses are as reliable as a calculator. In the case of the current chatbots, not so much yet.