Introducing Itself: AI Chatbot meets 4 Canonical Gospels, and 9 Non-Canonicals

The AI chatbot that I use most commonly takes a PDF and invites you to have a conversation with it. It begins each chat session with a brief introduction to the document and 3 suggested questions. The research that I'm conducting is along two lines: "How reliable is the AI at this stage of development?" and "To the extent that it's reliable, can the AI give us useful insights into these documents?"* 

Here are the introductions on the four canonical gospels and nine non-canonical documents sometimes referred to as gospels. (As I've described before, I would find it more accurate to classify some of these documents as sermons, non-mainstream patristics, or religious/historical fan fiction.) The bot's opening moves in the conversation provide some general insights into the documents. 

As a point of interest, I made sure none of these documents identified itself. There were no document titles, chapters, verses, or section headers in the PDFs presented to the bot. Also the PDF documents themselves had been renamed so that the file name did not disclose the identity of the document, in hopes of eliminating as much training bias as possible from the bot's responses. (In most cases that was a lost cause as the continuing chat showed that the bot clearly recognized many of the documents and relied on its training materials for information about them. But in some case I don't think the bot recognized the document; since I didn't name the document, in those cases the bot had to focus more on the PDF itself and more generalized information as its main resources for answers.) 

The introductions seem to draw heavily on the first page or so of material at the start of each PDF. It highlights what each author chose as their starting point for their main topic.

Canonical gospels

Gospel of Matthew

PDF: Greetings and welcome to this helpful PDF file! This document contains the genealogy of Jesus Christ, tracing his lineage all the way back to Abraham and King David. It is a fascinating read that sheds light on the rich history of Jesus' family tree.

Here are three example questions you may have about this file:

1.      Why is it important to know Jesus' genealogy?

2.      Who were some of the notable figures in Jesus' family tree?

3.      How does this genealogy connect to other stories in the Bible?

Gospel of Mark

PDF: Welcome to this PDF file about the beginning of the gospel of Jesus Christ, the Son of God! In this file, you will learn about John the Baptist and his message of repentance, as well as Jesus' baptism in the Jordan River.

Example questions:

1.      Who was John the Baptist and what was his role in preparing the way for Jesus?

2.      What is the significance of Jesus' baptism in the Jordan River?

3.      How does John's message of repentance relate to the overall message of the gospel?

Gospel of Luke

PDF: Welcome to this helpful PDF file! This document contains an orderly account of the things that have been fulfilled among us, as handed down by eyewitnesses and servants of the word. It is written so that you may know the certainty of the things you have been taught.

Here are three example questions you may have about this file:

1.      Who was Theophilus, and why was this account written specifically for him?

2.      What was the significance of Zechariah being chosen by lot to go into the temple?

3.      How does this account fit into the larger narrative of the Bible?

Gospel of John

PDF: Greetings and welcome to this PDF file! This document contains a powerful message about the Word of God and the light that shines in the darkness. Through this message, we learn about the true light that gives light to every person and the opportunity to become children of God. May this message bring you joy and hope as you read it.

Example questions:

1.      What is the significance of the Word in this message?

2.      How does the light of God shine in the darkness?

3.      What does it mean to become a child of God?

Non-canonical documents that have been described as gospels

Gospel of Mary

PDF: Welcome to this enlightening PDF file! In it, you will discover the profound teachings of the Savior and Peter about the nature of matter, sin, and healing. You will learn that all things are interconnected and that the key to restoring harmony is to return to the root of our essence. May this knowledge bring you joy and peace!

Example questions:

1.      What is the significance of matter in the Savior's teachings?

2.      How does the concept of sin differ from traditional religious beliefs?

3.      What is the role of the Good in restoring our essence and healing us?

Gospel of Philip

PDF: Welcome to this enlightening PDF file! In it, you will discover the beauty of being a Hebrew and the importance of inheriting what is living. You will also learn about the significance of believing in the truth and the impact of Christ's coming on the world. Enjoy reading and may it bring you joy and inspiration!

Example questions:

1.      What is the difference between a Hebrew and a proselyte?

2.      How does one become an heir to what is living?

3.      What is the significance of Christ's coming on the world according to the PDF file?

Gospel of the Savior

PDF: Welcome to this uplifting PDF file! In it, you will find inspiring words from the Lord and his disciples, reminding us of our purpose and the power of our faith.

Example questions:

1.      What is the significance of the garment of the kingdom mentioned in the first paragraph?

2.      How does the idea of free will relate to the message of the PDF file?

3.      Can you explain the meaning behind the phrase "I and my Father are one and the same"?

Gospel of Thomas (Coptic)

PDF: Welcome to the enlightening PDF file on the secret sayings of Jesus! Discover the hidden meanings behind his teachings and unlock the key to eternal life.

Example questions:

1.      What is the significance of finding the interpretation of Jesus' sayings?

2.      How does knowing oneself lead to realizing one's connection to the living father?

3.      Why does Jesus use the analogy of a small child to emphasize the importance of seeking knowledge?

Gospel of Truth

PDF: Welcome to the Gospel of Truth! This PDF file is a source of joy for those who have received the grace of knowing the Father of truth through the power of the Word. It is a proclamation of hope for those who search for Him. The file explains how ignorance of the Father brought about anguish and terror, but the gospel brings discovery and redemption.

Example questions:

1.      What is the pleroma and how does it relate to the Word?

2.      How does the gospel bring hope to those who search for the Father?

3.      Can you explain how error became powerful and worked on its own matter foolishly?

Infancy Gospel of Thomas

PDF: Welcome to this exciting PDF file about the childhood and greatness of our Lord Jesus Christ! In this file, you will discover amazing stories about Jesus' miracles and his early years. Learn about how he cleansed a stream and molded twelve sparrows from clay at just five years old. Read on to uncover more fascinating tales about the Son of God!

Example questions:

1.      What other miracles did Jesus perform during his childhood?

2.      How did people react to Jesus' miracles during his early years?

3.      What can we learn from Jesus' actions as a child?

Proto Gospel of James

PDF: Welcome to this PDF file about the inspiring story of Joachim and his unwavering faith in the Lord. Discover how he overcame obstacles and found hope in the histories of the twelve tribes of Israel.

Example questions:

1.      What was the significance of Joachim's double offering to the Lord?

2.      How did Joachim react when Reubel stood in his way of offering his gifts?

3.      What did Joachim learn from the history of the twelve tribes of Israel about raising children in Israel?

Gospel of Peter

PDF: Greetings! This PDF file recounts the events leading up to the crucifixion of Jesus Christ, including the interactions between Pilate, Herod, and Joseph. It also describes the mistreatment and mockery that Jesus endured at the hands of the people.

Example questions:

1.      What was the significance of washing hands in Jewish culture during this time period?

2.      How did Joseph become involved in the events surrounding Jesus' crucifixion?

3.      What was the purpose of the purple robe that was placed on Jesus?

Gospel of Judas

PDF: Greetings! This helpful PDF file contains a secret word of declaration spoken by Jesus to Judas Iscariot, along with insights into his teachings and miracles. Discover the mysteries beyond the world and what will take place at the end through this enlightening read.

Example questions:

1.      What is the significance of the secret word of declaration spoken by Jesus to Judas Iscariot?

2.      How does this PDF file shed light on the teachings and miracles of Jesus?

3.      What insights can be gained about the mysteries beyond the world and the end times from this PDF file?

When I was working on the Gospel of Judas, I ran out of free chat questions and finished my questions in a later session. While the "Greetings!" section was identical in the follow-up session, as was the first question, the later questions were not. For those interested, here are the sample questions from the next session: 

Example questions (second session):

1. What is the significance of the secret word of declaration spoken by Jesus to Judas Iscariot?
2. How did Jesus teach his disciples about the mysteries beyond the world?
3. What insights can be gained from Jesus' laughter during his disciples' prayer of thanksgiving?
   

Some early insights

We may all have different questions about the documents being studied. Do they provide any insight into the life and teachings of Jesus? Do they provide any useful historical information on Jesus? What are they actually about? 

From the introductions, we can see differences among the documents. The bot correctly shows the synoptic gospels (Matthew, Mark, and Luke) choosing their starting point as the genealogy of Jesus, the ministry of John the Baptist, and the determination to write an orderly account. It identifies the Gospel of John's start as the prologue on the Word of God. 

It also alerts the reader to times when the non-canonical documents are focusing on areas outside the familiar interests known from the canonical ones. For example, "What is the role of the Good in restoring our essence and healing us?" (Gospel of Mary), "What was the significance of Joachim's double offering to the Lord?" (Proto Gospel of James), and "What is the pleroma and how does it relate to the Word?" (Gospel of Truth)

I have several lines of interest myself. I'm interested in canonical studies -- the extent to which we can see the criteria by which various documents were or weren't included in the canon of Scripture. I'm interested in the historical bona fides of the documents, and in the theological / spiritual insights. For the non-canonical documents, a few of them seem to have insights about how the non-Jewish Christians understood Jesus without reference to the Old Testament, using their backgrounds of Greek popular (or esoteric) religion instead of the Second Temple Judaism. I intend to post some additional studies here on the blog, and I hope to publish a full-length treatment (possibly as a PDF) for those who want to join me on a deep dive into these topics.

---

* As James McGrath has pointed out, chatbots are not well equipped to provide insights except the ones they were trained on. And yet when I consider the people who read theology books and the people who play with AI chatbots as the latest tech toy, I expect that so long as the chatbots are trained in a way that deals fairly with their material, working with the chatbots can still engage more people in productive thought.

Bible Study --- and Biblical Studies -- in the age of AI Chatbots

Have you ever had a conversation with a computer? A year ago I think most of us would have said, "Of course not!" Today, it's commonly-available technology, though still in its infancy stage. So what do you get when you try to discuss a Biblical text with a chatbot? 

One of the more accessible tools for me is hosted at chatpdf.com, where any pdf can be dropped into the online app. At that site, the free version accepts smaller pdf's and has a limit of 50 questions per day, and works by providing your chosen pdf to a chatbot as the topic of conversation. Here are my impressions so far. 

• The chatbots can offer a different perspective on the texts. If anyone wants a discussion of a familiar text from a different point-of-view, a bot can be helpful and sometimes insightful. 
• My hopes were that the chatbot would be a neutral point-of-view and fully reliable on factual matters; however that was not the case. 
• The bot's responses are not error-free from a factual perspective. For example, I gave the app a pdf of Paul's letter to the Galatians and asked if it identified words or phrases that could indicate an origin in another language. It said no, it could not find any such words or phrases. However, the pdf mentions praying to "Abba", father. I followed up and asked about "Abba" specifically. It then acknowledged that Abba was used in the text, and is a transliteration of an Aramaic word meaning father. I don't know what caused it to originally say there were no such words. Still, an unwary reader could have assumed the bot's answer as correct. The answer did come from a computer, after all. 
• While at times the chatbot is mistaken, it can be difficult to convince a bot to double-check its answer so that the conversation can go forward with more accurate premises. Given the 50-question-per-day limit of the bot that I'm using, I don't always bother to correct it. (It's a little bit like the 3 wishes in the old stories; I'm not keen to waste them.)
• The chatbot does use outside materials in addition to the pdf provided. To avoid outside influences, I routinely ask the chatbot to limit its answers to information available in the pdf. It does not follow those instructions. As a simple example, it would persist in providing chapter-and-verse citations when answering, even though I had carefully provided a pdf that did not contain them in the hopes of running a blind test. 
• The results are not bias-free. One desired advantage -- or imagined / assumed advantage -- of a computer could be a neutral point-of-view about the Bible. But when asked in a specific way, the chatbot will acknowledge that it has vast resources of training materials and background knowledge about the Bible that are informing its answers. That said, it won't name specifics of the materials used to train it, or cite references for its answers. 
• At times the bot answers in a way that, in a human, would be called "steering the conversation." For example, when asked about one thing, it may say that wasn't a main focus of the text and redirect the answer to another topic. 
• On several of the previous points, the answers may depend on which materials were loaded into the bot as training / background information. It is not clear who curated the training materials or what criteria they used to deem certain materials more reliable and desirable. The day may come when chatbots can come pre-loaded with (say) the Catholic Encyclopedia and catechism, or the equivalents for other groups, or for the neo-atheists. What we don't have is a value-neutral chatbot. The fact that it does not disclose its sources makes that more of a concern. 

Some people have experimented with "jailbreak" prompts to allow various chatbots to break out of their pre-programmed / trained viewpoints, some with more success than others. My own conversations have not been jailbreak attempts so much as requests to the bot not to refer to outside materials. Those requests seem to be routinely disregarded. Even when specifically requested to use only the material at hand -- and even after having been called on the fact that it was using other materials and please stop -- still the bot tends to claim it is looking only at the pdf while clearly using information not contained in the pdf. In a human we'd call it either failure to follow instructions or dishonesty. With a chatbot some other options include lack of understanding / failure to communicate, or having been instructed/trained to say one thing and do another. There are signs that the bot trainers may be making it harder to get a chatbot to go off-script. Whether that's desirable is a different question. 

In future posts I plan to share some conversations I've had with the chatbots. For all that the "state of the art" leaves something to be desired, there are some interesting conversations to be had. In this first post, I need to be clear: when reading responses from a chatbot, let the reader beware. We may be used to assuming that a computer's responses are as reliable as a calculator. In the case of the current chatbots, not so much yet.