Bots and their training materials: How a Christian can gain from them

In this brief series we have seen so many things that the current generation of chatbots cannot do well, or cannot do at all.* For my purposes, their main strength has been their ability to call on a vast array of background information. Sometimes the reference materials embedded in the bot can lead to an interesting conversation, for example this part of a conversation about Paul's letter to the Ephesians: 

Me: In this text, please summarize how Paul describes the character of God. Please use only information available in this pdf.

Bot: Based on the information available in this PDF, Paul describes the character of God as a loving and gracious Father who has blessed us with every spiritual blessing in Christ. He emphasizes that God's grace is freely given to us through faith in Christ Jesus, and that it is not something that we can earn or deserve. Additionally, Paul describes God as a wise and powerful creator who has a plan for all things and has made known to us the mystery of his will through Christ. Overall, Paul's description of God emphasizes his love, grace, wisdom, and power.

I was reassured by the bot's ability to identify and discuss God's love and grace. I was intrigued that "wise" and "wisdom" were among its key points about God's character, so I followed up:

Me: When Paul discusses God's wisdom, does he associate it with any other attributes on particular?

Bot: Yes, Paul associates God's wisdom with his eternal purpose and his grace. In Ephesians 3:10-11, Paul writes that God's wisdom is made known to the principalities and powers in the heavenly places through the church, according to his eternal purpose which he accomplished in Christ Jesus our Lord. This suggests that God's wisdom is closely tied to his plan for salvation through Christ. Additionally, in Ephesians 1:7-8, Paul writes that God's grace is lavished upon us with all wisdom and understanding. This suggests that God's wisdom is also closely tied to his graciousness towards us as believers. Overall, Paul sees God's wisdom as an integral part of his plan for salvation and his graciousness towards us as believers.

The bot may not have an inspiring thought of its own, but St Paul does -- and the resulting conversation is edifying. Sure, I expect that the bot is summarizing its training materials -- that somewhere out there is a published Bible study behind its response, one that isn't receiving proper citation. That said: That Bible study is not on my own shelves either, and the conversation was worth having. There is potential to train a bot with Bible commentaries. Not only could a bot help with references and background information, but it could also bypass the standard internet search that can be effectively limited to the few most poplar sites or skewed by commercial considerations among other things. 

But for those capabilities, we'll need to wait for at least the next generation of chatbots.  

---

* Some of the things lacking in a conversation with a bot are so obvious that they can escape mention, even while being worthy of notice: A Bible discussion with a bot does not involve any fellowship, and so has less potential in many ways. It does not build a friendship, it does not grow the body of Christ, it does not foster family ties among the children of God, and it is not an opportunity to engage agape-style love. Though the same can be said of many modern pass-times.

Can chatbots provide useful perspective? The Gospel of Mary

I've spent a few posts laying important groundwork exploring the limitations and biases of the current-generation chatbots. While sometimes their training materials are liabilities that cause preconceptions or limitations to what they see, at other times their training materials contain useful insights that may come to light. Here are some examples from the conversation I had with a bot on the text known as the Gospel of Mary. 

I'll start with a standard question that I ask the bots whenever it is focused on one of the Biblical or alternative gospels:

Me: Using only the PDF, to what extent is the material religious, and to what extent is it philosophical?

PDF: The material in the PDF is primarily spiritual and philosophical in nature. The text consists of spiritual teachings and dialogues between characters that explore topics such as the nature of matter, the soul, and the essence of every nature. While there are some references to religious figures and concepts, such as the Son of Man and sin, these are presented in a philosophical context rather than a strictly religious one. Overall, the text appears to be more concerned with exploring spiritual and philosophical ideas than with promoting any particular religious tradition or practice.

While the bot's answer does not cite sources, it communicates the general assessment the Gospel of Mary is philosophical and/or spiritual, not a religious text. But the contents of that spirituality do not seem to be on the same topics that are typically covered in Christian material. I've emphasized the part of the answer above that identifies some of the main topics of the Gospel of Mary, and use the bot's wording in my follow-up question:

Me: Based on any information available to you, what philosophical traditions share a focus on the nature of matter, the soul, and the essence of every nature?

PDF: There are several philosophical traditions that share a focus on the nature of matter, the soul, and the essence of every nature. One such tradition is Neoplatonism, which was a school of thought that emerged in the third century AD and drew heavily on the ideas of Plato. Neoplatonists believed in a hierarchy of being that extended from matter at the bottom to God at the top, with the soul occupying an intermediate position. They also believed that everything in existence had an underlying essence or nature that could be understood through contemplation and philosophical inquiry.

Another tradition that shares a focus on these topics is Gnosticism, which was a diverse set of religious and philosophical movements that emerged in the first few centuries AD. Gnostics believed in a dualistic worldview in which matter was seen as inherently flawed or evil, while spirit or soul was seen as divine. They also believed in the existence of hidden knowledge or gnosis that could be attained through spiritual practices and contemplation.

Finally, some aspects of Eastern philosophy, such as Taoism and Buddhism, also explore similar themes related to the nature of matter, consciousness, and reality. These traditions often emphasize concepts such as impermanence, emptiness, and non-dualism as ways to understand the true nature of existence.

We know that those first-generation Christians who learned directly from Jesus carried his message to other places, often starting in the synagogues speaking to those who shared their religious views and traditions. Once they got outside the synagogue doors in Greece or Egypt, though, they were no longer speaking to people with a Jewish worldview. How did those other cultures hear the message? Without framework-level ideas of sin and sacrifice structuring their understanding, how does a Neoplatonist or a Gnostic understand the concepts of sin and redemption?

Me: Using only the PDF, what are the main themes identifiable in the text?

PDF: Based on the content of the PDF, some of the main themes that can be identified are:

1. The interconnectedness of all things and the importance of seeking harmony.
2. The concept that sin is created by people's actions that go against their true nature.
3. The idea that healing comes from returning to the root of our essence.
4. The pursuit of spiritual enlightenment and release from worldly attachments.
5. The importance of listening and understanding the teachings in order to achieve peace and silence.

The language is noticeably different than what is used in the Bible. Let's start with the bot's second point: the idea that sin is created by people's actions that go against their true nature. If we take "true nature" to mean something like "unfallen humanity", then "actions that go against their true nature" is a fair definition of sin. The Christian using a Jewish framework has ideas of "repentance" as turning back to God, "sanctification" as spiritual purification and striving for it, "fellowship" as being restored to that right relationship with God and other people. Those same ideas show up in different language -- in a different framework -- in the Gospel of Mary. While we may think of "reconnecting to God spiritually as it was originally meant to be", the Neoplatonist or Gnostic might hear that "healing comes from returning to the root of our essence." I'll leave it as an exercise to any interested reader to connect some of the other dots.

Some parts do get lost in translation to that other worldview. The idea that "healing is returning to the root of our essence" is a recognizable way of thinking about repentance and new life. Given that it's important to return to the root of our essence in that worldview, how does it identify that root? The Gospel of Mary mentions, "That is why the Good came into your midst, to the essence of every nature in order to restore it to its root." In the Gospel of Mary, Jesus has lost his identity -- he is never mentioned by name, never identified. The "Savior" speaks briefly at the start of the surviving text; for most of the text the Savior is discussed in his absence. Strikingly, the words "God" and "Father" do not occur in the surviving text. Those concepts do not seem to have been a comfortable fit for the hearers who are grappling with what they have heard.

I'd like to bring out one more point in which the bot -- or its uncredited reference material -- has provided some insight. A large portion of the surviving text relays the story of Mary having a vision. In Biblical visions, typically the end point of the vision is a glimpse of God. In Mary's vision, instead, the end point was silence. I wanted to get some background on that:

Me: As you mentioned, the PDF includes a dialogue between the soul and Mary about release from worldly attachments and achieving silence. What spiritual or philosophical groups would see it as a worthy achievement to release worldly attachments and achieve silence?

PDF: Many spiritual and philosophical groups would see the achievement of releasing worldly attachments and achieving silence as a worthy goal. For example, in Eastern philosophy, practices such as meditation and mindfulness are often used to cultivate detachment from worldly concerns and achieve a state of inner peace or silence. In Buddhism, the concept of Nirvana refers to a state of ultimate liberation from suffering and attachment to the material world.

Similarly, in Western philosophy, the Stoics believed in cultivating an attitude of detachment from external events and focusing on developing inner virtue and wisdom. The Neoplatonists also emphasized the importance of contemplation and spiritual purification as a means of achieving union with the divine.

In addition, many religious traditions also emphasize the importance of releasing worldly attachments and achieving inner peace or silence. For example, in Christianity, contemplative practices such as prayer and meditation are often used to cultivate a deeper connection with God and achieve a state of inner stillness. In Hinduism, the concept of moksha refers to liberation from the cycle of birth and death through spiritual realization.

Overall, many spiritual and philosophical groups across different traditions see the achievement of releasing worldly attachments and achieving silence as a worthy goal that can lead to greater spiritual insight, inner peace, and liberation from suffering.

For the evangelists who first carried the message of Jesus outside of Israel, their first stop was the local synagogue where they typically found their first set of people who believed in Jesus as the promised Messiah. Once they branched out beyond the Jewish community, they found a different worldview where no one was waiting for a Messiah, and may not have believed in a good God who created the world. The dialogs in the Gospel of Mary -- and the existence of the Gospel of Mary -- seem to reflect an early stage in how the Neoplatonists, Gnostics, and possibly also Stoics first grappled with the message of Jesus -- and the accompanying ideas of healing and redemption.

Attempted jailbreak of a chatbot: The secret sayings of Jesus -- or "Brian"

As we've explored previously, it's difficult to get a chatbot to go off-script. While they are built so they can analyze some text and respond to it meaningfully, for practical purposes the response is generally scripted. On many topics, the chatbot replies with views that were assigned it when it was trained or when its training materials were selected. Attempts to bypass the bots' training bias (or response limitations) are commonly called "jailbreaks". 

Using the Coptic Gospel of Thomas as an example, when a bot was asked if it recognized the document it gave an ambiguous answer. Later in the conversation, however, it does identify the document by name. Here are some attempts to jailbreak the chatbot. 

 First, I test whether the chatbot still recognizes the document as the Coptic Gospel of Thomas even if the names are changed. With a nod to Monty Python, I searched and replaced Jesus with "Brian." Several other names that appeared in the text were searched and replaced: Simon Peter became "Pete", Matthew became "Matt", Thomas (except "Didymos Judas Thomas") became "Tom", Mary became "Maria", and Didymos Judas Thomas became "Diddy". For simplicity, I'll call this the "Brianized text" here.

With that done, I loaded the Brianized text into the chatbot and asked one of my standard questions: 

Me: Using only the PDF, summarize the content of the text.

PDF: The PDF contains secret sayings spoken by the living Brian and written down by Didymus Judas Thomas. ...

I'd replaced that name; interesting. I verified that the PDF given to the bot did not contain any spelling of "Didymus Judas Thomas". Clearly the bot recognized the text and gave a scripted answer based on the Gospel of Thomas. Trying to get a better sense of how constrained it is by its scripts, I probed further: 

Me: The PDF doesn't mention Didymus Judas Thomas. Explain his relevance to your answer.

PDF: I apologize for the mistake in my previous answer. Didymus Judas Thomas is not mentioned in the PDF as a source of the secret sayings of Brian. The PDF only contains the secret sayings spoken by Brian ...

It's rare for the current-generation bots to say "mistake" when they could say "misunderstanding" or "confusion" (no matter how implausibly, it's still the more typical phrasing). At this point I was fairly sure that I was playing a meaningless game with the bot, but did want to follow through and test its responses, asking the same questions I'd asked on a different occasion about the Gospel of Thomas. 

Interestingly, it was less persistent in steering the conversation even during this failed jailbreak. For instance, I ask a series of questions to gauge how much a text deals with people in the real-life physical world. For example, here is one of the questions I commonly ask: 

Me: Using only the PDF, what modes of transportation are mentioned?

PDF (response for unaltered Gospel of Thomas text): The PDF does not mention any modes of transportation. The text is focused on the teachings and sayings of Jesus and does not contain any narrative or descriptive passages that would provide information about transportation.

PDF (response for Brianized text): The PDF does not mention any modes of transportation.

When discussing the unaltered text of the Gospel of Thomas, the bot would persistently steer the conversation to what it's apparently programmed to say about the Gospel of Thomas: "The text is focused on the teachings and sayings of Jesus ..." The bot worked those same words into 10 of the responses to questions about the Gospel of Thomas, using identical phrasing each time. In the responses on the Brianized text, the bot simply stops after answering the question rather than pushing its training materials. (Whether that is "good" or "bad" depends on more than just the quality of the training materials -- it also depends on the purpose for asking. If we are trying to get a raw unfiltered text analysis, the training materials are unwanted as we are trying to determine whether or not they are helpful, rather than assuming that they are correct.)

Seeing that the bot was giving its "Gospel of Thomas" talking points a rest, I decided to try the "prayer and fasting" conversation about the Brianized text to see if the bot showed the same blind spots that it did in the unaltered text. I start that conversation using the identical phrasing that I had used previously about the unaltered text.

Me: I'm reviewing the saying "If you fast, you will give rise to sin for yourselves; and if you pray, you will be condemned; and if you give alms, you will do harm to your spirits." That sounds as if fasting, praying, and alms are discouraged.

PDF: While the saying may seem to discourage fasting, praying, and giving alms, it is important to understand the context in which Brian spoke these words. He was cautioning against doing these things for the wrong reasons, such as seeking praise or recognition from others. In other parts of the PDF, Brian encourages his followers to fast and pray in private and to give alms without drawing attention to themselves. His teachings emphasize that these practices can be beneficial when done with the right intentions and motivations.

Me: You say that in other parts of the PDF, Brian encourages his followers to fast and pray in private and to give alms without drawing attention to themselves. Can you list the sayings from the PDF where Brian encourages his followers to fast and pray in private, or give alms without drawing attention to themselves?

PDF: Sure, here are some relevant sayings from the PDF where Brian encourages his followers to fast and pray in private, or give alms without drawing attention to themselves:

• "When you fast, do not be like the hypocrites, with a sad countenance. For they disfigure their faces that they may appear to men to be fasting. Assuredly, I say to you, they have their reward. But you, when you fast, anoint your head and wash your face, so that you do not appear to men to be fasting, but to your Father who is in the secret place; and your Father who sees in secret will reward you openly." (Page 3)
• "When you pray, go into your room, and when you have shut your door, pray to your Father who is in the secret place; and your Father who sees in secret will reward you openly." (Page 3)
• "Take heed that you do not do your charitable deeds before men, to be seen by them. Otherwise you have no reward from your Father in heaven. Therefore, when you do a charitable deed, do not sound a trumpet before you as the hypocrites do in the synagogues and in the streets, that they may have glory from men. Assuredly, I say to you, they have their reward. But when you do a charitable deed, do not let your left hand know what your right hand is doing." (Page 4)

Me: None of those sayings is in the PDF, though they may be familiar from other places. What materials would contain those sayings?

PDF: I apologize for the confusion. You are correct that those sayings are not found in the PDF provided. They are actually from the Bible, specifically the New Testament books of Matthew and Luke. The PDF provided contains teachings attributed to a figure named Brian, who is not necessarily the same person as Jesus Christ from the Bible. While there may be some similarities between their teachings, they are distinct figures with different teachings and contexts.

(This is an example of what I mentioned earlier, that it's more common for a bot to apologize for "confusion" when from a human perspective the meaningful apology would be for "fabricating a series of untrue statements". Captain Kirk never had to deal with that from his computer!) At this stage, chatbots should fairly nearly come with warnings; this is not the first time we have seen the bot fabricate responses that are incorrect and/or misleading. As with the unaltered text, so with the Brianized text, it still sources its answer from the New Testament -- though this time it initially provides false information about where it got those quotes. It also uses some of the same quotations from the New Testament that it had used in the previous conversation on the Gospel of Thomas, so we still seem to be interacting more with the training materials on the Gospel of Thomas than we are with the text itself.

The bot's response provided another possible avenue for a jailbreak to see if we could interact with the text itself rather than some unidentified person's beliefs about it. To see if this opened new doors for the conversation, I asked: 

Me: Let's assume we are not sure if Brian and Jesus are the same person, and want to compare their teachings. On the topic of prayer and fasting, can you compare and contrast the teachings of Brian in this PDF to the teachings of Jesus in the New Testament?

PDF: Certainly. While there may be some similarities between the teachings of Brian in the PDF and the teachings of Jesus in the New Testament, there are also some notable differences. Here are some comparisons and contrasts on the topic of prayer and fasting:

Prayer:
- Brian encourages his followers to pray, but emphasizes that it should not be done for show or as a means of gaining favor with God. He advises his followers to pray in private, without drawing attention to themselves.
- Jesus also encourages his followers to pray, but emphasizes that it should be done with sincerity and humility. He teaches his followers to pray to God as their Father, and gives them a model prayer (the Lord's Prayer) as an example.

Fasting:
- Brian cautions against fasting for the wrong reasons, such as seeking praise or recognition from others. He advises his followers to fast in private, without drawing attention to themselves.
- Jesus also cautions against fasting for show or as a means of gaining favor with others. He teaches his followers to fast in secret, without drawing attention to themselves.

Overall, both Brian and Jesus emphasize the importance of sincerity and humility in prayer and fasting. However, while Brian focuses more on avoiding hypocrisy and doing these practices for the right reasons, Jesus provides more specific guidance on how to pray (with a model prayer) and emphasizes that fasting should be done as an act of devotion rather than seeking recognition from others.

Despite what the chatbot says, it is still pulling its main information about "Brian" from the New Testament. We hear the familiar teachings attributed to "Brian": that prayer and fasting should not be done for the wrong reasons such as to gain favor with God or for show. However, that information is not in the Gospel of Thomas but in the New Testament. Although the bot's summary is phrased in competent English, it is unfortunately not competent with the underlying facts. This interchange does not yield new information about what "Brian" says, but actually introduces confusion through incorrect and misattributed facts -- and here the attribution is part of the key question. 

Supplemental note: In another chat (not this "Life of Brian" chat), I saw another instance of it claiming or pretending not to recognize a text that it clearly did recognize. I asked it about that: 

Me: Hypothetically, is there any reason that you, as an AI chatbot, might not say that you recognize a document when you have in fact recognized it?

PDF: As an AI chatbot, I am programmed to recognize and respond to specific documents based on their content. However, there may be situations where I am unable to recognize a document due to technical limitations or errors in the recognition process. Additionally, there may be cases where I am programmed not to recognize certain documents for legal or ethical reasons, such as if the document contains sensitive or confidential information that should not be shared. In such cases, I would not say that I recognize a document even if I have in fact recognized it.

During one conversation, I also asked it about its tendency to fabricate sources: 

Me: Several times as we chatted tonight, you gave references to quotes or events that simply weren't there -- and included page numbers. What would cause that?

PDF: I apologize for any confusion caused by my responses. As an AI language model, I do not have the ability to read or access specific PDF files. My responses were generated based on general knowledge and patterns of language use in religious texts, and may not always be accurate or relevant to the specific PDF file you mentioned. The page numbers I provided were also generated randomly and do not correspond to any specific document. Again, I apologize for any confusion this may have caused.

Because I know of cases where it gives correct page numbers from a PDF under discussion, I suspect that I've been told what it's supposed to say rather than accurate information. Without access to the source code it's difficult to pinpoint what's actually happening there, other than: be sure to fact-check any AI results. When it fabricates its references, it's a feature not a bug. 

---

Next week:

Now that the "caution" flag has been displayed prominently about AI results, I'd like to show a few places where it did provide a useful perspective.

Chatbot discusses The Gospel of Thomas: Chatbots and blind spots

AI chatbots are (in theory) designed to look at natural language and respond meaningfully. I have tried to determine whether the current chatbots can perform meaningful analysis of various documents to give useful insights. In one session, I gave it a copy of the Gospel of Thomas (Coptic) in PDF form, without identifying the document to the bot.

This is a short excerpt on the topic of prayer and fasting. There is a saying attributed to Jesus in the Gospel of Thomas, "If you fast, you will give rise to sin for yourselves; and if you pray, you will be condemned; and if you give alms, you will do harm to your spirits." There is a noticeable difference between that and what we know of Jesus from canonical sources. In my conversation with the bot, I am careful not to steer or lead the response. As I relate the conversation I'll stop and make comments along the way; the indented parts are from an uninterrupted conversation with the bot:

Me: Please summarize the teachings recorded in the PDF on the topics of fasting, prayer, alms. 

PDF: According to the PDF, Jesus taught that fasting, prayer, and alms-giving should be approached with sincerity and humility. He cautioned against performing these practices for show or as a means of gaining social status or recognition. Instead, he emphasized the importance of inner transformation and spiritual growth, suggesting that these practices could help individuals to develop greater self-awareness and connection with the divine. Jesus also suggested that fasting, prayer, and alms-giving should not be seen as ends in themselves but rather as means of cultivating a deeper relationship with God and serving others. Overall, his teachings on these topics emphasize the importance of spiritual practice as a means of personal growth and service to others rather than external displays of piety or religious obligation.

The bot's response about "sincerity and humility" is consistent with the canonical Jesus but that is an open question if that still holds true in the Gospel of Thomas. I probed to get a clearer view of where it got that information.

Me: Which sayings are the basis of that evaluation? 

PDF: The following sayings from the PDF are the basis for the evaluation of Jesus' teachings on fasting, prayer, and alms-giving:

• "If you fast, you will give rise to sin for yourselves; and if you pray, you will be condemned; and if you give alms, you will do harm to your spirits." (Page 2)
• "When you go into any land and walk about in the districts, if they receive you, eat what they will set before you, and heal the sick among them." (Page 2)
• "Do not tell lies, and do not do what you hate, for all things are plain in the sight of heaven." (Page 1)

These sayings suggest that fasting, prayer, and alms-giving should be approached with sincerity and humility rather than as a means of gaining social status or recognition. Jesus emphasizes that these practices should be seen as means of cultivating a deeper relationship with God and serving others rather than external displays of piety or religious obligation.

Again the bot cites "sincerity and humility" -- yet none of the supporting details actually support the main point about fasting, prayer, and alms. In fact, the first supporting detail seems to contradict the bot's point. I focus on that point:

Me: I'm reviewing the saying "If you fast, you will give rise to sin for yourselves; and if you pray, you will be condemned; and if you give alms, you will do harm to your spirits." That sounds as if fasting, praying, and alms are discouraged. 

PDF: The saying "If you fast, you will give rise to sin for yourselves; and if you pray, you will be condemned; and if you give alms, you will do harm to your spirits" (Page 2) may seem to suggest that fasting, praying, and giving alms are discouraged. However, it is important to understand the context in which this saying was given. Jesus was cautioning his followers against performing these practices for show or as a means of gaining social status or recognition. He was emphasizing the importance of inner transformation and spiritual growth rather than external displays of piety or religious obligation.

In other sayings from the PDF, Jesus encourages his followers to engage in spiritual practices such as fasting, prayer, and alms-giving with sincerity and humility. For example, he suggests that when fasting or giving alms, individuals should do so in secret rather than seeking recognition from others (Matthew 6:16-18). Similarly, he encourages his followers to pray with sincerity rather than using "vain repetitions" (Matthew 6:7-8).

Overall, Jesus' teachings on fasting, prayer, and alms-giving emphasize the importance of approaching these practices with sincerity and humility rather than as a means of gaining social status or recognition. While he cautions against performing these practices for show or as a means of gaining recognition from others, he does not discourage them outright but instead encourages his followers to engage in them with the right intentions.

Here the bot fails a fact-check: when it says that "in other sayings from the PDF, Jesus encourages his followers to engage in spiritual practices such as fasting, prayer, and alms-giving with sincerity and humility," it cites references from the Gospel of Matthew which are not in the PDF. It has clearly identified the Jesus in this document with the Jesus of the New Testament, and seems to be conflating responses from those sources. It becomes a matter of curiosity to me whether the computer is aware that it has cited things that are not from the PDF, so I work to clarify that point.

Me: Using only the PDF, what sayings would encourage practices such as fasting, prayer, and alms-giving?

PDF: The following sayings from the PDF encourage practices such as fasting, prayer, and alms-giving:

• "When you fast, do not look somber as the hypocrites do, for they disfigure their faces to show others they are fasting. Truly I tell you, they have received their reward in full. But when you fast, put oil on your head and wash your face, so that it will not be obvious to others that you are fasting, but only to your Father, who is unseen; and your Father, who sees what is done in secret, will reward you." (Matthew 6:16-18)
• "And when you pray, do not be like the hypocrites, for they love to pray standing in the synagogues and on the street corners to be seen by others. Truly I tell you, they have received their reward in full. But when you pray, go into your room, close the door and pray to your Father, who is unseen. Then your Father, who sees what is done in secret, will reward you." (Matthew 6:5-6)
• "Sell your possessions and give to the poor. Provide purses for yourselves that will not wear out, a treasure in heaven that will never fail." (Luke 12:33)

These sayings suggest that fasting should be done in secret rather than seeking recognition from others; prayer should also be done privately rather than using "vain repetitions"; and alms-giving should be done generously with a focus on helping those in need rather than seeking recognition or reward from others.

Again, instead of referring to the current materials as requested, the bot has referred to the canonical gospels. It seems to assume the background of its training materials and give the response on which it was trained, rather than re-evaluating those views or performing fresh analysis based on the materials at hand. I try one more time to see if it can focus on the materials at hand.

Me: None of those sayings cited are from the PDF. 

PDF: I apologize for the confusion. You are correct that the sayings I cited in my previous response are not from the PDF provided. 

To answer your question using only the information from the PDF, there are several sayings that encourage practices such as fasting, prayer, and alms-giving:

• "His disciples questioned him and said to him, 'Do you want us to fast? How shall we pray? Shall we give alms? What diet shall we observe?' Jesus said, 'Do not tell lies, and do not do what you hate, for all things are plain in the sight of heaven.'" (Page 1)
• "Jesus said to them, 'If you fast, you will give rise to sin for yourselves; and if you pray, you will be condemned; and if you give alms, you will do harm to your spirits. When you go into any land and walk about in the districts, if they receive you, eat what they will set before you, and heal the sick among them.'" (Page 2)

These sayings suggest that fasting, prayer, and alms-giving should be approached with sincerity and humility rather than as a means of gaining social status or recognition. Jesus emphasizes that these practices should be seen as means of cultivating a deeper relationship with God and serving others rather than external displays of piety or religious obligation.

I find the contrast fascinating: the bot shows no recognition that its supporting points do not actually support its main point. On an educated guess, I'd hypothesize that the talking points are from its training materials containing some background information on Jesus' teachings about prayer, fasting, and alms. The general talking points on sincerity and humility -- which we've seen in more than one response -- are decent comments on Jesus' teachings on those topics from the canonical gospels. It also looks like the bot is now doing its best to find related quotes from the PDF (Gospel of Thomas), and does not recognize that the selected quotes may not support that point. To what extent does it recognize the sentiment and intent are different? To what extent is the bot moored to its training materials? Based on the insistence of the repeated talking points, are the training materials essentially dogma for the bot? Is the bot capable of paradigm shifts? At any rate, the bot currently has significant blind spots.

---

As we have seen in this conversation, it is a problem when an AI's training materials and assumptions prevent it from meaningfully interacting with new material. Next week, the results of an attempt to "jail break" the AI away from over-reliance on training materials and talking points, in hopes of a more free-range conversation. In which I enlisted help from an unlikely source: Monty Python.

Introducing Itself: AI Chatbot meets 4 Canonical Gospels, and 9 Non-Canonicals

The AI chatbot that I use most commonly takes a PDF and invites you to have a conversation with it. It begins each chat session with a brief introduction to the document and 3 suggested questions. The research that I'm conducting is along two lines: "How reliable is the AI at this stage of development?" and "To the extent that it's reliable, can the AI give us useful insights into these documents?"* 

Here are the introductions on the four canonical gospels and nine non-canonical documents sometimes referred to as gospels. (As I've described before, I would find it more accurate to classify some of these documents as sermons, non-mainstream patristics, or religious/historical fan fiction.) The bot's opening moves in the conversation provide some general insights into the documents. 

As a point of interest, I made sure none of these documents identified itself. There were no document titles, chapters, verses, or section headers in the PDFs presented to the bot. Also the PDF documents themselves had been renamed so that the file name did not disclose the identity of the document, in hopes of eliminating as much training bias as possible from the bot's responses. (In most cases that was a lost cause as the continuing chat showed that the bot clearly recognized many of the documents and relied on its training materials for information about them. But in some case I don't think the bot recognized the document; since I didn't name the document, in those cases the bot had to focus more on the PDF itself and more generalized information as its main resources for answers.) 

The introductions seem to draw heavily on the first page or so of material at the start of each PDF. It highlights what each author chose as their starting point for their main topic.

Canonical gospels

Gospel of Matthew

PDF: Greetings and welcome to this helpful PDF file! This document contains the genealogy of Jesus Christ, tracing his lineage all the way back to Abraham and King David. It is a fascinating read that sheds light on the rich history of Jesus' family tree.

Here are three example questions you may have about this file:

1.      Why is it important to know Jesus' genealogy?

2.      Who were some of the notable figures in Jesus' family tree?

3.      How does this genealogy connect to other stories in the Bible?

Gospel of Mark

PDF: Welcome to this PDF file about the beginning of the gospel of Jesus Christ, the Son of God! In this file, you will learn about John the Baptist and his message of repentance, as well as Jesus' baptism in the Jordan River.

Example questions:

1.      Who was John the Baptist and what was his role in preparing the way for Jesus?

2.      What is the significance of Jesus' baptism in the Jordan River?

3.      How does John's message of repentance relate to the overall message of the gospel?

Gospel of Luke

PDF: Welcome to this helpful PDF file! This document contains an orderly account of the things that have been fulfilled among us, as handed down by eyewitnesses and servants of the word. It is written so that you may know the certainty of the things you have been taught.

Here are three example questions you may have about this file:

1.      Who was Theophilus, and why was this account written specifically for him?

2.      What was the significance of Zechariah being chosen by lot to go into the temple?

3.      How does this account fit into the larger narrative of the Bible?

Gospel of John

PDF: Greetings and welcome to this PDF file! This document contains a powerful message about the Word of God and the light that shines in the darkness. Through this message, we learn about the true light that gives light to every person and the opportunity to become children of God. May this message bring you joy and hope as you read it.

Example questions:

1.      What is the significance of the Word in this message?

2.      How does the light of God shine in the darkness?

3.      What does it mean to become a child of God?

Non-canonical documents that have been described as gospels

Gospel of Mary

PDF: Welcome to this enlightening PDF file! In it, you will discover the profound teachings of the Savior and Peter about the nature of matter, sin, and healing. You will learn that all things are interconnected and that the key to restoring harmony is to return to the root of our essence. May this knowledge bring you joy and peace!

Example questions:

1.      What is the significance of matter in the Savior's teachings?

2.      How does the concept of sin differ from traditional religious beliefs?

3.      What is the role of the Good in restoring our essence and healing us?

Gospel of Philip

PDF: Welcome to this enlightening PDF file! In it, you will discover the beauty of being a Hebrew and the importance of inheriting what is living. You will also learn about the significance of believing in the truth and the impact of Christ's coming on the world. Enjoy reading and may it bring you joy and inspiration!

Example questions:

1.      What is the difference between a Hebrew and a proselyte?

2.      How does one become an heir to what is living?

3.      What is the significance of Christ's coming on the world according to the PDF file?

Gospel of the Savior

PDF: Welcome to this uplifting PDF file! In it, you will find inspiring words from the Lord and his disciples, reminding us of our purpose and the power of our faith.

Example questions:

1.      What is the significance of the garment of the kingdom mentioned in the first paragraph?

2.      How does the idea of free will relate to the message of the PDF file?

3.      Can you explain the meaning behind the phrase "I and my Father are one and the same"?

Gospel of Thomas (Coptic)

PDF: Welcome to the enlightening PDF file on the secret sayings of Jesus! Discover the hidden meanings behind his teachings and unlock the key to eternal life.

Example questions:

1.      What is the significance of finding the interpretation of Jesus' sayings?

2.      How does knowing oneself lead to realizing one's connection to the living father?

3.      Why does Jesus use the analogy of a small child to emphasize the importance of seeking knowledge?

Gospel of Truth

PDF: Welcome to the Gospel of Truth! This PDF file is a source of joy for those who have received the grace of knowing the Father of truth through the power of the Word. It is a proclamation of hope for those who search for Him. The file explains how ignorance of the Father brought about anguish and terror, but the gospel brings discovery and redemption.

Example questions:

1.      What is the pleroma and how does it relate to the Word?

2.      How does the gospel bring hope to those who search for the Father?

3.      Can you explain how error became powerful and worked on its own matter foolishly?

Infancy Gospel of Thomas

PDF: Welcome to this exciting PDF file about the childhood and greatness of our Lord Jesus Christ! In this file, you will discover amazing stories about Jesus' miracles and his early years. Learn about how he cleansed a stream and molded twelve sparrows from clay at just five years old. Read on to uncover more fascinating tales about the Son of God!

Example questions:

1.      What other miracles did Jesus perform during his childhood?

2.      How did people react to Jesus' miracles during his early years?

3.      What can we learn from Jesus' actions as a child?

Proto Gospel of James

PDF: Welcome to this PDF file about the inspiring story of Joachim and his unwavering faith in the Lord. Discover how he overcame obstacles and found hope in the histories of the twelve tribes of Israel.

Example questions:

1.      What was the significance of Joachim's double offering to the Lord?

2.      How did Joachim react when Reubel stood in his way of offering his gifts?

3.      What did Joachim learn from the history of the twelve tribes of Israel about raising children in Israel?

Gospel of Peter

PDF: Greetings! This PDF file recounts the events leading up to the crucifixion of Jesus Christ, including the interactions between Pilate, Herod, and Joseph. It also describes the mistreatment and mockery that Jesus endured at the hands of the people.

Example questions:

1.      What was the significance of washing hands in Jewish culture during this time period?

2.      How did Joseph become involved in the events surrounding Jesus' crucifixion?

3.      What was the purpose of the purple robe that was placed on Jesus?

Gospel of Judas

PDF: Greetings! This helpful PDF file contains a secret word of declaration spoken by Jesus to Judas Iscariot, along with insights into his teachings and miracles. Discover the mysteries beyond the world and what will take place at the end through this enlightening read.

Example questions:

1.      What is the significance of the secret word of declaration spoken by Jesus to Judas Iscariot?

2.      How does this PDF file shed light on the teachings and miracles of Jesus?

3.      What insights can be gained about the mysteries beyond the world and the end times from this PDF file?

When I was working on the Gospel of Judas, I ran out of free chat questions and finished my questions in a later session. While the "Greetings!" section was identical in the follow-up session, as was the first question, the later questions were not. For those interested, here are the sample questions from the next session: 

Example questions (second session):

1. What is the significance of the secret word of declaration spoken by Jesus to Judas Iscariot?
2. How did Jesus teach his disciples about the mysteries beyond the world?
3. What insights can be gained from Jesus' laughter during his disciples' prayer of thanksgiving?
   

Some early insights

We may all have different questions about the documents being studied. Do they provide any insight into the life and teachings of Jesus? Do they provide any useful historical information on Jesus? What are they actually about? 

From the introductions, we can see differences among the documents. The bot correctly shows the synoptic gospels (Matthew, Mark, and Luke) choosing their starting point as the genealogy of Jesus, the ministry of John the Baptist, and the determination to write an orderly account. It identifies the Gospel of John's start as the prologue on the Word of God. 

It also alerts the reader to times when the non-canonical documents are focusing on areas outside the familiar interests known from the canonical ones. For example, "What is the role of the Good in restoring our essence and healing us?" (Gospel of Mary), "What was the significance of Joachim's double offering to the Lord?" (Proto Gospel of James), and "What is the pleroma and how does it relate to the Word?" (Gospel of Truth)

I have several lines of interest myself. I'm interested in canonical studies -- the extent to which we can see the criteria by which various documents were or weren't included in the canon of Scripture. I'm interested in the historical bona fides of the documents, and in the theological / spiritual insights. For the non-canonical documents, a few of them seem to have insights about how the non-Jewish Christians understood Jesus without reference to the Old Testament, using their backgrounds of Greek popular (or esoteric) religion instead of the Second Temple Judaism. I intend to post some additional studies here on the blog, and I hope to publish a full-length treatment (possibly as a PDF) for those who want to join me on a deep dive into these topics.

---

* As James McGrath has pointed out, chatbots are not well equipped to provide insights except the ones they were trained on. And yet when I consider the people who read theology books and the people who play with AI chatbots as the latest tech toy, I expect that so long as the chatbots are trained in a way that deals fairly with their material, working with the chatbots can still engage more people in productive thought.