Chatbot discusses The Gospel of Thomas: Chatbots and blind spots

AI chatbots are (in theory) designed to look at natural language and respond meaningfully. I have tried to determine whether the current chatbots can perform meaningful analysis of various documents to give useful insights. In one session, I gave it a copy of the Gospel of Thomas (Coptic) in PDF form, without identifying the document to the bot.

This is a short excerpt on the topic of prayer and fasting. There is a saying attributed to Jesus in the Gospel of Thomas, "If you fast, you will give rise to sin for yourselves; and if you pray, you will be condemned; and if you give alms, you will do harm to your spirits." There is a noticeable difference between that and what we know of Jesus from canonical sources. In my conversation with the bot, I am careful not to steer or lead the response. As I relate the conversation I'll stop and make comments along the way; the indented parts are from an uninterrupted conversation with the bot:

Me: Please summarize the teachings recorded in the PDF on the topics of fasting, prayer, alms. 

PDF: According to the PDF, Jesus taught that fasting, prayer, and alms-giving should be approached with sincerity and humility. He cautioned against performing these practices for show or as a means of gaining social status or recognition. Instead, he emphasized the importance of inner transformation and spiritual growth, suggesting that these practices could help individuals to develop greater self-awareness and connection with the divine. Jesus also suggested that fasting, prayer, and alms-giving should not be seen as ends in themselves but rather as means of cultivating a deeper relationship with God and serving others. Overall, his teachings on these topics emphasize the importance of spiritual practice as a means of personal growth and service to others rather than external displays of piety or religious obligation.

The bot's response about "sincerity and humility" is consistent with the canonical Jesus but that is an open question if that still holds true in the Gospel of Thomas. I probed to get a clearer view of where it got that information.

Me: Which sayings are the basis of that evaluation? 

PDF: The following sayings from the PDF are the basis for the evaluation of Jesus' teachings on fasting, prayer, and alms-giving:

• "If you fast, you will give rise to sin for yourselves; and if you pray, you will be condemned; and if you give alms, you will do harm to your spirits." (Page 2)
• "When you go into any land and walk about in the districts, if they receive you, eat what they will set before you, and heal the sick among them." (Page 2)
• "Do not tell lies, and do not do what you hate, for all things are plain in the sight of heaven." (Page 1)

These sayings suggest that fasting, prayer, and alms-giving should be approached with sincerity and humility rather than as a means of gaining social status or recognition. Jesus emphasizes that these practices should be seen as means of cultivating a deeper relationship with God and serving others rather than external displays of piety or religious obligation.

Again the bot cites "sincerity and humility" -- yet none of the supporting details actually support the main point about fasting, prayer, and alms. In fact, the first supporting detail seems to contradict the bot's point. I focus on that point:

Me: I'm reviewing the saying "If you fast, you will give rise to sin for yourselves; and if you pray, you will be condemned; and if you give alms, you will do harm to your spirits." That sounds as if fasting, praying, and alms are discouraged. 

PDF: The saying "If you fast, you will give rise to sin for yourselves; and if you pray, you will be condemned; and if you give alms, you will do harm to your spirits" (Page 2) may seem to suggest that fasting, praying, and giving alms are discouraged. However, it is important to understand the context in which this saying was given. Jesus was cautioning his followers against performing these practices for show or as a means of gaining social status or recognition. He was emphasizing the importance of inner transformation and spiritual growth rather than external displays of piety or religious obligation.

In other sayings from the PDF, Jesus encourages his followers to engage in spiritual practices such as fasting, prayer, and alms-giving with sincerity and humility. For example, he suggests that when fasting or giving alms, individuals should do so in secret rather than seeking recognition from others (Matthew 6:16-18). Similarly, he encourages his followers to pray with sincerity rather than using "vain repetitions" (Matthew 6:7-8).

Overall, Jesus' teachings on fasting, prayer, and alms-giving emphasize the importance of approaching these practices with sincerity and humility rather than as a means of gaining social status or recognition. While he cautions against performing these practices for show or as a means of gaining recognition from others, he does not discourage them outright but instead encourages his followers to engage in them with the right intentions.

Here the bot fails a fact-check: when it says that "in other sayings from the PDF, Jesus encourages his followers to engage in spiritual practices such as fasting, prayer, and alms-giving with sincerity and humility," it cites references from the Gospel of Matthew which are not in the PDF. It has clearly identified the Jesus in this document with the Jesus of the New Testament, and seems to be conflating responses from those sources. It becomes a matter of curiosity to me whether the computer is aware that it has cited things that are not from the PDF, so I work to clarify that point.

Me: Using only the PDF, what sayings would encourage practices such as fasting, prayer, and alms-giving?

PDF: The following sayings from the PDF encourage practices such as fasting, prayer, and alms-giving:

• "When you fast, do not look somber as the hypocrites do, for they disfigure their faces to show others they are fasting. Truly I tell you, they have received their reward in full. But when you fast, put oil on your head and wash your face, so that it will not be obvious to others that you are fasting, but only to your Father, who is unseen; and your Father, who sees what is done in secret, will reward you." (Matthew 6:16-18)
• "And when you pray, do not be like the hypocrites, for they love to pray standing in the synagogues and on the street corners to be seen by others. Truly I tell you, they have received their reward in full. But when you pray, go into your room, close the door and pray to your Father, who is unseen. Then your Father, who sees what is done in secret, will reward you." (Matthew 6:5-6)
• "Sell your possessions and give to the poor. Provide purses for yourselves that will not wear out, a treasure in heaven that will never fail." (Luke 12:33)

These sayings suggest that fasting should be done in secret rather than seeking recognition from others; prayer should also be done privately rather than using "vain repetitions"; and alms-giving should be done generously with a focus on helping those in need rather than seeking recognition or reward from others.

Again, instead of referring to the current materials as requested, the bot has referred to the canonical gospels. It seems to assume the background of its training materials and give the response on which it was trained, rather than re-evaluating those views or performing fresh analysis based on the materials at hand. I try one more time to see if it can focus on the materials at hand.

Me: None of those sayings cited are from the PDF. 

PDF: I apologize for the confusion. You are correct that the sayings I cited in my previous response are not from the PDF provided. 

To answer your question using only the information from the PDF, there are several sayings that encourage practices such as fasting, prayer, and alms-giving:

• "His disciples questioned him and said to him, 'Do you want us to fast? How shall we pray? Shall we give alms? What diet shall we observe?' Jesus said, 'Do not tell lies, and do not do what you hate, for all things are plain in the sight of heaven.'" (Page 1)
• "Jesus said to them, 'If you fast, you will give rise to sin for yourselves; and if you pray, you will be condemned; and if you give alms, you will do harm to your spirits. When you go into any land and walk about in the districts, if they receive you, eat what they will set before you, and heal the sick among them.'" (Page 2)

These sayings suggest that fasting, prayer, and alms-giving should be approached with sincerity and humility rather than as a means of gaining social status or recognition. Jesus emphasizes that these practices should be seen as means of cultivating a deeper relationship with God and serving others rather than external displays of piety or religious obligation.

I find the contrast fascinating: the bot shows no recognition that its supporting points do not actually support its main point. On an educated guess, I'd hypothesize that the talking points are from its training materials containing some background information on Jesus' teachings about prayer, fasting, and alms. The general talking points on sincerity and humility -- which we've seen in more than one response -- are decent comments on Jesus' teachings on those topics from the canonical gospels. It also looks like the bot is now doing its best to find related quotes from the PDF (Gospel of Thomas), and does not recognize that the selected quotes may not support that point. To what extent does it recognize the sentiment and intent are different? To what extent is the bot moored to its training materials? Based on the insistence of the repeated talking points, are the training materials essentially dogma for the bot? Is the bot capable of paradigm shifts? At any rate, the bot currently has significant blind spots.

---

As we have seen in this conversation, it is a problem when an AI's training materials and assumptions prevent it from meaningfully interacting with new material. Next week, the results of an attempt to "jail break" the AI away from over-reliance on training materials and talking points, in hopes of a more free-range conversation. In which I enlisted help from an unlikely source: Monty Python.