Wednesday October 25, 2023; 7:15 AM EDT
- I remember growing suspicious of a certain best-selling NYer nonfiction writer after he wrote a piece about a topic I already knew fairly well. Now, instead of seeing a playful intelligence and verbal flash, I could see an underlying glibness. So now, to myself at least, I recommend testing authors and AI tools on topics I know well, just to see more clearly what their real strengths and weaknesses are.#
- Taking my own advice, I read a ChatGPT document about what writing offers us when employed at the start of a big project. And to keep your readerly suspense to a manageable level, let me say that I found that ChatGPT 3.5 currently loves to exaggerate, to speak with greater confidence than it should, to make things sound more fixed and definitive than it should, in its findings on non-technical topics.#
- [Maybe ChatGPT doesn't yet grapple with the differences in data and language use, the differences in possible certainty, when shifting from technical to non-technical topics. I don't know.]#
- And I found that you can sometimes press ChatGPT to do a better job (of not exaggerating its claims about non-technical matters) via a series of follow-up questions. Previously, I wasn't sure that this was the case, but this time around I could see an improvement. [Since we were operating on territory familiar to me.]#
- For example#
- In a passage about starting a big project by doing some writing, ChatGPT opens with this claim:#
- Clarity of Thought: Writing forces you to clarify your thoughts. When you have to put your ideas on paper, you're compelled to think through them more thoroughly, which can help in identifying potential pitfalls or areas of improvement.#
- Well, that seems false to me. It speaks with too much confidence, too categorically, about a trait of writing. If writing forced everyone to think more clearly, then it wouldn't be so gosh darn easy to find bad writing and lousy thinking everywhere you turn in this troubled world of ours. So, written that way by ChatGPT 3.5, it's just not true. It's misleading. The passage doesn't help a person understand what it is about writing that can and should serve us well. In this passage ChatGPT uses strong verbs, [forces, compelled] which all the writing guides recommend, to make the prose sound vigorous and decisive, but it forgets to choose strong verbs that actually reveal the truth.*#
- So I tried a follow-up question. "ChatGPT, friend, what is the evidence that writing forces a person to clarify their thinking?" Here's part of my friend's answer:#
- The act of writing serves as a cognitive tool that externalizes thoughts, making them more tangible, structured, and accessible for further reflection and refinement.#
- Surely the act of writing doesn't make thoughts more structured all by itself. So that's not true. ChatGPT is wrapping a blanket of generalization over a more fine-grained process, which is often a mistake when we reach for clearer thinking, yes? So I followed up again, with this question:#
- "ChatGPT, buddy, you say that 'The act of writing serves as a cognitive tool that externalizes thoughts, making them more tangible, structured, and accessible for further reflection and refinement.' Is it the tool itself that makes thoughts more tangible, structured, and accessible for further reflection and refinement, or is it the approach a particular person takes to using the tool that makes these possible?"#
- Finally, ChatGPT was willing to put the brakes on the misleadingly overconfident verbs, the generalizations that mask useful particulars, and reflect in a more fine-grained fashion on the topic. Here is its concluding paragraph in answer to my second round of questioning:#
- In essence, while writing serves as the medium for externalizing thoughts, it's the active, purposeful, and reflective approach a person takes to using this tool that makes thoughts more tangible, structured, and accessible for further reflection and refinement. Writing becomes a powerful cognitive tool when individuals consciously employ it as a means to clarify their thinking, actively engage with their ideas, and remain open to revision and feedback. The process is dynamic and interactive, with both the tool and the approach being essential components of effective thinking and expression.#
- Now we're getting somewhere. A person could think usefully about the role of writing in a big team project with the help of these more fine-grained ideas.#
- The software itself is powerful and fascinating. On non-technical topics, though, it appears that a person who relies on current ChatGPT 's composing skills for something they really care about risks sounding like an exaggerating, imprecise enthusiast who doesn't know when to move up close to the particulars of a topic. #
- As a writing teacher on the university level for many years, I read many dozens of papers by students who were working through that same stage in their writing and thinking but who had not yet completed the passage across to the other side. The sentences often flickered back and forth between promising ideas and nonsense.** In part, the students were not precise enough readers to see that their own drafts had these rather particular problems.#
- _____________#
- *Yes, ChatGPT uses one trait of good writing more or less in isolation from other traits of good writing, and doing so it weakens its writing. How about that?#
- **This is not just a trait of many writers early in their college years. It's a trait of the drafting process, which is all about crossing over from the disarray of early thoughts to the clarity that writing can help a person find.#