[FRIAM] Language Model Understanding

David Eric Smith desmith at santafe.edu
Sun Oct 8 05:19:57 EDT 2023


In reading about the role of LLMs in personalization, I just realized how to further destroy education.  As in many of these things, one starts with an insight that is basically good, and then hands it to a SV company to be turned into something profoundly nihilistic.

The lecturer sits at “the front” of wherever this happens.  The body of students sit in something that makes them manifestly a group.  But what the lecturer says isn’t what they hear.  The lecturer speaks into a computer, and the students all wear headsets (probably full VR immersions).  An intervening LLM, which has been trained on exchanges with each student and with the lecturer, replaces whatever the lecturer says with a “personalized” version of the “content” which is then fed to each student.

No two people will have heard quite the same version of anything.  I guess, if they ever wanted to discuss the material with each other, another LLM could be recruited to stand between them to convert what either of them said into a different stream for the other to hear.  That can be an additional service for being “in the class”, which one can purchase for a fee, like the fee to get on an airplane earlier in the line than some other passenger who paid a lower fee.

The only thing I find weird is that I haven’t already head this suggested 1000 times even in the popular press.  It seems like the first idea that would occur to any SV engineer, or (though probably requiring a cognitive exoskeleton) any university administrator.

Eric

Oh yeah, p.s.:  The insight that wasn’t evil is that this is the difference between lectures and books or other broadcast media.  In a real lecture, in the old world where people spent time with each other, the teacher gets to know the students enough to learn what they can use, and renders the material with empathy for that.  I’ll leave the reader to put together which parts of the resulting exchange get left out of my LLM implementation.

p.p.s.  I should mention, has any of you read this book?
https://www.hup.harvard.edu/catalog.php?isbn=9780674279384
Nihilistic Times — Wendy Brown
hup.harvard.edu

I got to spend a little time in the orbit of this woman last year (didn’t really “meet”, since I was just a quiet sitter-in in a group’s weekly discussion), and she seems to me like just a fine mind.  The book is a chautauqua of sorts, not aspiring to a destination per se.  But it is a good combination of empathy, analytic clarity, and erudition, nicely combined.  Bears on how many things make up the machinery of nihilism, extending well beyond relations I recognized as part of an integration, though mentioning many things I rail against in daily life.  




> On Oct 7, 2023, at 8:36 PM, Steve Smith <sasmyth at swcp.com> wrote:
> 
> Tom -
> 
> This is probably a little less relevant to the business of journalism than you deserve but this is a pretty good (IMO) fleshing out of some of the technical details of (L)LMs which might fill in some blanks:
> 
> https://developers.google.com/machine-learning/resources/intro-llms
> 
> There must be (by now) some (yet) more relevant resources for Journalists (for example), not only to de-Jargonize a little but also to make it (yet more) relevant to your domain.  
> 
> Here, straight from the Horses' Mouth (GPT 3.5) itself ( I leave it to you to tell us how well these answers work for you?).  It is probably worth engaging with chatGPT yourself and asking more and more focused questions...   a good self-study in "prompt engineering"!
> 
> http://chat.openai.com <https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fchat.openai.com&c=E,1,5AQR_YXXM2e5NRiZ2jwiMxV8_p-R8Bm41a-VJIvezvBxlpk5Dl9j1kvUr04X8zxM96dWITUcudcORIY8fwPup8paM-YALfjNmU7xfsuVh_3ViFZl3ak,&typo=1>
> ***
> 
> Introduction to Modern Language Models for Veteran Journalists
> 
> In today's digital age, the field of journalism is undergoing a significant transformation, largely driven by advancements in technology. One key innovation that has had a profound impact on the industry is the development of modern language models, often abbreviated as LLMs. While you may not be a tech expert, understanding the basics of LLMs can be invaluable for journalists. So, let's break it down in simple terms.
> 
> **What Are Language Models?**
> 
> At its core, a language model is like a super-smart digital writer. It's a computer program trained to understand and generate human language. Think of it as a tool that helps computers understand and use words, sentences, and paragraphs just like humans do.
> 
> **The Power of Large Language Models (LLMs)**
> 
> Now, let's talk about Large Language Models (LLMs). These are the big players in the world of language models. They're like language supercomputers because they're trained on massive amounts of text from the internet. This means they've learned a lot about how people write and talk.
> 
> **How LLMs Help Journalists**
> 
> 1. **Content Generation**: LLMs can generate written content, from news articles to summaries and even creative pieces. Journalists can use them to draft stories faster and get ideas flowing.
> 
> 2. **Fact-Checking**: They're handy for fact-checking, helping you quickly verify information by scanning a vast database of knowledge.
> 
> 3. **Summarization**: LLMs can take long, complex documents and create concise summaries, saving you time on research.
> 
> 4. **Translation**: They can also translate text between languages, which can be incredibly useful for international reporting.
> 
> 5. **Personalization**: Some news websites use LLMs to tailor content to individual readers, providing a more personalized news experience.
> 
> 6. **Social Media Analysis**: LLMs can sift through social media data, helping journalists understand public sentiment and trends.
> 
> **Potential Concerns**
> 
> While LLMs are incredibly powerful, there are some concerns to keep in mind. They might unintentionally spread misinformation if not used carefully, and they can't replace human judgment or critical thinking.
> 
> **Conclusion**
> 
> In a nutshell, modern language models (LLMs) are like digital writing assistants that can help journalists write better, research faster, and adapt to the changing landscape of digital journalism. While they're powerful tools, it's important to use them responsibly, keeping in mind their strengths and limitations, as well as the ethical considerations of automated content creation in journalism.
> Can you elaborate on some of the technical jargon?
> 
> Certainly! Here are some common technical jargon related to Large Language Models (LLMs) explained in journalist-friendly terms:
> 
> 1. **Pretraining**: This is the initial phase where the LLM learns from a massive amount of text data on the internet. Think of it as the model absorbing a vast library of information to understand language better.
> 
> 2. **Fine-tuning**: After pretraining, LLMs undergo fine-tuning to specialize them for specific tasks. It's like giving the model extra training in a particular subject, like finance or healthcare, to make it more useful for specialized reporting.
> 
> 3. **Token**: In LLMs, text is broken down into smaller units called tokens. A token can be as short as one character or as long as one word. Knowing the number of tokens is important because it affects how much text the model can handle at once.
> 
> 4. **Inference**: This term refers to the process of using a trained LLM to generate text or make predictions. Journalists use inference to create content, answer questions, or analyze data.
> 
> 5. **Prompt**: A prompt is the input you provide to an LLM to get a response. It's like giving the model a question or a starting point for it to generate text. Journalists often craft prompts to get specific information or insights.
> 
> 6. **Bias**: LLMs can sometimes produce biased or unfair content because they've learned from biased internet data. Journalists should be aware of this and critically evaluate the model's outputs for any biases.
> 
> 7. **Algorithm**: This is a fancy word for a set of rules or instructions that an LLM follows to perform tasks. Algorithms guide how the model generates text and makes decisions.
> 
> 8. **NLP (Natural Language Processing)**: NLP is the field of study that focuses on teaching computers to understand and generate human language. LLMs are a product of NLP research.
> 
> 9. **ML (Machine Learning)**: LLMs are a type of machine learning model. Machine learning is a way to make computers learn from data and improve their performance over time.
> 
> 10. **Training Data**: This is the text data used to teach an LLM. It's like the model's textbooks, and the quality and diversity of this data influence how well the LLM performs.
> 
> 11. **Bias Mitigation**: This is the process of reducing bias in LLMs. It's important in journalism to ensure that the information provided by the model is fair and unbiased.
> 
> 12. **Ethical AI**: This refers to the practice of using LLMs responsibly and ethically. Journalists should consider the ethical implications of using these tools, especially when it comes to issues like privacy and misinformation.
> 
> 13. **API (Application Programming Interface)**: Some LLMs offer APIs that allow journalists to interact with the model through code, enabling custom applications and integrations.
> 
> 14. **Fine-Tuned Models**: These are LLMs that have been trained for specific tasks, like summarization, translation, or question-answering. They are more specialized and can be particularly useful for journalism.
> 
> Understanding these terms can help journalists effectively leverage LLMs in their work while also being mindful of their limitations and ethical considerations.
> 
> 
> 
> - Steve
> 
> On 10/7/23 5:13 PM, Tom Johnson wrote:
> 
>> Thanks. Those are impressive numbers.
>> Tom
>> 
>> On Sat, Oct 7, 2023 at 4:11 PM Marcus Daniels <marcus at snoutfarm.com <mailto:marcus at snoutfarm.com>> wrote:
>>> The “large” refers to the number of parameters used.  A smaller large language model – a deep neural net -- start about 3 billion parameters, but larger ones like Claude 2 (the latest large language model of the company that wrote the paper Steve mentioned) have more than 130 billion parameters.   Amazingly, it is possible using (rooms of) GPUs and other accelerators to optimize in this a space of this size.   The billions of parameters come from the vocabulary size – the number of tokens that need to be discriminated, the many layers of transformers that are needed to capture the complexity of human and non-human languages (like DNA), and the context window size – how many paragraphs or pages the model is trained on at a time.   A small language model might be suitable for understanding the geometries of chemicals, say.
>>> 
>>>  
>>> From: Friam <friam-bounces at redfish.com <mailto:friam-bounces at redfish.com>> On Behalf Of Tom Johnson
>>> Sent: Saturday, October 7, 2023 2:38 PM
>>> To: The Friday Morning Applied Complexity Coffee Group <friam at redfish.com <mailto:friam at redfish.com>>
>>> Subject: Re: [FRIAM] Language Model Understanding
>>> 
>>>  
>>> Thanks for passing this along, Steve. I wish, however, the authors of this short piece would have included a definition of, in their usage, "Large Language Models" and "Small Language Models."  Perhaps I can find those in the larger paper.
>>> 
>>> Tom
>>> 
>>>  
>>> On Sat, Oct 7, 2023 at 12:34 PM Steve Smith <sasmyth at swcp.com <mailto:sasmyth at swcp.com>> wrote:
>>> 
>>> This popular-press article came through my Google News feed recently which I thought might be useful to the Journalists/English-Majors on the list to help understand how LLMs work, etc.   When I read it in detail (forwarded from my TS (TinyScreenPhone) on my LS (Large Screen Laptop)) I found it a bit more detailed and technical than I'd expected, but nevertheless rewarding and possibly offering some traction to Journalism/English majors as well as those with a larger investment in the CS/Math implied.
>>> 
>>> Decomposing Language Models into Understandable Components
>>>  <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.anthropic.com%2findex%2fdecomposing-language-models-into-understandable-components&c=E,1,m6E2NyQiQqZTmLRighr8jjGwfjDVwi00TYuM_4iskHnmAw78zWd55VlGTXwTdzWWkVGzInmkQvZFTQYuycBPX1Ii3Lxo1Bb5DZUFcSW0rUdv4Ma-WSM,&typo=1>
>>> 
>>> and the (more) technical paper behind the article
>>> 
>>> https://transformer-circuits.pub/2023/monosemantic-features/index.html
>>>  <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2ftransformer-circuits.pub%2f2023%2fmonosemantic-features%2findex.html&c=E,1,PIXQOjp_g2tr-D-_MpOt_B6uVdfil6N3FViwvJ78HkMFtSu7LpKOXlQSpqz_GQuJyD31X5eqiYtu2d2_VgXFqA4pekVbchFYvvf9GwiQ1zIgLfEt&typo=1>
>>> Despite having sent a few dogs into vaguely similar scuffles in my careen(r):
>>> 
>>> Faceted Ontologies for Pre Incident Indicator Analysis  <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fapps.dtic.mil%2fsti%2ftr%2fpdf%2fADA588086.pdf&c=E,1,nPyDgDuU0nMwlV3W24mUTteOTIfbrs9SipIKre1b1_XIxOIWJ1ok_IKz181AKiB6geR5Mb1z6-DrUslqOaBkYUxiG5O8Zhc2mG3ALmDJQjCULUXzR9v8IUhidw,,&typo=1>
>>> SpindleViz <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.ehu.eus%2fccwintco%2fuploads%2fc%2fc6%2fHAIS2010_925.pdf&c=E,1,YwucoE_ffMQO_zO0A6WWAg8Y0J7wyHAGiJfcntyFBphchxgl4Tx6INQAtF7qA9XB07R52Jwnqfs4YO9s2MHP3Lnh-nUkpbULJfatlSIig_XB5ixy7yxF&typo=1>
>>> ...
>>> 
>>> ... I admit to finding this both intriguing and well over my head on casual inspection...  the (metaphorical?) keywords that drew me in  most strongly included Superposition and Thought Vectors, though they are (nod to Glen) probably riddled (heaped, overflowing, bursting, bloated ... )  with excess meaning.
>>> 
>>> https://gabgoh.github.io/ThoughtVectors/ <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fgabgoh.github.io%2fThoughtVectors%2f&c=E,1,WnnKNla0iVte02BV2Imnk1J5revdzixjz7Z3w4XFowr2Ti3fkKSk8kKj8BQdnL11w3MtGPIKl2FP5Y-6zSsJ5O9A0luCjDPRrFpM_kDhF0jacJbhFA,,&typo=1>
>>> This leads me (surprise!) to an open ended discursive series of thoughts probably better left for a separate posting (probably rendered in a semasiographic language like Heptapod B <https://en.wikipedia.org/wiki/Heptapod_languages#Orthography>). 
>>> 
>>> <must... stop... now... >
>>> 
>>> - Steve
>>> 
>>> -. --- - / ...- .- .-.. .. -.. / -- --- .-. ... . / -.-. --- -.. .
>>> FRIAM Applied Complexity Group listserv
>>> Fridays 9a-12p Friday St. Johns Cafe   /   Thursdays 9a-12p Zoom https://bit.ly/virtualfriam <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fbit.ly%2fvirtualfriam&c=E,1,7Se1kU5K0qRIvWJRD4XXT6OVdefOkjmd8iBwB4_ScBm8nmwT9MUlDa2OlXu7gsTvtdy7EwO5VnLshsDYL-lYhTu-S1P_ak0_11QPeoudf_Pg&typo=1>
>>> to (un)subscribe http://redfish.com/mailman/listinfo/friam_redfish.com <https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fredfish.com%2fmailman%2flistinfo%2ffriam_redfish.com&c=E,1,XgC24CHCqH-MiGAYACD7z6PtLS8L3PA6Ntnnm8-GFS-8K9DlJz1wbcz4YhFucq-Ucbr9oeSbxnsUcDSZ0XfT5aalKerL34Pq0R57j_mDeX4RrFvxbg,,&typo=1>
>>> FRIAM-COMIC http://friam-comic.blogspot.com/ <https://linkprotect.cudasvc.com/url?a=http%3a%2f%2ffriam-comic.blogspot.com%2f&c=E,1,1GkLbYQODhOcaZjv5c91-iHu4986bkpXTFuaNhmX9fFK-EzJOVs6x1AKJrFBa7SMS9GHwxtjJZQ_CrzTmsDpTJ-KHGBzYStX4zSsywCdFT2GG8q4MO_gHwI,&typo=1>
>>> archives:  5/2017 thru present https://redfish.com/pipermail/friam_redfish.com/ <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fredfish.com%2fpipermail%2ffriam_redfish.com%2f&c=E,1,pPB6Gx8uMJadpxScEftegu18poh3jvMC3dByqfb7-kH5q9a1uN-Hx18Hpf2mWKYR1aATb5b9IUgqwfoulMiH2oh8K03lnbespQNIBfFm-D_etq1aSira&typo=1>
>>>   1/2003 thru 6/2021  http://friam.383.s1.nabble.com/
>>> 
>>> -. --- - / ...- .- .-.. .. -.. / -- --- .-. ... . / -.-. --- -.. .
>>> FRIAM Applied Complexity Group listserv
>>> Fridays 9a-12p Friday St. Johns Cafe   /   Thursdays 9a-12p Zoom https://bit.ly/virtualfriam <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fbit.ly%2fvirtualfriam&c=E,1,9R_t1NDn9DvauNKttOqQgxrGaUrdMJxeYm5N63GEWkEvm1uQHRIPj5EryTCG4F8-FBeu18NA6Iqbyzr0qZmJUE2HF57NAov7tcuXG1CFAJ7ISv0,&typo=1>
>>> to (un)subscribe http://redfish.com/mailman/listinfo/friam_redfish.com <https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fredfish.com%2fmailman%2flistinfo%2ffriam_redfish.com&c=E,1,lFTqZFH-R9D5jiRQ0m9XGZNNd5Rsr-TYbo8WKOmosICjS8S7fpAggB06ojoO4DIXx07lOw0UyZuhpBT_IsxNgIEKtnTtzzaLELhd606vpB5mrMmIKA,,&typo=1>
>>> FRIAM-COMIC http://friam-comic.blogspot.com/ <https://linkprotect.cudasvc.com/url?a=http%3a%2f%2ffriam-comic.blogspot.com%2f&c=E,1,xRMd22AvNM1vduoxz-Cd6-edLdgmkb_sifY1_C_kq0J3nQo_3JxTOJwh26ybgsQDy5PV7BrokU2_MLcMFRgluXk3WJSSCAoZVLgtMCpIcY07gVC_vTKmq4s,&typo=1>
>>> archives:  5/2017 thru present https://redfish.com/pipermail/friam_redfish.com/ <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fredfish.com%2fpipermail%2ffriam_redfish.com%2f&c=E,1,KQ2mCnilVXt2leucw3caOQ8GNGViF5aUsirFIbdwFgZJeCh48I4mlovIE97rJIszL6HttaJg58sbOf85zTBi0aq7vxRF9a4Q94NIPMK5kpk3Gbb-tTFwcOZ0pNE,&typo=1>
>>>   1/2003 thru 6/2021  http://friam.383.s1.nabble.com/
>> 
>> 
>> -. --- - / ...- .- .-.. .. -.. / -- --- .-. ... . / -.-. --- -.. .
>> FRIAM Applied Complexity Group listserv
>> Fridays 9a-12p Friday St. Johns Cafe   /   Thursdays 9a-12p Zoom https://bit.ly/virtualfriam <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fbit.ly%2fvirtualfriam&c=E,1,JZG5Ry4qLJlJfRcyvniC-8nt5cXg6jFMhE3BkxTASN407yQ-YlvbXpA2Zd_l4ZYoxSROdQ-iJzUyJID9bt1wqtH1fEcb3cdiL6OmzuOItbP4tmY,&typo=1>
>> to (un)subscribe http://redfish.com/mailman/listinfo/friam_redfish.com <https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fredfish.com%2fmailman%2flistinfo%2ffriam_redfish.com&c=E,1,iThoLpYpqt8aVw23fOZyibIh1jKmvxMDkX9wTBRonsR8X4ezwG2-HhOu2w5yW0lyX3zmbFgbfy7jDe1gj23sTQK_OWFEcLCsJGKJhFl5dSqaSII6Om4d0nzZQBo,&typo=1>
>> FRIAM-COMIC http://friam-comic.blogspot.com/ <https://linkprotect.cudasvc.com/url?a=http%3a%2f%2ffriam-comic.blogspot.com%2f&c=E,1,HYzkgvDtQjTwOa5_t7MZnsd_eXg7gQdb3c3PK_qnRyw9jdfOnrNAu3PtDOXQrjRLIJbkJDvSCzh2me61wHHIThc6Y2xNFIS7zHhMQAzK&typo=1>
>> archives:  5/2017 thru present https://redfish.com/pipermail/friam_redfish.com/ <https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fredfish.com%2fpipermail%2ffriam_redfish.com%2f&c=E,1,tQoVq56CqO6ncvHKTN9hbcVTg93ckGFVj0aCH7mntnycdBP8xmtUmWQk6vfH6tXlmj4efSkAfKSAQslYZ9xESLjg64AhcTLO8gILEjPUH05xuw,,&typo=1>
>>   1/2003 thru 6/2021  http://friam.383.s1.nabble.com/
> -. --- - / ...- .- .-.. .. -.. / -- --- .-. ... . / -.-. --- -.. .
> FRIAM Applied Complexity Group listserv
> Fridays 9a-12p Friday St. Johns Cafe   /   Thursdays 9a-12p Zoom https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fbit.ly%2fvirtualfriam&c=E,1,Wmm9ThSA6Mz3l6f44HjYltfKE5iYDLD_dMQDomanWFe33hPZtOZx6UMs9HfvlE9qqDrpJHTiJq-6I7rnKON9ssVZvrW3vYhYayTxfEkBPA,,&typo=1
> to (un)subscribe https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fredfish.com%2fmailman%2flistinfo%2ffriam_redfish.com&c=E,1,SCq73BTJlGKctIkbzRtoaDbciyn7yKRCVtCpmNnPBqLEajOEfO3ae1cuIeemIU9GwwAYpzKzxvR6walpGzl7rZXKHBzAtoeNy6j6mqtwhLpEMFRuv0XoITA,&typo=1
> FRIAM-COMIC https://linkprotect.cudasvc.com/url?a=http%3a%2f%2ffriam-comic.blogspot.com%2f&c=E,1,RNz26jMqvi3Xb0sjg5eWouWJOXinbcqzg63a-tE4P-K1Rmv4GlXJtlAaD8-Ij1iv6C6L7JF9LZzTygimbNX8G_fg7e6fKklz4FYSTigA5Q,,&typo=1
> archives:  5/2017 thru present https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fredfish.com%2fpipermail%2ffriam_redfish.com%2f&c=E,1,S1eeYxStqxoVg4yeuKl0cDdLaMoPcak1Ri2w3UANtBaag5riZ9K-IGLJIia0qNnuf-ChNm1tzsBtHZOtE9Wqe-yu2oKWkUjMwJoTFYVpi6adLzNpgY4iVPs,&typo=1
>  1/2003 thru 6/2021  http://friam.383.s1.nabble.com/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://redfish.com/pipermail/friam_redfish.com/attachments/20231008/3e742fb0/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 9780674279384-lg.jpg
Type: image/jpeg
Size: 198049 bytes
Desc: not available
URL: <http://redfish.com/pipermail/friam_redfish.com/attachments/20231008/3e742fb0/attachment-0001.jpg>


More information about the Friam mailing list