[FRIAM] FFing the ineFFable with LLMs (Large and Little)...

[Steve Smith]

I just encountered the Stanford L(ittle)LanguageModel called Alpaca:

https://crfm.stanford.edu/2023/03/13/alpaca.html

Apparently fine-tuned from Meta's LLaMa model.

In the writeup, the authors refer to the reduction in size involving 
training on a 52K subset bootstrapped out of a larger set with 175 "seed 
tasks". (if I am reading this correctly).  It seems to suggest that 
there is something of a "kernel" L(ittle)LM within the L(arge)LM?

Alpaca pipeline


I was triggered to speculate about the ?coincidental? scale of a 52K  
training set and the scale of the traditional chinese character set 
(estimated 50-60K?) which are reputed to be composed from order 215 
semantic elements (and 1200 phonetic), etc.   Does this rich logographic 
character set represent in some sense a basis set for expression and 
even conceptualization of human linguistic practice?  Is it a 
coincidence that LLaMA's 52k examples might represent a similar 
complexity?  And the full training set of the GPT source might be on the 
order of *all* of the utterances ever written in chinese characters as 
the original set was normalized (by use rather than edict like the 
efforts to reduce the character set formally starting in the early 
1900's culminated during Mao's reign?).

Folks here with much more formal linguistic training than I may see huge 
holes in this speculation as well as those who know more about the 
Chinese language and Logo/Ideo-graphic languages.    I'd be interested 
in any reflection on this speculation.

It also has some implications for our "FFing the ineFFable" threads perhaps?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://redfish.com/pipermail/friam_redfish.com/attachments/20230322/031589d9/attachment.html>