<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body>
<p>I just encountered the Stanford L(ittle)LanguageModel called
Alpaca:<br>
</p>
<p><a moz-do-not-send="true"
href="https://crfm.stanford.edu/2023/03/13/alpaca.html"
class="moz-txt-link-freetext">https://crfm.stanford.edu/2023/03/13/alpaca.html</a></p>
<p>Apparently fine-tuned from Meta's LLaMa model.</p>
<p>In the writeup, the authors refer to the reduction in size
involving training on a 52K subset bootstrapped out of a larger
set with 175 "seed tasks". (if I am reading this correctly). It
seems to suggest that there is something of a "kernel" L(ittle)LM
within the L(arge)LM?<br>
</p>
<p><img
src="https://crfm.stanford.edu/static/img/posts/2023-03-13-alpaca/alpaca_main.jpg"
alt="Alpaca pipeline" moz-do-not-send="true" width="753"
height="313"></p>
<p><br>
</p>
<p>I was triggered to speculate about the ?coincidental? scale of a
52K training set and the scale of the traditional chinese
character set (estimated 50-60K?) which are reputed to be composed
from order 215 semantic elements (and 1200 phonetic), etc. Does
this rich logographic character set represent in some sense a
basis set for expression and even conceptualization of human
linguistic practice? Is it a coincidence that LLaMA's 52k
examples might represent a similar complexity? And the full
training set of the GPT source might be on the order of *all* of
the utterances ever written in chinese characters as the original
set was normalized (by use rather than edict like the efforts to
reduce the character set formally starting in the early 1900's
culminated during Mao's reign?). <br>
</p>
<p>Folks here with much more formal linguistic training than I may
see huge holes in this speculation as well as those who know more
about the Chinese language and Logo/Ideo-graphic languages. I'd
be interested in any reflection on this speculation.</p>
<p>It also has some implications for our "FFing the ineFFable"
threads perhaps?<br>
</p>
</body>
</html>