[FRIAM] A very good piece about ChatGPT.

[Russ Abbott]

Very nice piece by Rich Sutton!

Re NanoGPT. Even though I'm retired, I wanted to learn more about how the
LLMs work. So I started a project in which a number of students and i work
through Karpathy's nanoGPT. We've just started, but I think it will be
worth the effort. Karpathy uses non-trivial Python. Many students don't
know Python at all. So learning Python along with its non-trivial features
as well as becoming familiar with PyTorch, is something of a challenge for
the project. Students (and I, in fact) were not prepared for the amount of
scaffolding background knowledge one had to build for oneself.

The amount fo work required to build the intellectual scaffolding almost
always seems to be an unanticipated problem when learning something new. In
many cases the scaffolding is not just intellectual. Think of what it takes
to learn to play musical instrument!

-- Russ


On Fri, Feb 10, 2023 at 10:09 AM Jochen Fromm <jofr at cas-group.net> wrote:

> Interesting article. I have found this link to "GPT in 60 Lines of NumPy"
> on Hacker News
> https://jaykmody.com/blog/gpt-from-scratch/
>
> It is similar to nanoGPT and minGPT from Andrej Karpathy, who has joined
> OpenAI again recently
> https://github.com/karpathy/nanoGPT
>
> The secret is apparently not to use a complicated algorithm but just a
> huge amount of data, as Rich Sutton writes here
> http://www.incompleteideas.net/IncIdeas/BitterLesson.html
>
> -J.
>
>
> -------- Original message --------
> From: Russ Abbott <russ.abbott at gmail.com>
> Date: 2/10/23 5:53 PM (GMT+01:00)
> To: The Friday Morning Applied Complexity Coffee Group <friam at redfish.com>
>
> Subject: [FRIAM] A very good piece about ChatGPT.
>
> It's in the New Yorker
> <https://www.newyorker.com/tech/annals-of-technology/chatgpt-is-a-blurry-jpeg-of-the-web>
> and intended for a non-scientific audience.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://redfish.com/pipermail/friam_redfish.com/attachments/20230210/fc9c3581/attachment.html>