[FRIAM] ockham's razor losing its edge?

[Marcus Daniels]

Typically training includes a holdout dataset to watch how well training generalizes as it proceeds. 
With modern LLMs, there aren’t five points, there are trillions. 

From: Friam <friam-bounces at redfish.com> on behalf of Pieter Steenekamp <pieters at randcontrols.co.za>
Date: Thursday, January 30, 2025 at 7:55 PM
To: The Friday Morning Applied Complexity Coffee Group <friam at redfish.com>
Subject: Re: [FRIAM] ockham's razor losing its edge? 

In my experience, working with deep learning AI models can very easily lead to overtraining if you're low on data or computing resources. It's like trying to fit a fifth-order polynomial to five random data points - you'll get a perfect match for those points, but the model becomes useless for anything else. The thing is, with traditional methods, nobody would be foolish enough to use such a complex curve for so few points, but in deep learning, it's not always obvious when you're overtraining.

The landscape has changed because we now have access to vast amounts of data and powerful computing resources. This allows us to train models with many parameters without them falling into the trap of overfitting. Essentially, the barriers of limited data and computation have been removed, enabling the creation of high-performing models even with complicated architectures.

Einstein's "make it as simple as possible, but not simpler" remains relevant. Even with all the advancements, there's still a balance to strike - we need complexity to capture the nuances of real-world data, but not so much that we lose the model's ability to generalize to new situations.

Note the message is 100% mine but I use AI to assist my writing. 


On Thu, 30 Jan 2025 at 19:24, Roger Critchlow <rec at elf.org <mailto:rec at elf.org>> wrote: 

This was in the Complexity Digest feed this morning, it looks like fun. 


https://www.pnas.org/doi/10.1073/pnas.2401230121 <https://www.pnas.org/doi/10.1073/pnas.2401230121> 



What makes a model good or bad or useful or risible? 



-- rec -- 


.- .-.. .-.. / ..-. --- --- - . .-. ... / .- .-. . / .-- .-. --- -. --. / ... --- -- . / .- .-. . / ..- ... . ..-. ..- .-..
FRIAM Applied Complexity Group listserv
Fridays 9a-12p Friday St. Johns Cafe / Thursdays 9a-12p Zoom https://bit.ly/virtualfriam <https://bit.ly/virtualfriam>
to (un)subscribe http://redfish.com/mailman/listinfo/friam_redfish.com <http://redfish.com/mailman/listinfo/friam_redfish.com>
FRIAM-COMIC http://friam-comic.blogspot.com/ <http://friam-comic.blogspot.com/>
archives: 5/2017 thru present https://redfish.com/pipermail/friam_redfish.com/ <https://redfish.com/pipermail/friam_redfish.com/>
1/2003 thru 6/2021 http://friam.383.s1.nabble.com/ <http://friam.383.s1.nabble.com/> 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://redfish.com/pipermail/friam_redfish.com/attachments/20250131/d33a2dba/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 8422 bytes
Desc: not available
URL: <http://redfish.com/pipermail/friam_redfish.com/attachments/20250131/d33a2dba/attachment.bin>