<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang=EN-US link=blue vlink=purple style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt'>Typically training includes a holdout dataset to watch how well training generalizes as it proceeds.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'>With modern LLMs, there aren’t five points, there are trillions. <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal style='margin-bottom:12.0pt'><b><span style='font-size:12.0pt;color:black'>From: </span></b><span style='font-size:12.0pt;color:black'>Friam <friam-bounces@redfish.com> on behalf of Pieter Steenekamp <pieters@randcontrols.co.za><br><b>Date: </b>Thursday, January 30, 2025 at 7:55 PM<br><b>To: </b>The Friday Morning Applied Complexity Coffee Group <friam@redfish.com><br><b>Subject: </b>Re: [FRIAM] ockham's razor losing its edge?<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:11.0pt'>In my experience, working with deep learning AI models can very easily lead to overtraining if you're low on data or computing resources. It's like trying to fit a fifth-order polynomial to five random data points - you'll get a perfect match for those points, but the model becomes useless for anything else. The thing is, with traditional methods, nobody would be foolish enough to use such a complex curve for so few points, but in deep learning, it's not always obvious when you're overtraining.<br><br>The landscape has changed because we now have access to vast amounts of data and powerful computing resources. This allows us to train models with many parameters without them falling into the trap of overfitting. Essentially, the barriers of limited data and computation have been removed, enabling the creation of high-performing models even with complicated architectures.<br><br>Einstein's "make it as simple as possible, but not simpler" remains relevant. Even with all the advancements, there's still a balance to strike - we need complexity to capture the nuances of real-world data, but not so much that we lose the model's ability to generalize to new situations.<br><br>Note the message is 100% mine but I use AI to assist my writing.<o:p></o:p></span></p></div><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><div><div><p class=MsoNormal><span style='font-size:11.0pt'>On Thu, 30 Jan 2025 at 19:24, Roger Critchlow <<a href="mailto:rec@elf.org">rec@elf.org</a>> wrote:<o:p></o:p></span></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><div><p class=MsoNormal><span style='font-size:11.0pt'>This was in the Complexity Digest feed this morning, it looks like fun.<o:p></o:p></span></p><div><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:11.0pt'><a href="https://www.pnas.org/doi/10.1073/pnas.2401230121">https://www.pnas.org/doi/10.1073/pnas.2401230121</a><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:11.0pt'>What makes a model good or bad or useful or risible?<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-size:11.0pt'>-- rec --<o:p></o:p></span></p></div></div><p class=MsoNormal><span style='font-size:11.0pt'>.- .-.. .-.. / ..-. --- --- - . .-. ... / .- .-. . / .-- .-. --- -. --. / ... --- -- . / .- .-. . / ..- ... . ..-. ..- .-..<br>FRIAM Applied Complexity Group listserv<br>Fridays 9a-12p Friday St. Johns Cafe / Thursdays 9a-12p Zoom <a href="https://bit.ly/virtualfriam">https://bit.ly/virtualfriam</a><br>to (un)subscribe <a href="http://redfish.com/mailman/listinfo/friam_redfish.com">http://redfish.com/mailman/listinfo/friam_redfish.com</a><br>FRIAM-COMIC <a href="http://friam-comic.blogspot.com/">http://friam-comic.blogspot.com/</a><br>archives: 5/2017 thru present <a href="https://redfish.com/pipermail/friam_redfish.com/">https://redfish.com/pipermail/friam_redfish.com/</a><br> 1/2003 thru 6/2021 <a href="http://friam.383.s1.nabble.com/">http://friam.383.s1.nabble.com/</a><o:p></o:p></span></p></blockquote></div></div></body></html>