[FRIAM] ockham's razor losing its edge?

[Pieter Steenekamp]

In my experience, working with deep learning AI models can very easily lead
to overtraining if you're low on data or computing resources. It's like
trying to fit a fifth-order polynomial to five random data points - you'll
get a perfect match for those points, but the model becomes useless for
anything else. The thing is, with traditional methods, nobody would be
foolish enough to use such a complex curve for so few points, but in deep
learning, it's not always obvious when you're overtraining.

The landscape has changed because we now have access to vast amounts of
data and powerful computing resources. This allows us to train models with
many parameters without them falling into the trap of overfitting.
Essentially, the barriers of limited data and computation have been
removed, enabling the creation of high-performing models even with
complicated architectures.

Einstein's "make it as simple as possible, but not simpler" remains
relevant. Even with all the advancements, there's still a balance to strike
- we need complexity to capture the nuances of real-world data, but not so
much that we lose the model's ability to generalize to new situations.

Note the message is 100% mine but I use AI to assist my writing.

On Thu, 30 Jan 2025 at 19:24, Roger Critchlow <rec at elf.org> wrote:

> This was in the Complexity Digest feed this morning, it looks like fun.
>
> https://www.pnas.org/doi/10.1073/pnas.2401230121
>
> What makes a model good or bad or useful or risible?
>
> -- rec --
> .- .-.. .-.. / ..-. --- --- - . .-. ... / .- .-. . / .-- .-. --- -. --. /
> ... --- -- . / .- .-. . / ..- ... . ..-. ..- .-..
> FRIAM Applied Complexity Group listserv
> Fridays 9a-12p Friday St. Johns Cafe   /   Thursdays 9a-12p Zoom
> https://bit.ly/virtualfriam
> to (un)subscribe http://redfish.com/mailman/listinfo/friam_redfish.com
> FRIAM-COMIC http://friam-comic.blogspot.com/
> archives:  5/2017 thru present
> https://redfish.com/pipermail/friam_redfish.com/
>   1/2003 thru 6/2021  http://friam.383.s1.nabble.com/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://redfish.com/pipermail/friam_redfish.com/attachments/20250131/3b9de5bd/attachment.html>