<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>I was hoping this article would have more meat to it, but the
main point seems highly relevant to a practical/introductory
workshop such as the one you are developing:<br>
</p>
<p><a moz-do-not-send="true"
href="https://dl.acm.org/doi/10.1145/3544903.3544905">The Need
for Interpretable Features: Taxonomy and Motivation <br>
</a></p>
<p><a moz-do-not-send="true"
href="https://scitechdaily.com/mit-taxonomy-helps-build-explainability-into-the-components-of-machine-learning-models/"
class="moz-txt-link-freetext">https://scitechdaily.com/mit-taxonomy-helps-build-explainability-into-the-components-of-machine-learning-models/</a></p>
<p>My limited experience with ML is that the convo/invo-lutions that
developers go through to make their learning models work well tend
to obscure the interpretability of the results. Some of this
seems unavoidable (inevitable?), particularly the highly technical
reserved terminology that often exists in a given domain, but this
team purports to help provide guidelines for minimizing the
consequences...</p>
<p>My main experience with this domain involved an attempt to find
distance measures between the flake patterns in 3D scanned
archaeological artifacts, starting with lithics but also aspiring
to work with pottery and textiles... essentially trying to
recognize similarities in the "hand" of the artisans involved.<br>
</p>
<p><br>
</p>
<div class="moz-cite-prefix">On 1/9/23 1:29 AM, Pieter Steenekamp
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAPerSO+-K2nrQUD2SLbHMN8m74N395QdJAFtu82DaJBQ57OXfQ@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr"><a class="gmail_plusreply" id="plusReplyChip-0"
moz-do-not-send="true">@Russ, </a>
<div><a class="gmail_plusreply" moz-do-not-send="true"><br>
</a>a) I appreciate the suggestion to include a simple neural
network that can make predictions based on inputs and be
trained using steepest descent to optimize its weights. This
would be a valuable addition to my training material, as it
provides a foundational understanding of how neural networks
work. <br>
<br>
b) My focus is on providing practical training for
professionals with deep domain knowledge but limited software
experience, who are looking to utilize deep learning as a tool
in their respective fields. In contrast, it seems that your
focus is on understanding the inner workings of deep learning.
Both approaches have their own merits, and it is important to
cater to the needs and goals of different learners.<br>
<br>
Pieter</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sun, 8 Jan 2023 at 21:20,
Marcus Daniels <<a href="mailto:marcus@snoutfarm.com"
moz-do-not-send="true" class="moz-txt-link-freetext">marcus@snoutfarm.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="auto">
The main defects of both R and Python are a lack of a typing
system and high performance compilation. I find R still
follows (is used by) the statistics research community more
than Python. Common Lisp was always better than either.<br>
<br>
<div dir="ltr">Sent from my iPhone</div>
<div dir="ltr"><br>
<blockquote type="cite">On Jan 8, 2023, at 11:03 AM, Russ
Abbott <<a href="mailto:russ.abbott@gmail.com"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">russ.abbott@gmail.com</a>>
wrote:<br>
<br>
</blockquote>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">As
indicated in my original reply, my interest in this
project grows from my relative ignorance of Deep
Learning. My career has focussed exclusively on
symbolic computing. I've worked with and taught (a)
functional programming, logic programming, and
related issues in advanced Python; (b) complex
systems, agent-based modeling, genetic algorithms,
and related evolutionary processes, (c) a bit of
constraint programming, especially in MiniZinc, and
(d) reinforcement learning as Q-learning, which is
reinforcement learning without neural nets. I've
always avoided neural nets--and more generally
numerical programming of any sort. </div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)"><br>
</div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">Deep
learning has produced so many impressive results
that I've decided to devote much of my retirement
life to learning about it. I retired at the end of
Spring 2022 and (after a break) am now devoting much
of my time to learning more about Deep Neural Nets.
So far, I've dipped my brain into it at various
points. I think I've learned a fair amount. For
example, </div>
<div class="gmail_default"
style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,0,0)">
<ul>
<li>I now know how to build a neural net (NN) that
adds two numbers using a single layer with a
single neuron. It's really quite simple and is,
I think, a beautiful example of how NNs work. If
I were to teach an intro to NNs I'd start with
this.</li>
<li>I've gone through the Kaggle Deep Learning
sequence mentioned earlier. </li>
<li>I found a paper that shows how you can
approximate any differentiable function to any
degree of accuracy with a single-layer NN. (This
is a very nice result, although I believe it's
not used explicitly in building serious Deep NN
systems.)</li>
<li>From what I've seen so far, most serious DNNs
are built using Keras rather than PyTorch.</li>
<li>I've looked at Jeremy Howard's <a
href="http://fast.ai" target="_blank"
moz-do-not-send="true">fast.ai</a> material. I
was going to go through the course but stopped
when I found that it uses PyTorch. Also, it
seems to be built on
<a href="http://fast.ai" target="_blank"
moz-do-not-send="true">fast.ai</a> libraries
that do a lot of the work for you without
explanation. And it seems to focus almost
exclusively on Convolutional NNs. </li>
<li>My impression of DNNs is that to a great
extent they are <i>ad hoc</i>. There is no good
way to determine the best architecture to use
for a given problem. By architecture, I mean the
number of layers, the number of neurons in each
layer, the types of layers, the activation
functions to use, etc. </li>
<li>All DNNs that I've seen use Python as code
glue rather than R or some other language. I
like Python--so I'm pleased with that.</li>
<li>To build serious NNs one should learn the
Python libraries Numpy (array manipulation) and
Pandas (data processing). Numpy especially seems
to be used for virtually all DNNs that I've
seen. </li>
<li>Keras and probably PyTorch include a number of
special-purpose neurons and layers that can be
included in one's DNN. These include: a DropOut
layer, LSTM (short-long-term memory) neurons,
convolutional layers, recurrent neural net
layers (RNN), and more recently transformers,
which get credit for ChatGPT and related
programs. My impression is that these
special-purpose layers are
<i>ad hoc</i> in the same sense that functions
or libraries that one finds useful in a
programming language are <i>ad hoc</i>. They
have been very important for the success of
DNNs, but they came into existence because
people invented them in the same way that people
invented useful functions and libraries. </li>
<li>NN libraries also include a menagerie of
activation functions. An activation function
acts as the final control on the output of a
layer. Different activation functions are used
for different purposes. To be successful in
building a DNN, one must understand what those
activation functions do for you and which ones
to use. </li>
<li>I'm especially interested in DNNs that use
reinforcement learning. That's because the first
DNN work that impressed me was DeepMind's DNNs
that learned to play Atari games--and then Go,
etc. An important advantage of Reinforcement
Learning (RL) is that it doesn't depend on
mountains of labeled data. </li>
<li>I find RL systems more interesting than image
recognition systems. One of the striking
features of many image recognition systems is
that they can be thrown off by changing a small
number of pixels in an image. The changed image
would look to a human observer just like the
original, but it might fool a trained NN into
labeling the image as a banana rather than, say,
an automobile, which is what it really is. To
address this problem people have developed
Generative Adversarial Networks (GANs) which
attempt to find such weaknesses in a neural net
during training and then to train the NN not to
have those weaknesses. This is a fascinating
result, but as far as I can tell, it mainly
shows how fragile some NNs are and doesn't add
much conceptual depth to one's understanding of
how NNs work. </li>
</ul>
<div>I'm impressed with this list of things I sort
of know. If you had asked me before I started
writing this email I wouldn't have thought I had
learned as much as I have. Even so, I feel like I
don't understand much of it beyond a superficial
level. </div>
<div><br>
</div>
<div class="gmail_default">
<div>So far I've done all my exploration using
Google's Colab (Google's Python notebook
implementation) and Kaggle's similar Python
notebook implementation. (I prefer Colab to
Kaggle.) Using either one, it's super nice not
to have to download and install anything!</div>
<div><br>
</div>
</div>
<div
style="color:rgb(34,34,34);font-family:Arial,Helvetica,sans-serif">
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>I'm continuing my journey to learn more about
DNNs. I'd be happy to have company and to help
develop materials to teach about DNNs. (Developing
teaching materials always helps me learn the
subject being covered.)</div>
<div><br>
</div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr"><span
style="color:rgb(33,33,33);font-family:"Helvetica
Neue",Helvetica,Arial,sans-serif;font-size:16.5px;line-height:24.75px"></span>--
Russ Abbott
<br>
Professor
Emeritus,
Computer
Science<br>
California
State
University,
Los Angeles<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sun, Jan 8, 2023
at 1:48 AM glen <<a
href="mailto:gepropella@gmail.com" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">gepropella@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px
0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
Yes, the money/expertise bar is still pretty high.
But TANSTAAFL still applies. And the overwhelming
evidence is coming in that specific models do better
than those trained up on diverse data sets, "better"
meaning less prone to subtle bullsh¡t. What I find
fascinating is tools like OpenAI *facilitate*
trespassing. We have a wonderful bloom of
non-experts claiming they understand things like
"deep learning". But do they? An old internet meme
is brought to mind: "Do you even Linear Algebra,
bro?" >8^D<br>
<br>
On 1/8/23 01:06, Jochen Fromm wrote:<br>
> I have finished a number of Coursera courses
recently, including "Deep Learning & Neural
Networks with Keras" which was ok but not great. The
problems with deep learning are<br>
> <br>
> * to achieve impressive results like chatGPT
from OpenAi or LaMDA from Goggle you need to spend
millions on hardware<br>
> * only big organisations can afford to create
such expensive models<br>
> * the resulting network is s black box and it
is unclear why it works the way it does<br>
> <br>
> In the end it is just the same old back
propagation that has been known for decades, just on
more computers and trained on more data. Peter
Norvig calls it "The unreasonable effectiveness of
data"<br>
> <a
href="https://research.google.com/pubs/archive/35179.pdf"
rel="noreferrer" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">
https://research.google.com/pubs/archive/35179.pdf</a><br>
> <br>
> -J.<br>
> <br>
> <br>
> -------- Original message --------<br>
> From: Russ Abbott <<a
href="mailto:russ.abbott@gmail.com"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">russ.abbott@gmail.com</a>><br>
> Date: 1/8/23 12:20 AM (GMT+01:00)<br>
> To: The Friday Morning Applied Complexity
Coffee Group <<a href="mailto:friam@redfish.com"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">friam@redfish.com</a>><br>
> Subject: Re: [FRIAM] Deep learning training
material<br>
> <br>
> Hi Pieter,<br>
> <br>
> A few comments.<br>
> <br>
> * Much of the actual deep learning material
looks like it came from the Kaggle "Deep Learning
<<a
href="https://www.kaggle.com/learn/intro-to-deep-learning"
rel="noreferrer" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">https://www.kaggle.com/learn/intro-to-deep-learning</a>>"
sequence.<br>
> * In my opinion, R is an ugly and /ad hoc/
language. I'd stick to Python.<br>
> * More importantly, I would put the
How-to-use-Python stuff into a preliminary class.
Assume your audience knows how to use Python and
focus on Deep Learning. Given that, there is only a
minimal amount of information about Deep Learning in
the write-up. If I were to attend the workshop and
thought I would be learning about Deep Learning, I
would be disappointed--at least with what's covered
in the write-up.<br>
> <br>
> I say this because I've been looking for a
good intro to Deep Learning. Even though I taught
Computer Science for many years, and am now retired,
I avoided Deep Learning because it was so
non-symbolic. My focus has always been on symbolic
computing. But Deep Learning has produced so many
extraordinarily impressive results, I decided I
should learn more about it. I haven't found any
really good material. If you are interested, I'd be
more than happy to work with you on developing some
introductory Deep Learning material. <br>
> <br>
> -- Russ Abbott<br>
> Professor Emeritus, Computer Science<br>
> California State University, Los Angeles<br>
> <br>
> <br>
> On Thu, Jan 5, 2023 at 11:31 AM Pieter
Steenekamp <<a
href="mailto:pieters@randcontrols.co.za"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">pieters@randcontrols.co.za</a>
<mailto:<a
href="mailto:pieters@randcontrols.co.za"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">pieters@randcontrols.co.za</a>>>
wrote:<br>
> <br>
> Thanks to the kind support of OpenAI's
chatGPT, I am in the process of gathering materials
for a comprehensive and hands-on deep learning
workshop. Although it is still a work in progress, I
welcome any interested parties to take a look and
provide their valuable input. Thank you!<br>
> <br>
> You can get it from:<br>
> <a
href="https://www.dropbox.com/s/eyx4iumb0439wlx/deep%20learning%20training%20rev%2005012023.zip?dl=0"
rel="noreferrer" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">https://www.dropbox.com/s/eyx4iumb0439wlx/deep%20learning%20training%20rev%2005012023.zip?dl=0</a>
<<a
href="https://www.dropbox.com/s/eyx4iumb0439wlx/deep%20learning%20training%20rev%2005012023.zip?dl=0"
rel="noreferrer" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">https://www.dropbox.com/s/eyx4iumb0439wlx/deep%20learning%20training%20rev%2005012023.zip?dl=0</a>><br>
> <br>
<br>
-- <br>
ꙮ Mɥǝu ǝlǝdɥɐuʇs ɟᴉƃɥʇ' ʇɥǝ ƃɹɐss snɟɟǝɹs˙ ꙮ<br>
<br>
-. --- - / ...- .- .-.. .. -.. / -- --- .-. ... . /
-.-. --- -.. .<br>
FRIAM Applied Complexity Group listserv<br>
Fridays 9a-12p Friday St. Johns Cafe / Thursdays
9a-12p Zoom <a href="https://bit.ly/virtualfriam"
rel="noreferrer" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">
https://bit.ly/virtualfriam</a><br>
to (un)subscribe <a
href="http://redfish.com/mailman/listinfo/friam_redfish.com"
rel="noreferrer" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">
http://redfish.com/mailman/listinfo/friam_redfish.com</a><br>
FRIAM-COMIC <a
href="http://friam-comic.blogspot.com/"
rel="noreferrer" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">
http://friam-comic.blogspot.com/</a><br>
archives: 5/2017 thru present <a
href="https://redfish.com/pipermail/friam_redfish.com/"
rel="noreferrer" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">
https://redfish.com/pipermail/friam_redfish.com/</a><br>
1/2003 thru 6/2021 <a
href="http://friam.383.s1.nabble.com/"
rel="noreferrer" target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">
http://friam.383.s1.nabble.com/</a><br>
</blockquote>
</div>
<span>-. --- - / ...- .- .-.. .. -.. / -- --- .-. ... .
/ -.-. --- -.. .</span><br>
<span>FRIAM Applied Complexity Group listserv</span><br>
<span>Fridays 9a-12p Friday St. Johns Cafe /
Thursdays 9a-12p Zoom <a
href="https://bit.ly/virtualfriam" target="_blank"
moz-do-not-send="true" class="moz-txt-link-freetext">https://bit.ly/virtualfriam</a></span><br>
<span>to (un)subscribe <a
href="http://redfish.com/mailman/listinfo/friam_redfish.com"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">http://redfish.com/mailman/listinfo/friam_redfish.com</a></span><br>
<span>FRIAM-COMIC <a
href="http://friam-comic.blogspot.com/"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">http://friam-comic.blogspot.com/</a></span><br>
<span>archives: 5/2017 thru present <a
href="https://redfish.com/pipermail/friam_redfish.com/"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://redfish.com/pipermail/friam_redfish.com/</a></span><br>
<span> 1/2003 thru 6/2021 <a
href="http://friam.383.s1.nabble.com/"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">http://friam.383.s1.nabble.com/</a></span><br>
</div>
</blockquote>
</div>
-. --- - / ...- .- .-.. .. -.. / -- --- .-. ... . / -.-. ---
-.. .<br>
FRIAM Applied Complexity Group listserv<br>
Fridays 9a-12p Friday St. Johns Cafe / Thursdays 9a-12p
Zoom <a href="https://bit.ly/virtualfriam" rel="noreferrer"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://bit.ly/virtualfriam</a><br>
to (un)subscribe <a
href="http://redfish.com/mailman/listinfo/friam_redfish.com"
rel="noreferrer" target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">http://redfish.com/mailman/listinfo/friam_redfish.com</a><br>
FRIAM-COMIC <a href="http://friam-comic.blogspot.com/"
rel="noreferrer" target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">http://friam-comic.blogspot.com/</a><br>
archives: 5/2017 thru present <a
href="https://redfish.com/pipermail/friam_redfish.com/"
rel="noreferrer" target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">https://redfish.com/pipermail/friam_redfish.com/</a><br>
1/2003 thru 6/2021 <a
href="http://friam.383.s1.nabble.com/" rel="noreferrer"
target="_blank" moz-do-not-send="true"
class="moz-txt-link-freetext">http://friam.383.s1.nabble.com/</a><br>
</blockquote>
</div>
<br>
<fieldset class="moz-mime-attachment-header"></fieldset>
<pre class="moz-quote-pre" wrap="">-. --- - / ...- .- .-.. .. -.. / -- --- .-. ... . / -.-. --- -.. .
FRIAM Applied Complexity Group listserv
Fridays 9a-12p Friday St. Johns Cafe / Thursdays 9a-12p Zoom <a class="moz-txt-link-freetext" href="https://bit.ly/virtualfriam">https://bit.ly/virtualfriam</a>
to (un)subscribe <a class="moz-txt-link-freetext" href="http://redfish.com/mailman/listinfo/friam_redfish.com">http://redfish.com/mailman/listinfo/friam_redfish.com</a>
FRIAM-COMIC <a class="moz-txt-link-freetext" href="http://friam-comic.blogspot.com/">http://friam-comic.blogspot.com/</a>
archives: 5/2017 thru present <a class="moz-txt-link-freetext" href="https://redfish.com/pipermail/friam_redfish.com/">https://redfish.com/pipermail/friam_redfish.com/</a>
1/2003 thru 6/2021 <a class="moz-txt-link-freetext" href="http://friam.383.s1.nabble.com/">http://friam.383.s1.nabble.com/</a>
</pre>
</blockquote>
</body>
</html>