[FRIAM] Run with a single bit?

[Stephen Guerin]

Looking into Ofer Dekel's work, the journalist messed up this sentence:

compress neural networks, the synapses of Machine Learning, down from 32
bits to, sometimes, a single bit



The team is not compressing neural networks to one-bit. They are
compressing the weights used in a neural network from 32-bits to one-bit.

Weights (or sometimes called the biases) are proxies for synapses in
biological neural networks. The weights in a neural network are the
activation strengths (or inhibition if negative) on each incoming link to a
node which are multiplied by the outgoing signal strength of the uplink
neighbor. As a number, it can be expressed in 32-bits or as low as 1 bit.
Though at 1-bit weight would not allow for inhibition.The weights are what
get tuned during the machine learning. One can also explore the topology of
the neural network (which nodes are connected to which) during learning and
is the basis for the new craze around Deep Learning. This  technique has
been around since the 90s but has now realized its use with the
availability of data. Eg, here's a paper of mine
<http://www.academia.edu/download/5213114/objectgarden.pdf> from '99
implementing some of research from UT Austin at the time.

I think it would have been more clear for the journalist to write :

Ofer Dekel's team is researching methods to reduce the memory and
processing requirements of Neural Networks to run on smaller devices on the
edge of the network closer to the sensors. One method is to reduce the
number of bits necessary describe the weights between nodes in a neural
network down from 32-bits to as little as one-bit.

-S
_______________________________________________________________________
Stephen.Guerin at Simtable.com <stephen.guerin at simtable.com>
CEO, Simtable  http://www.simtable.com
1600 Lena St #D1, Santa Fe, NM 87505
office: (505)995-0206 <(505)%20995-0206> mobile: (505)577-5828
<(505)%20577-5828>
twitter: @simtable

On Sat, Jul 1, 2017 at 11:42 PM, Tom Johnson <tom at jtjohnson.com> wrote:

> Friam Friends:
>
> A recent article
> <http://mashable.com/2017/06/29/microsoft-puts-ai-on-a-raspberry-pi/#HKAb_h1pvaqc>
> passed along by George Duncan says:
>
> "Now, Varma's team in India and Microsoft researchers in Redmond,
> Washington, (the entire project is led by lead researcher Ofer Dekel) have
> figured out how to *compress neural networks, the synapses of Machine
> Learning, down from 32 bits to, sometimes, a single bit *and run them on
> a $10 Raspberry Pi
> <http://%20%28the%20entire%20project%20is%20led%20by%20ofer%20dekel%29/>,
> a low-powered, credit-card-sized computer with a handful of ports and no
> screen."
>
> How, or what, can you do with a "single bit."?
>
> TJ
>
> ============================================
> Tom Johnson
> Institute for Analytic Journalism   --     Santa Fe, NM USA
> 505.577.6482 <(505)%20577-6482>(c)
> 505.473.9646 <(505)%20473-9646>(h)
> Society of Professional Journalists <http://www.spj.org>
> *Check out It's The People's Data
> <https://www.facebook.com/pages/Its-The-Peoples-Data/1599854626919671>*
> http://www.jtjohnson.com                   tom at jtjohnson.com
> ============================================
>
> ============================================================
> FRIAM Applied Complexity Group listserv
> Meets Fridays 9a-11:30 at cafe at St. John's College
> to unsubscribe http://redfish.com/mailman/listinfo/friam_redfish.com
> FRIAM-COMIC http://friam-comic.blogspot.com/ by Dr. Strangelove
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://redfish.com/pipermail/friam_redfish.com/attachments/20170702/af30b8e5/attachment.html>