<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Yes, it seems that the kind of reporting that would be natural for expert systems does not have a good reason to map to deep neural nets. To the extent that I refer to the right architectures as “expert systems”, I think of them as having a knowledge structure dictated by the architect. Reading out parts of it that are used short of the conclusions as an “explanation” integrates naturally into the architecture as it is.<div class=""><br class=""></div><div class="">For neural nets, the part that one engineers is a combinatorial parameter space that is both very large and very high-dimensional. The learning loop guide the dynamics into some region in that space, but in what sense “being in that region” is a representation of regularities in the inputs is for the architect to discover, but largely not to have designed.</div><div class=""><br class=""></div><div class="">Thus it seems that “reporting” entails the invention of a representation that is matched both to the particular state of the neural net, which it may not have a way to survey, and to the comprehension standards of people, to which it has no way to be coupled at all. That kind of invention seems like a meta-operation of the process on its own internal state, and a higher-order function.</div><div class=""><br class=""></div><div class="">I wonder if there is some crowd-sourcing protocol, by which the AI could propose representations of its internal state, and a distributed pool of users through feedback to it (kind of like Ken Stanley’s picbreeder) could represent criteria for human comprehension, by which the meta-operation would be implemented as just one more layer of reinforcement learning. Probably end up with 70 million Fox viewers, and 70 million more Elvis fans, standing in for human comprehension.</div><div class=""><br class=""></div><div class="">But I don’t work in this area, and will stop now.</div><div class=""><br class=""></div><div class="">Eric</div><div class=""><br class=""></div><div class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Nov 11, 2020, at 11:21 AM, Frank Wimberly <<a href="mailto:wimberly3@gmail.com" class="">wimberly3@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="auto" class="">I used to teach a course called "AI and expert systems" for management students at Carnegie Mellon. I remember that Mycin, a diagnosis system for infectious diseases and one of the most famous expert systems, had some explanation capability. According to Wikipedia, "<span style="color:rgb(32,33,34);font-family:-apple-system,blinkmacsystemfont,"segoe ui",roboto,lato,helvetica,arial,sans-serif;font-size:16px;background-color:rgb(255,255,255)" class="">At the end, it [Mycin] provided a list of possible culprit bacteria ranked from high to low based on the probability of each diagnosis, its </span><a href="https://en.m.wikipedia.org/wiki/Confidence_interval" style="margin:0px;padding:0px;border:0px;line-height:inherit;font-family:-apple-system,blinkmacsystemfont,"segoe ui",roboto,lato,helvetica,arial,sans-serif;font-size:16px;vertical-align:baseline;background:none rgb(255,255,255);color:rgb(107,75,161);text-decoration-line:none" class="">confidence</a><span style="color:rgb(32,33,34);font-family:-apple-system,blinkmacsystemfont,"segoe ui",roboto,lato,helvetica,arial,sans-serif;font-size:16px;background-color:rgb(255,255,255)" class=""> in each diagnosis' probability, the reasoning behind each diagnosis..."</span><div dir="auto" class=""><br class=""><div data-smartmail="gmail_signature" dir="auto" class="">But after publishing a few papers in the area I left that field. I assumed, mistakenly it seems, that later AI implementations did the same. Too bad.</div><div data-smartmail="gmail_signature" dir="auto" class=""><br class=""></div><div data-smartmail="gmail_signature" dir="auto" class="">By the way, I realize, that AI systems can't assess non-respondent bias in the way it was done in Project Talent but perhaps there is some "machine readable" way to do it.</div><div data-smartmail="gmail_signature" dir="auto" class=""><br class=""></div><div data-smartmail="gmail_signature" dir="auto" class="">---<br class="">Frank C. Wimberly<br class="">140 Calle Ojo Feliz, <br class="">Santa Fe, NM 87505<br class=""><br class="">505 670-9918<br class="">Santa Fe, NM</div></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Nov 11, 2020, 9:05 AM David Eric Smith <<a href="mailto:desmith@santafe.edu" class="">desmith@santafe.edu</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word;line-break:after-white-space" class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Nov 11, 2020, at 8:54 AM, Frank Wimberly <<a href="mailto:wimberly3@gmail.com" target="_blank" rel="noreferrer" class="">wimberly3@gmail.com</a>> wrote:</div><div class=""><div dir="auto" class=""><div dir="auto" class=""><br class=""></div><div dir="auto" class="">Don't some AI systems include subsystems for explaining their reasoning?</div></div></div></blockquote><div class=""><br class=""></div><div class="">I don’t know how common that is, Frank. The few people I know who are active and skilled deep-learning practitioners have told me (if I have understood) that it is rare and limited. I spent some time looking at the zero-shot language translation, as something I wanted to convene at SFI, with the AI goal of unpacking what was the “universal language” internal representation, and with the linguistics goal of using it in cognate classification and historical reconstruction. Never could get a call-back from any of the google people. But I didn’t think at the time that zero-shot had been unpacked.</div><div class=""><br class=""></div><div class="">Probably some on this list know much more about the state of play.</div><div class=""><br class=""></div><div class="">Eric</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="auto" class=""><div dir="auto" class=""><br class=""></div><div dir="auto" class="">Happy Veterans Day,</div><div dir="auto" class=""><br class=""></div><div dir="auto" class="">Frank<br class=""><div dir="auto" class=""><br class=""><div data-smartmail="gmail_signature" dir="auto" class="">---<br class="">Frank C. Wimberly<br class="">140 Calle Ojo Feliz, <br class="">Santa Fe, NM 87505<br class=""><br class="">505 670-9918<br class="">Santa Fe, NM</div></div></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Nov 11, 2020, 3:25 AM David Eric Smith <<a href="mailto:desmith@santafe.edu" target="_blank" rel="noreferrer" class="">desmith@santafe.edu</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word;line-break:after-white-space" class="">Friam poll:<div class=""><br class=""></div><div class="">How soon until classical telephone polling is just gone?</div><div class=""><a href="https://www.nytimes.com/2020/11/10/upshot/polls-what-went-wrong.html" rel="noreferrer noreferrer" target="_blank" class="">https://www.nytimes.com/2020/11/10/upshot/polls-what-went-wrong.html</a></div><div class=""><br class=""></div><div class="">If, as boasted, facebook knows when their users are pregnant before the users know, they know who someone supports and whether that person is likely to vote. </div><div class=""><br class=""></div><div class="">At this stage, trying to get accurate statistics from cold calls on the phone seems as quixotic as trying to infer something from the people who read books. But if there’s anything we can count on, it is that the number of people who don’t leave an internet fingerprint is too small to have any political impact at all.</div><div class=""><br class=""></div><div class="">How much effort they put into getting reliable calibrations will depend on what ways they see to monetize it, but the diversity of cash-outs should be nearly inexhaustible, for years to come.</div><div class=""><br class=""></div><div class="">So one more thing goes into what is both a black box and a private rather than public box. It will take over after the first few times it produces much more reliable results, but since we won’t know what it is based on — AIs don’t explain themselves — we will have no ability to extrapolate out of sample.</div><div class=""><br class=""></div><div class="">Eric</div><div class=""><br class=""></div><div class=""><br class=""></div></div>- .... . -..-. . -. -.. -..-. .. ... -..-. .... . .-. .<br class="">
FRIAM Applied Complexity Group listserv<br class="">
Zoom Fridays 9:30a-12p Mtn GMT-6 <a href="https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fbit.ly%2fvirtualfriam&c=E,1,MIlehtrxasGxzXCBAG-Wuptie4wVpHZCod2H9fKLPnfs1yGvGgHBDARSwVq4VdCdtzq4Dow-kWiVv4jc9hIVdl8zyYLkXNCJ81WlPwU_cCV-Wj_FaKxE&typo=1" rel="noreferrer noreferrer noreferrer" target="_blank" class="">bit.ly/virtualfriam</a><br class="">
un/subscribe <a href="https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fredfish.com%2fmailman%2flistinfo%2ffriam_redfish.com&c=E,1,eEyTIKAnKXKW32Kg5QjvWagEgRoLZi2PtyoqSEHHUN3kS4q1_QFqepVMevdBzSM8Z9F4aZohLrRrIwqCA-ExUAfBhUBVHO7gf4xaB87f6I6MuI4i&typo=1" rel="noreferrer noreferrer noreferrer" target="_blank" class="">http://redfish.com/mailman/listinfo/friam_redfish.com</a><br class="">
archives: <a href="http://friam.471366.n2.nabble.com/" rel="noreferrer noreferrer noreferrer" target="_blank" class="">http://friam.471366.n2.nabble.com/</a><br class="">
FRIAM-COMIC <a href="https://linkprotect.cudasvc.com/url?a=http%3a%2f%2ffriam-comic.blogspot.com%2f&c=E,1,owZjlS1njud5BssZPtYtW4jYIrQySTobJWmn1LQwCyeJUf-KLCb4RIvTldGJmHVC8OL-2OunE-nqJmBcsyFM7sBJTgDRQlvwtgQpNHs3-mBAwQlugck,&typo=1" rel="noreferrer noreferrer noreferrer" target="_blank" class="">http://friam-comic.blogspot.com/</a> <br class="">
</blockquote></div>
- .... . -..-. . -. -.. -..-. .. ... -..-. .... . .-. .<br class="">FRIAM Applied Complexity Group listserv<br class="">Zoom Fridays 9:30a-12p Mtn GMT-6 <a href="https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fbit.ly%2fvirtualfriam&c=E,1,oqHFT1YdV0k75pKzx4YTF1iw_fTQ3iaPASzr-UyzEBQ9snxyN9TmgQOObG89c4hRDu0e22Rlo4Ua40Cy7mZIzFf1SCuw5JNkQYjfxrb5tGKG&typo=1" target="_blank" rel="noreferrer" class="">bit.ly/virtualfriam</a><br class="">un/subscribe <a href="https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fredfish.com%2fmailman%2flistinfo%2ffriam_redfish.com&c=E,1,ZhFECaCEstC6R1KARgPxbu9IEAqI9wjTi65WxBGjDM4gwbxMCgeSOBQ2EKWK35vV6tuhdQlPpQ3q0AmGgUXMwPCscl0M5mPqvGm002E2VyScyzizkyplPu9NMaOH&typo=1" target="_blank" rel="noreferrer" class="">https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fredfish.com%2fmailman%2flistinfo%2ffriam_redfish.com&c=E,1,ZhFECaCEstC6R1KARgPxbu9IEAqI9wjTi65WxBGjDM4gwbxMCgeSOBQ2EKWK35vV6tuhdQlPpQ3q0AmGgUXMwPCscl0M5mPqvGm002E2VyScyzizkyplPu9NMaOH&typo=1</a><br class="">archives: <a href="http://friam.471366.n2.nabble.com/" target="_blank" rel="noreferrer" class="">http://friam.471366.n2.nabble.com/</a><br class="">FRIAM-COMIC <a href="https://linkprotect.cudasvc.com/url?a=http%3a%2f%2ffriam-comic.blogspot.com%2f&c=E,1,bcNwVv6Xg-iYdul8iBL7SSR37PBueIDjkSji3dd3y4G3o_reGRlxBEs9K8ZfZPNKeOYrtHreLpQMofHuId25yG_iN3xq5W7MgsWxOvjO&typo=1" target="_blank" rel="noreferrer" class="">https://linkprotect.cudasvc.com/url?a=http%3a%2f%2ffriam-comic.blogspot.com%2f&c=E,1,bcNwVv6Xg-iYdul8iBL7SSR37PBueIDjkSji3dd3y4G3o_reGRlxBEs9K8ZfZPNKeOYrtHreLpQMofHuId25yG_iN3xq5W7MgsWxOvjO&typo=1</a> <br class=""></div></blockquote></div><br class=""></div>- .... . -..-. . -. -.. -..-. .. ... -..-. .... . .-. .<br class="">
FRIAM Applied Complexity Group listserv<br class="">
Zoom Fridays 9:30a-12p Mtn GMT-6 <a href="https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fbit.ly%2fvirtualfriam&c=E,1,NoYGq6JtoxCm1xKczIKhU3YLSYqufMlPmIaDtf39TAcVMqkJCuhfJkMbfVlEpdYRFc1cD0EgP-sLVuI3zPxvBvDWh61DF3U9lI4YNqReg0i97EFnaQ,,&typo=1" rel="noreferrer noreferrer" target="_blank" class="">bit.ly/virtualfriam</a><br class="">
un/subscribe <a href="https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fredfish.com%2fmailman%2flistinfo%2ffriam_redfish.com&c=E,1,iVIpbUAf1oxjjTBO9As2ReYhdIQm9v3GqeiCU6m3Z2Fo2VpBrvSH9ywZYDaQ2GuKEufkvEbsyIw6AyNeSMWEeOwSlhGnMsLm_KBlxorQsNlck8JoYLOE&typo=1" rel="noreferrer noreferrer" target="_blank" class="">http://redfish.com/mailman/listinfo/friam_redfish.com</a><br class="">
archives: <a href="http://friam.471366.n2.nabble.com/" rel="noreferrer noreferrer" target="_blank" class="">http://friam.471366.n2.nabble.com/</a><br class="">
FRIAM-COMIC <a href="https://linkprotect.cudasvc.com/url?a=http%3a%2f%2ffriam-comic.blogspot.com%2f&c=E,1,ZgiI8AhBAKEz7-zoY4fne03DZafBG3HaN3bcefzRb345LdKATHx-u7cbScxyxflCPclhZKFYRgY85o-dUS2uc6Ltqj_LW1P72voPVrzW2g,,&typo=1" rel="noreferrer noreferrer" target="_blank" class="">http://friam-comic.blogspot.com/</a> <br class="">
</blockquote></div>
- .... . -..-. . -. -.. -..-. .. ... -..-. .... . .-. .<br class="">FRIAM Applied Complexity Group listserv<br class="">Zoom Fridays 9:30a-12p Mtn GMT-6 <a href="http://bit.ly/virtualfriam" class="">bit.ly/virtualfriam</a><br class="">un/subscribe <a href="https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fredfish.com%2fmailman%2flistinfo%2ffriam_redfish.com&c=E,1,bWeUaJgfO8pAEvOoQ13EihOr5iHDAYpKEDbuVrrfH0tlm3YhEW0d3eC5694-3x7idkZniXSayxxaTRdW16u4_tX9cEEJFQRWctiHKSTz&typo=1" class="">https://linkprotect.cudasvc.com/url?a=http%3a%2f%2fredfish.com%2fmailman%2flistinfo%2ffriam_redfish.com&c=E,1,bWeUaJgfO8pAEvOoQ13EihOr5iHDAYpKEDbuVrrfH0tlm3YhEW0d3eC5694-3x7idkZniXSayxxaTRdW16u4_tX9cEEJFQRWctiHKSTz&typo=1</a><br class="">archives: <a href="http://friam.471366.n2.nabble.com/" class="">http://friam.471366.n2.nabble.com/</a><br class="">FRIAM-COMIC <a href="https://linkprotect.cudasvc.com/url?a=http%3a%2f%2ffriam-comic.blogspot.com%2f&c=E,1,OMiJ_4epUCa9FpA_LGrMSEzzBK2n-IaWeW3BeK9rkdJvD3QTGdmiPTj0T-oTTgv_r0ePB9dqJJa-mkFBHiMJtyjSGEcyQezaGiQkJddPwGD7GejfRYo,&typo=1" class="">https://linkprotect.cudasvc.com/url?a=http%3a%2f%2ffriam-comic.blogspot.com%2f&c=E,1,OMiJ_4epUCa9FpA_LGrMSEzzBK2n-IaWeW3BeK9rkdJvD3QTGdmiPTj0T-oTTgv_r0ePB9dqJJa-mkFBHiMJtyjSGEcyQezaGiQkJddPwGD7GejfRYo,&typo=1</a> <br class=""></div></blockquote></div><br class=""></div></body></html>