<div dir="auto"><div><br></div><div><br></div><div data-smartmail="gmail_signature">---<br>Frank C. Wimberly<br>140 Calle Ojo Feliz, <br>Santa Fe, NM 87505<br><br>505 670-9918<br>Santa Fe, NM</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">---------- Forwarded message ---------<br>From: <strong class="gmail_sendername" dir="auto">Diane Stidle</strong> <span dir="auto"><<a href="mailto:stidle@andrew.cmu.edu">stidle@andrew.cmu.edu</a>></span><br>Date: Mon, Jul 29, 2024, 1:38 PM<br>Subject: Thesis Proposal - July 30, 2024 - Ananye Agarwal - Pretrain in sim, adapt in real: A framework for sensorimotor intelligence<br>To: <a href="mailto:ml-seminar@cs.cmu.edu">ml-seminar@cs.cmu.edu</a> <<a href="mailto:ML-SEMINAR@cs.cmu.edu">ML-SEMINAR@cs.cmu.edu</a>>, Pieter Abbeel <<a href="mailto:pabbeel@berkeley.edu">pabbeel@berkeley.edu</a>>, <<a href="mailto:malik@eecs.berkeley.edu">malik@eecs.berkeley.edu</a>><br></div><br><br><u></u>
<div>
<p><b><i>Thesis Proposal</i></b></p>
<p>Date: July 30, 2024<br>
Time: 2:00pm<br>
Place: NSH 1305*<br>
Speaker: Ananye Agarwal</p>
<p><b>Title: <span style="color:rgb(0,0,0);background-color:transparent;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Pretrain in sim, adapt in real: A framework for sensorimotor intelligence</span></b></p>
<p><span style="color:rgb(0,0,0);background-color:transparent;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Abstract</span><font size="2"><span style="font-family:Arial,sans-serif;color:rgb(0,0,0);background-color:transparent;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">
</span></font></p>
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font size="2"><span style="font-family:Arial,sans-serif;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Machines today can write poetry, compose music, and create art but are incapable of mundane physical tasks like household chores or assembly – they lack simple sensorimotor skills. Internet data alone, while useful for high-level planning, might not be enough for learning how to affect the physical state of the world. In this proposal, I will talk about a route to acquiring these skills using large-scale learning from both simulation and real world data. </span></font></p>
<font size="2"><br>
</font>
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font size="2"><span style="font-family:Arial,sans-serif;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Affecting world state involves changing the robot state (locomotion/navigation) or that of the environment (manipulation) in a desired way. First, I present a simple framework that allows learning adaptive, agile locomotion for a low-cost quadruped from large-scale data in simulation. Next, I apply this to dexterous manipulation for functional grasping. Capable general-purpose robots must not only manipulate and locomote, but do so together in a seamless fashion. I present a mobile manipulation system that navigates, manipulates objects, and chooses where to look, all learnt concurrently using a single, end-to-end neural network.</span></font></p>
<p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><font size="2"><span style="font-family:Arial,sans-serif;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">
</span><span style="font-family:Arial,sans-serif;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Finally, for proposed work, I present roadblocks to scaling sensorimotor learning and an approach to leverage real-world data.
</span></font></p>
<font size="2"><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">
</span></span></font><b><span>Thesis</span> Committee:</b><br>
<div>Deepak Pathak (chair)</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div role="heading">Ruslan Salakhutdinov</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>Pieter Abbeel (UC Berkeley) <br>
</div>
<div>Jitendra Malik (UC Berkeley) <br>
<br>
</div>
<p>Zoom link: <span><a href="https://www.google.com/url?q=https://cmu.zoom.us/j/95525240691?pwd%3DnaKUBhWtQ2RXqv6ZRkLyeuzpuTVAHO.1&sa=D&source=calendar&ust=1722700104671601&usg=AOvVaw2qo2z4jsqzXMnlvRURlPH_" target="_blank" rel="noreferrer">https://cmu.zoom.us/j/95525240691?pwd=naKUBhWtQ2RXqv6ZRkLyeuzpuTVAHO.1</a><br>
</span></p>
<p><span>*</span>To get to NSH 1305, please take the elevator in
front of the RI reception down to floor 1, and find the lecture
room after exiting on the left.</p>
<p><span></span></p>
<p></p>
<pre cols="72">--
Diane Stidle
PhD Program Manager
Machine Learning Department
Carnegie Mellon University
<a href="mailto:stidle@andrew.cmu.edu" target="_blank" rel="noreferrer">stidle@andrew.cmu.edu</a></pre>
</div>
</div>