[FRIAM] Fwd: Thesis Proposal - July 30, 2024 - Ananye Agarwal - Pretrain in sim, adapt in real: A framework for sensorimotor intelligence

[Frank Wimberly]

---
Frank C. Wimberly
140 Calle Ojo Feliz,
Santa Fe, NM 87505

505 670-9918
Santa Fe, NM

---------- Forwarded message ---------
From: Diane Stidle <stidle at andrew.cmu.edu>
Date: Mon, Jul 29, 2024, 1:38 PM
Subject: Thesis Proposal - July 30, 2024 - Ananye Agarwal - Pretrain in
sim, adapt in real: A framework for sensorimotor intelligence
To: ml-seminar at cs.cmu.edu <ML-SEMINAR at cs.cmu.edu>, Pieter Abbeel <
pabbeel at berkeley.edu>, <malik at eecs.berkeley.edu>


*Thesis Proposal*

Date: July 30, 2024
Time: 2:00pm
Place: NSH 1305*
Speaker: Ananye Agarwal

*Title: Pretrain in sim, adapt in real: A framework for sensorimotor
intelligence*

Abstract

Machines today can write poetry, compose music, and create art but are
incapable of mundane physical tasks like household chores or assembly –
they lack simple sensorimotor skills. Internet data alone, while useful for
high-level planning, might not be enough for learning how to affect the
physical state of the world. In this proposal, I will talk about a route to
acquiring these skills using large-scale learning from both simulation and
real world data.

Affecting world state involves changing the robot state
(locomotion/navigation) or that of the environment (manipulation) in a
desired way. First, I present a simple framework that allows learning
adaptive, agile locomotion for a low-cost quadruped from large-scale data
in simulation. Next, I apply this to dexterous manipulation for functional
grasping. Capable general-purpose robots must not only manipulate and
locomote, but do so together in a seamless fashion. I present a mobile
manipulation system that navigates, manipulates objects, and chooses where
to look, all learnt concurrently using a single, end-to-end neural network.

Finally, for proposed work, I present roadblocks to scaling sensorimotor
learning and an approach to leverage real-world data.
*Thesis Committee:*
Deepak Pathak (chair)
Ruslan Salakhutdinov
Pieter Abbeel (UC Berkeley)
Jitendra Malik (UC Berkeley)

Zoom link:
https://cmu.zoom.us/j/95525240691?pwd=naKUBhWtQ2RXqv6ZRkLyeuzpuTVAHO.1
<https://www.google.com/url?q=https://cmu.zoom.us/j/95525240691?pwd%3DnaKUBhWtQ2RXqv6ZRkLyeuzpuTVAHO.1&sa=D&source=calendar&ust=1722700104671601&usg=AOvVaw2qo2z4jsqzXMnlvRURlPH_>

*To get to NSH 1305, please take the elevator in front of the RI reception
down to floor 1, and find the lecture room after exiting on the left.

-- 
Diane Stidle
PhD Program Manager
Machine Learning Department
Carnegie Mellon Universitystidle at andrew.cmu.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://redfish.com/pipermail/friam_redfish.com/attachments/20240729/b9d7b882/attachment.html>