Task-dependent grasp metric
Scoring a grasp by whether it can produce the motion a task needs, such as turning a knob or pivoting a box about its edge, computed as a second-order cone program.
Research
I study how manipulation tasks are specified to robots and how they are represented inside them, and what each choice assumes. A task can be given as a goal, a reward, a language instruction, or a set of demonstrations. Inside the robot it can be represented explicitly, as constraints on how the object should move, or implicitly, through what a learned model picked up from its training data. These choices decide what a method can do: which tasks it can be specified for, what it means for it to generalize, and how anyone can tell the task is done.
Ongoing
Task specification and representation across paradigms. Relating types of manipulation tasks to the specifications and representations that suit them, across planning, reinforcement learning, and learning from demonstration with diffusion policies and VLAs. In progress.
Multimodal aerial and underwater robots. As a postdoctoral associate in the Soft Flyers Group, I work on robots that operate both in the air and under water. More here once the work is published.
One representation, studied in depth
Most of my PhD explored one explicit representation: describing a task by the motion the object goes through, as a constant screw motion or a sequence of them. By Chasles' theorem, any rigid body motion can be approximated arbitrarily closely by such a sequence, so the same description covers opening a drawer, turning a knob, pivoting a box, pouring and scooping. I used it to judge whether a grasp can produce the motion, to find grasps from point clouds, and to plan the robot's motion, often from a single kinesthetic demonstration.
- 1 Describe the task as a sequence of constant screw motions of the object
- 2 Score grasps by whether they can produce that motion
- 3 Find grasps from point clouds, and regrasp when one grasp isn't enough
- 4 Plan the motion with screw linear interpolation, from demonstrations and with contacts
Projects
Task-oriented grasping from point clouds
Finding a good grasping region for a task directly from a partial point cloud, using a neural network trained on the grasp metric instead of labeled grasps.
Grasps and regrasps for multi-step tasks
Deciding whether one grasp can carry out a whole manipulation plan, and where the robot needs to regrasp when it can't.
Motion planning from demonstrations
Extracting the constraints of a task like pouring or scooping from a kinesthetic demonstration, and reusing them for new object positions, new objects, and harvesting in vertical farms.
How many demonstrations are enough?
Letting the robot evaluate its own plans and ask for kinesthetic demonstrations incrementally, so it can pour and scoop reliably over a region of its workspace with as few demonstrations as possible.
Manipulation with environment contact
Motion, force and trajectory planning for tasks where the object stays in contact with the environment, such as pivoting a heavy box about its edge.
Design of a robotic finger
A compact 3-DoF series-parallel finger with abduction/adduction and flexion/extension, with closed-form kinematics so the fingertip's motion and force can be controlled.
Code
tograsp-socp
The task-dependent grasp metric as a second-order cone program. Takes an object point cloud and a task screw, scores antipodal contacts on the object's bounding box, and returns the grasping region and candidate end-effector poses.
git clone https://github.com/apat20/tograsp-socp.git && cd tograsp-socp
conda env create -f environment.yml && conda activate tograsp_socp
python main_gcsm.py --filename nontextured.plyPyScLERP
Screw linear interpolation (ScLERP) for computing task-space paths in SE(3), with examples of pivoting and sliding a cuboid while it stays in contact with the table, and a pivot, pick, transfer and place plan.
git clone https://github.com/apat20/PyScLERP.git && cd PyScLERP
conda env create -f environment.yml && conda activate pysclerp
python main_pivoting.pyTask-Oriented Grasping from Point Clouds
Code for our IROS 2023 paper. A neural network predicts the grasp metric on the bounding box of a partial point cloud, giving a grasping region for pivoting a cuboidal object. Includes RealSense point clouds of three boxes to try it on.
git clone https://github.com/irsl-sbu/Task-Oriented-Grasping-from-Point-Cloud-Representation.git
cd Task-Oriented-Grasping-from-Point-Cloud-Representation
conda env create -f environment.yml && conda activate minimal_env
python -u main_pivoting.py --filename partial_point_cloud/cheezit_cracker_box.ply --visualizeHardware and tools
- Robots: Franka Emika Panda
- Sensors: Intel RealSense depth cameras
- Software: Python, C++, ROS, PyTorch, cvxpy, Open3D, OpenCV, PCL, Isaac Sim, PyBullet