Task-dependent grasp metric
Scoring a grasp by whether it can produce the motion a task needs, such as turning a knob or pivoting a box about its edge, computed as a second-order cone program.
I study how manipulation tasks are specified to robots and how they are represented, and what each choice assumes. For example, a task can be given during training, as a reward or a dataset of demonstrations, or at deployment, as a goal, a language instruction, or a single demonstration. Tasks can be represented explicitly, as mathematical constraints on how the object should move, or implicitly, through what a learned model captured from its training data. These choices decide what a method can do: which tasks it can be specified for, what it means for it to generalize, and how we can quantify if a task is successful.
Ongoing
Multimodal aerial and underwater robots. As a postdoctoral associate in the Soft Flyers Group, I work on robots that operate both in the air and under water. More here once the work is published.
Task specification and representation across paradigms. Relating types of manipulation tasks to the specifications and representations that suit them, across planning, reinforcement learning, and immitation learning with diffusion policies and VLAs. In progress.
Task and motion planning with screw motions. Research on utilizing the screw-geometric structure of motion in a task and motion planning system. In progress.
Most of my PhD explored one explicit representation: describing a task in terms of the motion the object goes through, as a constant screw motion or a sequence of them. By Chasles' theorem, any rigid body motion can be approximated arbitrarily closely as a sequence of constant screw motions, so the same description covers opening a drawer, turning a knob, pivoting a box, pouring, scooping or even transplanting saplings in a containerized vertical farm.
Most robotic manipulation today remains limited to pick-and-place operations, in which grasp synthesis and motion planning for manipulation are traditionally treated as isolated problems. On the contrary, grasping and motion planning are closely related. The manner in which an object is grasped should be determined by how it is supposed to be moved. While pivoting a box, for example, the grasping region depends on the axis of pivot, and good grasps for one pivoting axis may be bad for another.
Utilizing the screw-geometric structure of motion as the task representation, I developed algorithmic and learning-based approaches for task-oriented grasp synthesis from sensor data. The same representation can also be used to plan the motion of the object using Screw Linear Interpolation (ScLERP) with Resolved Motion Rate Control (RMRC). Moreover, if the sequence of screws, corresponding to the desired object's motion is not know beforehand, then it can be extracted from a single kinesthetic demonstration and used to plan the motion.
Scoring a grasp by whether it can produce the motion a task needs, such as turning a knob or pivoting a box about its edge, computed as a second-order cone program.
Finding a good grasping region for a task directly from a partial point cloud, using a neural network trained on the grasp metric instead of labeled grasps.
Deciding whether one grasp can carry out a whole manipulation plan, and where the robot needs to regrasp when it can't.
Extracting the constraints of a task like pouring or scooping from a kinesthetic demonstration, and reusing them for new object positions, new objects, and harvesting in vertical farms.
Letting the robot evaluate its own plans and ask for kinesthetic demonstrations incrementally, so it can pour and scoop reliably over a region of its workspace with as few demonstrations as possible.
Motion, force and trajectory planning for tasks where the object stays in contact with the environment, such as pivoting a heavy box about its edge.
A compact 3-DoF series-parallel finger with abduction/adduction and flexion/extension, with closed-form kinematics so the fingertip's motion and force can be controlled.
The task-dependent grasp metric as a second-order cone program. Takes an object point cloud and a task screw, scores antipodal contacts on the object's bounding box, and returns the grasping region and candidate end-effector poses.
git clone https://github.com/apat20/tograsp-socp.git && cd tograsp-socp
conda env create -f environment.yml && conda activate tograsp_socp
python main_gcsm.py --filename nontextured.plyScrew linear interpolation (ScLERP) for computing task-space paths in SE(3), with examples of pivoting and sliding a cuboid while it stays in contact with the table, and a pivot, pick, transfer and place plan.
git clone https://github.com/apat20/PyScLERP.git && cd PyScLERP
conda env create -f environment.yml && conda activate pysclerp
python main_pivoting.pyCode for our IROS 2023 paper. A neural network predicts the grasp metric on the bounding box of a partial point cloud, giving a grasping region for pivoting a cuboidal object. Includes RealSense point clouds of three boxes to try it on.
git clone https://github.com/irsl-sbu/Task-Oriented-Grasping-from-Point-Cloud-Representation.git
cd Task-Oriented-Grasping-from-Point-Cloud-Representation
conda env create -f environment.yml && conda activate minimal_env
python -u main_pivoting.py --filename partial_point_cloud/cheezit_cracker_box.ply --visualize