Research

I study how manipulation tasks are specified to robots and how they are represented, and what each choice assumes. For example, a task can be given during training, as a reward or a dataset of demonstrations, or at deployment, as a goal, a language instruction, or a single demonstration. Tasks can be represented explicitly, as mathematical constraints on how the object should move, or implicitly, through what a learned model captured from its training data. These choices decide what a method can do: which tasks it can be specified for, what it means for it to generalize, and how we can quantify if a task is successful.

Ongoing

Multimodal aerial and underwater robots. As a postdoctoral associate in the Soft Flyers Group, I work on robots that operate both in the air and under water. More here once the work is published.

Task specification and representation across paradigms. Relating types of manipulation tasks to the specifications and representations that suit them, across planning, reinforcement learning, and immitation learning with diffusion policies and VLAs. In progress.

Task and motion planning with screw motions. Research on utilizing the screw-geometric structure of motion in a task and motion planning system. In progress.

Utilizing the inherent geometric structure of motion for representing tasks

Most of my PhD explored one explicit representation: describing a task in terms of the motion the object goes through, as a constant screw motion or a sequence of them. By Chasles' theorem, any rigid body motion can be approximated arbitrarily closely as a sequence of constant screw motions, so the same description covers opening a drawer, turning a knob, pivoting a box, pouring, scooping or even transplanting saplings in a containerized vertical farm.

Most robotic manipulation today remains limited to pick-and-place operations, in which grasp synthesis and motion planning for manipulation are traditionally treated as isolated problems. On the contrary, grasping and motion planning are closely related. The manner in which an object is grasped should be determined by how it is supposed to be moved. While pivoting a box, for example, the grasping region depends on the axis of pivot, and good grasps for one pivoting axis may be bad for another.

Utilizing the screw-geometric structure of motion as the task representation, I developed algorithmic and learning-based approaches for task-oriented grasp synthesis from sensor data. The same representation can also be used to plan the motion of the object using Screw Linear Interpolation (ScLERP) with Resolved Motion Rate Control (RMRC). Moreover, if the sequence of screws, corresponding to the desired object's motion is not know beforehand, then it can be extracted from a single kinesthetic demonstration and used to plan the motion.

  1. 1 Describe the task as a sequence of constant screw motions of the object
  2. 2 Score grasps based on their feasibility to impart the desired motion
  3. 3 Find grasps from point clouds, and regrasp when one grasp isn't enough
  4. 4 Plan the motion with screw linear interpolation, from demonstrations and with contacts

Projects

Task-dependent grasp metric

Scoring a grasp by whether it can produce the motion a task needs, such as turning a knob or pivoting a box about its edge, computed as a second-order cone program.

Motion planning from demonstrations

Extracting the constraints of a task like pouring or scooping from a kinesthetic demonstration, and reusing them for new object positions, new objects, and harvesting in vertical farms.

How many demonstrations are enough?

Letting the robot evaluate its own plans and ask for kinesthetic demonstrations incrementally, so it can pour and scoop reliably over a region of its workspace with as few demonstrations as possible.

Design of a robotic finger

A compact 3-DoF series-parallel finger with abduction/adduction and flexion/extension, with closed-form kinematics so the fingertip's motion and force can be controlled.

Code

tograsp-socp

The task-dependent grasp metric as a second-order cone program. Takes an object point cloud and a task screw, scores antipodal contacts on the object's bounding box, and returns the grasping region and candidate end-effector poses.

git clone https://github.com/apat20/tograsp-socp.git && cd tograsp-socp
conda env create -f environment.yml && conda activate tograsp_socp
python main_gcsm.py --filename nontextured.ply

PyScLERP

Screw linear interpolation (ScLERP) for computing task-space paths in SE(3), with examples of pivoting and sliding a cuboid while it stays in contact with the table, and a pivot, pick, transfer and place plan.

git clone https://github.com/apat20/PyScLERP.git && cd PyScLERP
conda env create -f environment.yml && conda activate pysclerp
python main_pivoting.py

Task-Oriented Grasping from Point Clouds

Code for our IROS 2023 paper. A neural network predicts the grasp metric on the bounding box of a partial point cloud, giving a grasping region for pivoting a cuboidal object. Includes RealSense point clouds of three boxes to try it on.

git clone https://github.com/irsl-sbu/Task-Oriented-Grasping-from-Point-Cloud-Representation.git
cd Task-Oriented-Grasping-from-Point-Cloud-Representation
conda env create -f environment.yml && conda activate minimal_env
python -u main_pivoting.py --filename partial_point_cloud/cheezit_cracker_box.ply --visualize

Hardware and tools