HERB, the Home Exploring Robot Butler, is a bi-manual mobile manipulator designed and built by the Personal Robotics Lab at Carnegie Mellon University to complete household manipulation tasks. As a member of the Personal Robotics Lab, I developed many of the planning, perception, and control algorithms used by HERB to autonomously manipulate his environment.

I was the lead developer of HerbPy and PrPy, the core Python libraries used to control HERB. This software, developed in collaboration with my colleagues in the Personal Robotics Lab, spans the spectrum of robotics, including:

• Motion Planning. I designed and implemented HERB's motion planning pipeline, which uses a repertoire motion planners, run in sequence, to quickly solve motion planning problem. He begins by planning with fast heuristics and, if that fails, falls back on more sophisticated algorithms. These algorithms include a Jacobian-based controller, trajectory optimizers (CHOMP and Trajopt), and sample-based motion planners (CBiRRT and LEMUR).
  As part of this effort, I implemented OpenRAVE bindings for the Open Motion Planning Library (OMPL) and implemented a simple constrained motion planner that uses numerical integration to forward-simulate the execution of a Jacobian-based controller.

• Multi-Step Planning. I implemented a backtracking search strategy, based on HGPC, to find geometrically feasible solutions to multi-step manipulation tasks. This framework allows HERB to reason about dependencies between steps of the task, e.g. choosing a poor position to place an object may make it impossible to complete later step of the task.

• Trajectory Post-Processing. Most geometric motion planners return un-timed piecewise linear paths that must be post-processed, via shortcutting and timing, before execution. I implemented this functionality by integrated a parabolic trajectory smoother into HERB's motion planning pipeline.
  I implemented a framework that allows HERB to pipeline trajectory post-processing with execution to speed up execution of multi-step manipulation tasks. This is made possible by annotating each path with all data necessary to post-process it without altering its feasibility; e.g. active constraints.

• Perception. I worked with perception experts in the Personal Robotics Lab to develop a standard interface for object detection and pose estimation algorithms. HERB uses different approaches to detect objects with different properties, including feature matching for textured objects (SimTrack), template matching for textureless objects (VNCC), and visual fiducial detection for objects with challenging material properties (Apriltag).

• Control. I was the lead developer of a ros_control interface for controlling HERB's Barrett WAM arms and BarrettHand end-effectors over CAN. This is a hard real-time controller that commands torques generated by an inverse dynamics controller at 500 Hz–1 kHZ. We plan to release this controller as open source software.

Applications

We have used this software to make HERB complete a variety of tasks, including: stocking a bookshelf, clearing a table, starring in a play, and completing a number of YCB tasks. Videos of HERB completing these tasks has been widely featured by the press, including National Geographic's Robots movie shown in IMAX theaters, an Oreo cookie advertisement, and an episode of Katie Couric's Cities Rising: Rebuilding America series. Several highlights are included below.