Games with Panda

I worked on this project over the summer of 2021 with the Human-Robot-Interaction Lab with Sarah Sebo in collaboration with the UChicago Center for Data and Computing For this project we prepared a Franka-Emika Panda robot, to visually detect objects in the real world using an azure-kinect depth sensitive camera setup calculate the position of the objects relative to the robot, and grasp the objects when in range. This sets up the robot to play interactive games such as tic-tac-toe or chess.

For interfacing with the robot we used ROS (Robot Operating System) programmed with python and C++. We used the Movit planning framework in combination with our own custom reverse kinematics calculations that set the positions of the 7 joints of the robot correctly to reach the given coordinate. To parse through the image sensed, as well as recognizing and locating AR-tags with the Kinect we used . From this image we were able to calculate the world position of the camera and the world coordinates of the detected objects. The entire setup is very modular, with seperate steps running on separate ROS-nodes that can be run across different devices on the same network. This allows us to bypass compatibility issues (Kinect and OpenCV require strong graphics hardware, and that often conflicts with the current Franka-Emika drivers.)

Image from azure documentation: Sample of view provided by camera that we calculated object positions from.
Collaborators: Jason Lin
Mentor: Sarah Sebo