At the terminus of every robotic arm is an end-effector. These are the hand-like devices that directly interact with the environment. Collecting data from the vantage point of an end-effector is highly desirable, though difficult to accomplish in practice. Wired data transmission solutions which use internal cables suffer from signal degradation due to proximity to power conduits. Moreover, the solutions which use external cables are subject to tangling and often impose limits on the robot’s kinematics. Finally, existing wireless data transmission solutions are rarely used due to the low transmission rate, high latency time and unstable connection. This research focuses on developing a general approach to stream visual (RGBD) and haptic (force) data wirelessly to other devices. The resulting system should consist of an architecture which interfaces with a suite of end-effector sensors and handles signal compression, transmission, and decompression efficiently. Additionally, this project involves a hardware design portion which aims to neatly package and mount this infrastructure at the robot end-effector while satisfying power and kinematic constraints for the robot. This research chose the Intel Joule, an embedded Linux board, as the infrastructure to handle wireless transmission. The embedded board uses a low-level Python networking package, which forms the connection to transmit data to other devices on the same network. This project implements packages to encode, transmit and decode the data stream with Python. The receiving device decodes the compressed data and makes the data available to other devices on the same network. This project allows sensors and the main computer of the robot wireless connected and future robot would operate movement more freely.

The ultimate goal of robotics is to be able to interact with robots as one would with another person; hand in hand with this goal lies the ability for robots to understand natural language and carry out tasks in practical environments. Let us take for example, if a human were to command a robot to “pick up a can of coke inside a cardboard box on a table”. The robot should be able to extract relational information such as a can of coke is inside a cardboard box and a cardboard box is on a table. If “coke” is unknown, the robot should be able to ask clarifying questions to ascertain the qualitative aspects that define it. So, robots should be able to reason in the presence of uncertainties during object manipulation in the real world. We use a Fuzzy Markov Logic Network (Fuzzy-MLN) to learn and predict object classifications given input from multiple sensory modalities. Inputs from vision (RBG-D), haptics, and audio are used along with a prior knowledge base to identify objects in the robot’s environment and annotate various attributes which are added to the MLN. Audio input is processed for natural language which can provide information to contextualize the robot’s environment or give instructions to/answer queries from the robot in order to learn new relations, update Fuzzy-MLN weights, and improve annotator accuracy.