Vision Sharing for Collaboratively-teleoperated Mobile Manipulators
Haoran Zhao and Zhijun Li, Senior Member, IEEE Department of Computer Science Shanghai Jiao Tong University Shanghai, 200240 zhr5210@sjtu.edu.cn


Abstract—In this paper, we design the teleoperated mobile manipulators and build a system where they are able to cooperate with multiple users. The 6-DOF mobile manipulator with 6 joints, 1 microphone, 2 cameras and 2 wheels has a broad workspace and flexible mechanical arms as well as can see and hear. First, the structure of teleoperated mobile manipulator is provided and the motion control of it is introduced. Further, the collaboratively-teleoperated mobile manipulator is implemented which is also the goal of our research. Based on the network platform with Users-Server-Robots mode, both commands transport and multimedia data transport are realized. Therefore, controllers could obtain the real-time remote state information of manipulators which is called vision sharing, and send the control commands in time to them as feedback. Finally, several experiments are carried out with the actual mobile manipulator in our lab, which later has achieved the expected result. Index Terms—Cooperative Manipulators, Computer Vision for Robotics and Automation, Teleoperation

I. INTRODUCTION

T

mobile manipulator is a new-type robot designed by us which is integrated advantages of teleoperated camera, mobile robot and automatic arm while simultaneously reduced their drawbacks. It has a broader workspace and more flexible mechanical arms, even attain the ability of seeing and hearing. What is more, we have provided a network system for teleoperation, so people could attain the remote situation information and send corresponding commands.
ELEOPERATED

Fig 1 reveals the relationship of subordination and inheritance of teleoperated mobile manipulator. The applications of mobile manipulators are wide-ranging, such as construction, observation, exploration, transport, etc. A traditional telerobot is usually based on one-to-one master-slave mode. However, more and more complex tasks need cooperation among multiple users and robots, this mode will not be able to meet the demand. In this paper, we reform our teleoperated mobile manipulator to be controlled in many-to-one even many-to-many mode. Recently, “collaborative-teleoperation” is mentioned [2]. By “collaborative-teleoperation”, we mean a platform where a number of participants simultaneously share control. In our settings, it is possible to control multiple mobile manipulators at the same time. Thus we implement a system for multi-user & robot. Fig 2 (In this figure, the dashed indicates a remote linkage via network) illustrates the spectrum of the teleoperation control framework. We label those figures a-e, in order of increasing degree of automatization. On the far left would be a simple linkage where the human directly operates the robot by sliding mechanical bars, and on the far right would be an automatic system where multiple users and robots cooperate closely via the network server. The “collaborative-teleoperation” system we implemented belongs to type (e).

user

user

user

user

user

n …… teleoperated camera mobile manipulator teleoperated mobile manipulator automatic arms computer platform computer computer computer

n ……

server computer

mobile robot

robot A

robot B

robot C

robot D

n ……

E

Fig. 1

Fig. 2

The research goal of this paper is to build the “collaborative-teleoperation” system for mobile manipulators. The task of this object can be divided into two parts: the first one is to design the mobile manipulator while the second one is to implement collaborative-teleoperation function. The details of collaborative-teleoperation function are demonstrated in Fig 3. From this figure we know, mobile manipulators with camera and microphone should share their vision with observers, and controllers can send corresponding commands at any time [3]. Vision sharing as a hot task, can be found in many literatures such as [4], [5]. However, we will provide a new solution to it with the advantages of high quality and little delay which is more suitable for the collaboratively-teleoperated mobile manipulators.

The designed architecture of the mobile manipulator is shown in Fig 4, which includes a non-holonomic mobile platform and a 6-DOF manipulator. Obviously, the 6 revolute joints compose the more flexible mechanical arms and bring a broader workspace. A mobile manipulator consists of 11 components including 2 cameras, 1 microphone, 6 joints and 2 wheels. Every joint is driven by its corresponding DC motor and each DC motor is controlled by its corresponding driver. In Fig 5, the control schema is exhibited. The internal computer receives remote control commands from controllers and changes them into TTL signals, then these signals will be delivered to motor drivers. Motor drivers are able to provide expected voltage to DC motors based on the signals.

Board Computer

DIO PCI Servo Driver Servo Motor Encoder (Build in)

controller user port observer network server robot port

manipulator

PCI

camera&mike

Servo Driver Servo Motor Encoder Mobile Base

Counter Board

Encoder

Manipulator

Manipulator

Fig. 3

Fig. 5

This paper is organized as follows. In section 2, the design of teleoperated mobile manipulator is described. In section 3, the system for collaboratively-teleoperated mobile manipulators is designed. In section 4, the vision sharing for mobile manipulators is implemented. In section 5, the corresponding experimental results are illustrated, followed by conclusions in section 6. II. DESIGN OF TELEOPERATED MOBILE MANIPULATOR

It is an important and difficult task to let the mobile manipulator follows along a suitable trajectory when the manipulator is under teleoperation. However, the motion control of the mobile manipulator is expatiated in the previous paper of our lab [1]. Actually, the task can be divided into two parts: the first one is to find a suitable trajectory and the second one is to translate this trajectory into a series of corresponding control commands for the joints which will be applied as the input to the system. III. COLLABORATIVELY-TELEOPERATED MOBILE MANIPULATOR Traditional network platforms usually include two parts: server and client. Our platform is also based on server-client mode as a whole but has some transformations. The unique server port only exists in Data Center, the traditional client ports are reformed to user ports and robot ports, in which several instances could be in active state simultaneously. To let the server supervise all the teleoperation information from multiple users, we change all commands into text form and deliver them to a certain robot through the server. Since the internal control information also belongs to the text form, we design the Standard Text-information Transport Protocol (STTP) to achieve this object. Just as other protocols, STTP is made up with datagram and header. Datagram includes the

Fig. 4

content, header translates the datagram by the Flag (an integer parameter). The meaning of the Flag is as follows: const int PASS_COMMAND = 0; // command information const int USER_LOGIN = 1; //note of user log-in const int ROBOT_LOGIN = 2; //note of robot log-in const int USER_LOGOUT = 3; //note of user log-out const int ROBOT_LOGOUT = 4; //note of robot log-out const int ROBOT_LIST = 5; //request for active robots list const int CLIENT_IP = 6; //request for IP of related client const int INFO_CHANGE = 7; //note for changing self-information const int STATE_REQUIRE = 8; //request for vision sharing Since the commands packed by STTP are independent of multimedia information as well as can be transmitted among users, sever and robots at any time, users can get in touch with the latest status of the robots and do remote manipulation to them. Therefore, we apply CAsyncSocket class from MFC to realize socket transmitting. Considering its small size and important content, we use stable TCP to transmit the text to avoid the severe consequences of packet-loss. To let multi-user send commands simultaneously, Server port is based on message responding mechanism of multi-thread. We create the listen socket in main thread and keep it waiting for connection. While the listen socket detects a connection request, new socket will be created to accept it in a newly-built sub-thread. The sub-thread is also based on message responding mechanism of asynchronous socket, keeping waiting for receiving the STTP packet or the delivery request from other sub-threads. For implementing the communication among threads, we create a global hash table, which is used to search the thread name of related client port by the client ID. Hence we attain the commands transport part of collaborative-teleoperation function. IV. VISION SHARING FOR MOBILE MANIPULATOR Vision sharing is the key function of this system. Cameras and microphone are necessary hardware devices that enable the mobile manipulator to see and hear. We lean our port programming on MFC for its vast number of solutions to collection/compression multimedia and net-based transferring. As to attain little delay and high quality, we finally realize the vision sharing function by the means listed below. For the transmitting side, video collecting thread collects video data by VFW continuously from the camera. Then video encoding thread encodes the data into MPEG-4 format by Xvid technology and saves them to a linked list. Each node of the linked list includes a single frame of video stream. We add

a new node to the end of the list after getting a frame, and we take away an existing node from the head of the list for sending. The linked list maintains a dynamic transmitting buffer queue in theory. Image size is set to 640*480*24 bit per frame, data acquisition ratio is 25 frames per second and compression ratio is initialized as ARG_BITRATE level. Similarly, audio thread collects audio data by DirectSound from the microphone and encodes the data by G729A. We set data acquisition ratio to 100 frames per second, data size to 640 bytes per frame and compression ratio to 16:1. Sending thread will pick up one frame from video linked list and four from audio linked list (the collection rate of audio information is four times higher than that of video) and combine them into one node. According to RTP [6], we divide the node into several RTP packets by the size of one kilobyte. After that, each packet is attached with a header by enclosing time stamp according to system time for synchronizing audio and video data, and adding serial number to which it is attached, as well as setting marker bit abiding by the rule that the bit of last packet is marked with 1 and others 0. Every T time, a packet is sent to each IP address in the multicast list of related users. The value of T is initialized as the time of host machine performing 1,000 idle loops. For the receiving side, thread keeps receiving RTP packets and saves them in buffer array until the marker bit of packet turns to 1. Due to the instability of RTP, the order of received packets may differ from the original order, so packets in buffer array should be sorted according to the serial numbers. During the process of sorting, the packet-loss rate is able to be evaluated from No-off situation of the serial numbers. We regard the node as a lost node if packet-loss situation gets serious, and then send retransmission request to the transmitting side as feedback. Inversely, we consider the node to receive successfully and then this node will be decoded and played by another thread. The feedback mechanism constructs a significant basis for the transmitting side to calculate packet-loss rate.

User VFW Play video Play sound Collect video Xvid Decode video Xvid Sort packets Decode sound G729A Feedback of packet-loss Encode video

Robot DirectSound Collect sound G729A Encode sound

Transmitting buffer linked list

Receiving buffer array

RTP packets of multimedia

AIMD & Coderate-control

Fig. 6

The whole procedure is illustrated in Fig 6. Analysis of system efficiency is demonstrated in the following argument:

The size of each packet is 1KB and the interval between two transmissions is T, so the network footprint is 1KB/T when the network condition is quite good. However, when the high packet-loss rate is realized from the feedback, it means the value of T is not high enough. In this situation, the transmitting side is able to increase the value of T to reduce packet-loss rate. As shown in experiments, the video information average size is less than 4KB per frame after being compressed by Xvid with the image size of 640*480*24 bit per frame and data acquisition ratio of 25 frames per second. While the audio information size is: 640 bytes/16 = 40 bytes; So the size of a node is: 4KB+40B *4