2009 IEEE/ASME International Conference on Advanced Intelligent Mechatronics Suntec Convention and Exhibition Center Singapore, July 14-17, 2009

A Virtual Reality Operator Interface Station with Hydraulic Hardware-in-the-Loop Simulation for Prototyping Excavator Control Systems
Mark D. Elton, Aaron R. Enes, and Wayne J. Book, Fellow, IEEE

Abstract— A multimodal operator interface station is developed to display a realistic virtual reality depiction of a compact excavator performing general digging tasks. The interface station includes engine audio feedback and a near life-size operator display attached to a full-size cab. The excavator dynamics are determined by models of the hydraulic system, the linkage system, and the soil digging forces. To maximize the fidelity of the hydraulic model, certain “virtual” components of the model are replaced with real-time hardware-in-the-loop (HIL) simulations of the actual hardware. HIL simulation is done in a geographically isolated facility, with Internet based communication between HIL and the Remote Operator Interface. This is the first reported high-fidelity operator interface to be combined with remote hydraulic HIL simulations.

I. I NTRODUCTION Hydraulic hardware has undergone a great evolution in recent years, evolving from purely hydro-mechanical devices to electro-hydraulic systems controlled by microprocessors. The use of electronic controllers opens the door to improving dynamic performance and enhancing traditional hydraulic off-highway construction machines with new features such as increased energy efficiency, improved operator controllability, and overall increases in productivity. With these added capabilities often comes added system complexity, particularly in the area of system controls and operator interfaces. New controllers are designed and often tested on models of systems, but hydraulic components are highly nonlinear and difficult to model. To aide in the testing of new complex control systems, we developed a HIL testbed for emulating the hydraulic loads incident on a variable displacement pump during operation, allowing for easy changes to be made to the control architecture while avoiding errors caused by incomplete models. The most complicated part of mobile hydraulic machines such as farm, construction, and mining equipment is the human operator. To measure and evaluate energy efficiency
This work was made possible through the generous support of HUSCO International: particular acknowledgment is made to Gus Ramirez, CEO; and to Dwight Stephenson, VP of Engineering. Acknowledgment is also made to the Bobcat Company for in kind donations and to J.D. Huggins, a Research Engineer at Georgia Tech. This work is affiliated with the Center for Compact and Efficient Fluid Power, an Engineering Research Center funded by the National Science Foundation (contract EEC-0540834). A. Enes is a Ph.D. Candidate aaron.enes@gatech.edu M. Elton is a Ph.D. Student mark.elton@gatech.edu W. Book is a Professor and the HUSCO/Ramirez Distinguished Chair in Fluid Power and Motion Control wayne.book@me.gatech.edu All authors are in the Department of Mechanical Engineering at the Georgia Institute of Technology, Atlanta, GA 30332, USA

or productivity changes of a new control design, humangiven control signals must be input to the plant. Often during testing, these signals are pre-recorded and thus remove the operator dynamics from the control loop. However, an operator’s input is often a function of the system response; thus, if a new control design causes a different response, it is critical that the operator’s response to the change is also captured. We espouse that to get a true measure of a system’s performance under a given control design, the system (or close approximation to the system) must be tested in the field (or close analog) with a human at the controls. Implementing a new controller on a machine is time consuming and expensive. Virtual reality operator interfaces have long been employed as a method to study new human/machine interfaces before investing the time, effort, and money to implement that technology in the real world [1]. Virtual reality also allows operators to train without endangering others or equipment, and allows for easy manipulation of the environment to test circumstances that while possible, would be difficult to create and largely unrepeatable. Since real-time human commands are necessary to understand a human-operated, closed-loop control system, we propose using a remote operator interface (ROI) for the HIL testbed. The ROI simulates many aspects of a variable displacement pump controlled compact excavator and sends the HIL testbed real-time commands. The process will also work in reverse. To accurately test new operator interfaces, the system being controlled must be accurately modeled. To avoid incomplete models, the HIL testbed will give the ROI simulation accurate (non-modeled) information about the state of the pumps. This paper is organized as follows: background, overall system architecture, excavator simulator, HIL testbed description and capabilities, and conclusions. II. BACKGROUND An operator interface refers to the means by which a human operator interacts with a machine, for instance the type and location of input devices, and the details of any haptic or audio/visual feedback. Evaluation of operator interfaces is often done with the assistance of virtual reality. Before evaluations can take place, a virtual environment is constructed to represent, with sufficient fidelity, the actual environment. Graphical programs can easily consume a computer’s resources, so trade-offs are made between the speed and quality of the rendering. Past academic researchers used

978-1-4244-2853-3/09/$25.00 ©2009 IEEE

250

graphical simulations of excavators to test operator interface designs but the need for increased computation speed leads to sacrificed fidelity of the excavator and soil dynamic models and the graphical representation of the environment [1], [2], [3], [4]. Existing industrial simulations generally have much more realistic graphics, but their proprietary software and rigid design makes them inflexible and thus are typically used to test small changes from the state of the art. One problem is that some critical components of the system (i.e. those components that have a great impact on the final system performance or response) are difficult to model accurately, and the phenomenon neglected by inaccurate models may have appreciable impact on the final system performance. To avoid this, some operator interfaces (e.g. haptic input devices) have been successfully tested by installing the prototype interface on the actual mobile hydraulic equipment [4], [5], [6], [7]. These observations are often more thorough, but do not allow for easy testing of prototype control architectures because installation on a real machine necessarily establishes the system structure. HIL, on the other hand, offers more flexibility in the system architecture while not compromising the fidelity of fullscale experimentation. Hydraulic HIL testbeds are common [8], [9], [10], [11], but the main way they are used in practice is by tracking pre-recorded trajectories. Some systems allow real-time inputs from an operator, with the operator interface located in close proximity to the HIL testbed; often the focus is on the HIL testbed and little focus is placed on the operator interface. The HIL-ROI testbed presented here is unique because of its virtual reality operator interface used in tandem with a HIL simulation of more complex hydraulic components. III. OVERALL A RCHITECTURE Fig. 1 shows the HIL-ROI communication network (the four computers will be referred to by italicized names). The excavator simulation is run on the Sim PC with Mathwork’s Real Time Workshop and the xPC Target OS. The Sim PC solves the dynamics of the hydraulic and mechanical systems. The MAIN PC draws the graphical simulator, plays audio feedback, stores runtime data, and compiles and loads the real-time code for Sim PC. The Controls PC connects to the actual operator interface hardware (presently a SensAble Phantom Premium 1.0 device for omni-directional haptic feedback) and sends the user commands to Sim PC. The top three computers in Fig. 1 are all connected via Ethernet to a local network. The HIL PC and the MAIN computer communicate through TCP/IP, thereby allowing the operator interface station to be geographically remote from the HIL system to potentially allow several ROI stations to utilize a single HIL facility. Typical round-trip communication time is on the order of 10 ms. The MAIN computer sends the plant’s inputs (discussed in Section V) to the HIL PC, which in turn sends the actual pump flow to the Sim PC via the MAIN computer.

Controls PC Operator inputs

Sim PC

Pump pressures, plant command, graphics parameters Measured plant response

MAIN PC

Calculated feedback Operator Inputs Feedback HIL PC

Controls Measured plant response HIL

Measured plant response Internet Pump pressures, plant command

Low-level commands

Fig. 1: The HIL-ROI communication network

Fig. 2: The Remote Operator Interface consists of an excavator cab with an LCD screen mounted on the front window

IV. E XCAVATOR S IMULATOR A. Operator Workstation The operator workstation is installed within the cab of a Bobcat 435 Compact Excavator (Fig. 2 and Fig. 3). A 132 cm (52 inch) high-definition LCD monitor is mounted near the front window of the cab, spanning both the upper and lower windshields to produce a 135 degree vertical field of view. The monitor displays the simulated excavator boom structure that the operator is controlling and the environment representing the dig site. B. Graphical Interface The graphical representation of the boom structure is displayed on the monitor by a C++ program using OpenGL on the MAIN computer. The program uses the manufacturer’s CAD models of the offset, boom, stick, and bucket links

251

by the bucket. C. Excavator Linkage and Hydraulic Dynamics Dynamics for the excavator links (bucket, arm, boom, offset, and swing) are modeled using well-known NewtonEuler techniques and are solved in real-time on the Sim PC. Model parameters including inertia tensors and relevant dimensions are supplied by manufacturer’s CAD data. The excavator hydraulic system is a prototype valve-less control architecture, with a dedicated hydraulic pump for each of the degrees of freedom. Fig. 6 is a representative schematic of a single function (swing). The hydraulic system is modeled using standard fluid power modeling techniques [12] and simulated in real-time with the linkage dynamics. D. Audio Feedback The sound of the engine is played to further immerse the operator in the virtual environment. A five second audio clip is looped to play continually during simulation. The volume is varied as a function of the hydraulic power required as in V olume = V olumeidle + k(Prequired − Pidle ). This variation in perceived engine noise is an important feedback cue as it represents a metric used by operators for machine utilization. E. Soil Model Fig. 4: Screen shot of the graphics shown on the cab-mounted monitor Soil is difficult to model because its parameters (e.g. water content, density, shear strength) are not homogeneous and vary greatly with soil type. In the simulator, the soil is modeled as a homogeneous substance with constant parameters. The soil model is based upon previous work [4], [13], and most heavily on the work done by [14]. These other works consider only teeth-first bucket motion within the soil and neglect the forces from motion both opposite and perpendicular to that direction. However, novice and experienced operators routinely contact the soil with the sides and bottom of the bucket; hence, for a complete digging model, these existing soil models must be modified. The soil is analyzed in cylindrical coordinates, but instead of using the standard (and fixed) rθz directions, a new coordinate system is defined by the position of the teeth ˆ and flat of the bucket, as in Fig. 5: t and n are tangential ˆ ˆ and normal to the flat of the bucket and θ is into the page. With this the position of the bucket can be expressed as ˆ ˆ B(t, n, θ) = tt + nˆ + θθ. n A nonlinear spring and damper model was empirically found to correspond to the motion of an actual excavator during digging. The spring force exerted along n by the ˆ bucket is Fn = kn (nc − n), where n is the current position of the bucket teeth, and nc is the position where the soil last sheared. In previous work, kn was constant since motion was assumed to be teeth-first only. However, the soil acts as a stiff spring only if the bucket is moving forward through the soil. If the teeth penetrate the soil and then retract a

(a)

(b)

Fig. 3: (a) An operator inside the Operator Interface station. (b) The haptic joystick for controlling a prototype excavator system.

to render a realistic-to-production arm. The computational demand of detailed models requires multiple graphics cards to insure that the visualization can run at a 60 Hz refresh rate. Along with the excavator arm, a variety of trees are displayed to add a sense of depth and reality to the scene. For enhanced depth perception the shadows of the links are drawn on the ground. Fig. 4 shows a representative screen capture of the graphical display. The trench is shown as a flat green rectangle drawn on the ground where the operator should excavate. The soil surrounding the trench has a grass texture applied to it so that it is easily differentiated from the green of the trench. The soil in the bucket is displayed as a curved brown surface with the height of the surface dependent on the amount of soil in the bucket. Effort was taken to ensure a realistic visualization of the bucket emptying process. When soil is dumped from the bucket, the visualization shows the bucket empty and draws a large number of small soil particles falling. The number of falling particles is proportional to the amount of soil in the bucket. A pile of soil is displayed in the dump location after the soil particles reach the ground. If the bucket is unloaded over the trench then a sloping two-dimensional soil pile is drawn inside the trench. Currently soil can only be removed from the trench, but the soil outside the trench is penetrable

252

n
PA

QA

Bucket

Ground Level

Plant

d

PR J

-

t

N PB

cab & boom QB

Fig. 5: Axes used to determine soil interaction forces

small amount, the spring force of the soil is much smaller. Hence we allow kn to be a function of n and its velocity n. ˙ Similarly, in the other two directions: Ft = kt (tc − t) Fθ = kθ (θc − θ), ˙ where kt is a function of t and t , and kθ is a function of θ ˙ and θ. Soil also has a damping effect on the bucket’s motion. The damping parameters bt , bn , and bθ are also dependent on the subscripted coordinate and its derivative. If Sp is the length of the portion of flat of the bucket in contact with the soil and the velocity of the bucket tip is v, then the components of the damping force D are Dt = −Sp bt vt Dn = −Sp bn vn Dθ = −Sp bθ vθ . Further details of the complete soil model are discussed in forthcoming publications. V. T HE P LANT A prototype hydraulic control system for a Bobcat 435 Compact Excavator is being developed as part of the NSF Center for Compact and Efficient Fluid Power1 . The swing function of this machine causes the excavator cab and boom structure to rotate about a vertical axis. This function is powered by a hydraulic pump/motor (referred to as the “plant”) and supporting circuitry as in Fig. 6. Attached to the drive shaft of the plant is a diesel engine rotating with speed N (in reality, there is a bank of pump/motors–one for each DOF–driven by the same shaft; the speed N depends on the load of each pump/motor and the engine dynamics). Here, the plant is a Sauer-Danfoss 53 cc/rev H1 series variable displacement pump. The control input to the plant is the commanded displacement d, and the output, or response, of the plant is the actual flow QA which is an unknown (and notoriously difficult to model) dynamic function of N , d and workport pressures PA , PB . All dynamics–except the plant dynamics–of the excavator model (i.e. hydraulics, actuators, and rigid bodies) are simulated in real-time on the Sim PC computer. The plant
1 NSF

Fig. 6: Hydraulic circuit of swing function, with the Plant shown in shaded box

dynamic response QA is determined through HIL simulation, whereby the inputs (d, PA , PB , N ) are “displayed” to the real plant and the response is directly measured with a flow meter. VI. H ARDWARE - IN - THE -L OOP C ONTROL During a general dig cycle, the pump operates in each of four quadrants, defined by the signs of QA and the pressure difference (PA - PB ). This necessitates a hydraulic HIL system with capabilities not commonly discussed in literature, namely the ability to add and remove energy from the plant [8]. Fig. 7 is the circuit for emulating the loads on the plant. The HIL system is powered by two 340 VAC servo motors, capable of supplying 45 kW each. The speed N of the plant’s driven shaft is controlled with one motor, while a Wheatstone bridge arrangement of electro-hydraulic poppet valves (EHPVs) is the primary means of controlling the plant load PA and PB . A turbine flow meter allows measurement of either QA or QB with a resolution of 0.3 lpm. The workport through which flow exits (enters) is defined as the positive (negative) port. Note that during a dig cycle either port may become positive, depending on the direction of actuator motion. For ease of discussion, we assume port A is always positive; since the Wheatstone bridge valve arrangement is symmetric the following discussion is made valid by changing appropriate subscripts whenever port B is positive. All control signals and measurements (except flow) are transmitted to the HIL system on CANbus. Flow data is digitized with a National Instruments DAQ card on the HIL PC computer. A. Control of Pi The flow through the i-th EHPV is modeled as Qi = Ki (ui ) |Pi | sgn(Pi ), α (1)

Contract EEC-0540834, www.ccefp.org

where u is the electric current to the EHPV, K(u) is the valve flow conductance, and P is the pressure differential across the valve. Least-squares fitting of (1) to steady state

253

PA Plant M N PB

QA d

PR

PS QS M

K4 K2

K1 K3

QB

Fig. 7: Hydraulic circuit for hardware-in-the-loop load emulation

measurements give α = 0.37 and the map K(u). A secondorder fit to experimental frequency response data shows the valve dynamics are approximately exp(−.05s) K = . Kcmd (s/55 − 1)(s/12 − 1) (2)

Owing to realistic uncertainties, the model (2) is combined ¯ with feedback control to accurately control P to the value P ˙ ¯ + δKdt where by varying the valve command as K = K ¯ ¯ ¯ K = Q/(P )α ¯ ˙ = G − P α+1 (δP )2 δK ¯ αQ ¯ with Q the estimated flow through the valve and δP = ¯ P − Pmeasured . Q is estimated from the commanded plant displacement d and the pump workport pressure. The gain G is selected based on the dynamics (2). The workport pressure PA incident on the plant is controlled with the relationship PA = P1 + PR , where PR = 3 MPa is the (roughly constant) charge pressure of both the HIL simulation and the excavator being emulated. Similarly, the pressure PB is related to the controllable value P2 : PB = PS − P2 where the pressure P2 is held at 2 MPa and the supply pressure, PS , is modulated by a fast electrically controlled variable relief valve to cause PB to track its reference. The flow QS is controlled (in open loop) so that QS = Q + QM , where QM is a margin of flow (3 lpm) over the relief valve to prevent cavitation and maintain the pressure level PS . Time response data for several step changes in reference pressure are shown in Fig. 8 and frequency response data is shown in Fig. 9. VII. P ERFORMANCE The HIL-ROI simulation process can be adequately described by the following steps. 1) Allow HIL system to reach steady state at the simulation initial conditions

2) HIL system tracks newest reference pressures PA,B 3) Plant is given displacement command, dcmd , based on operator command at the ROI system 4) Measure actual plant response, Q 5) Send measured Q to ROI. Use Q to integrate model ˙ ˙ states (e.g. PA,B , joint angles θ, etc.). Use updated model states to update graphics, audio feedback, haptic feedback, etc. 6) Send updated pressures to HIL. Return to Step 2. Note that Steps 2 to 6 may not necessarily occur serially, e.g. TCP/IP data packets may be lost and the many computers may run with differing loop execution rates. The performance capabilities of the HIL system can be understood by considering the ‘dynamics’ of each of the steps. The HIL pressure tracking capabilities needed in Step 2 are illustrated by the step response and frequency response plots in Fig. 8 and 9, respectively. This performance is limited by several parameters including synchronization delays of approximately 0.015 s associated with the local HIL CANbus network, finite EHPV dynamics (2) and pressure-rise dynamics associated with the bulk modulus and conduit volume. The flow meter needed in Step 4 has a response time near 0.01 s. The roundtrip communication time over TCP/IP during Steps 5 and 6 averaged 0.06 s during tests. The limitations discussed above help elucidate the potential usefulness of HIL simulation concurrent with virtual reality operator interface evaluations. In particular, the HIL simulation is useful when the plant dynamics dominate the dynamics of each system previously discussed. The signals representing a portion of a dig cycle as controlled by an operator in the ROI station are shown in Fig. 10. The difference between the flow command sent by the operator at the ROI station and the flow produced and measured by the HIL testbed represent the dynamics of the plant. VIII. C ONCLUSION A testbed suitable for evaluating operator interface designs and machine control algorithms in a convenient way has been developed. The validity of forthcoming evaluations is improved by keeping the actual human operator in the control loop, displaying a dynamically-equivalent model to the operator, and utilizing HIL simulation to ensure the critical dynamics of certain components are displayed correctly. The HIL testbed has capabilities to emulate loads on the plant with a bandwidth up to 4 Hz (for controlling PA ) or 15 Hz (for PB ) over a wide range of flows. Future work will discuss the excavator and soil dynamic models further, and will present more detailed analysis of the effects that communication delays and non-synchronization of models have on the HIL-ROI system. R EFERENCES
[1] N. I. Durlach, and A. S. Mavor, Virtual Reality: Scientific and technological challenges. National Academy Press, 1995. [2] S. P. DiMaio, S.E. Salcudean, C. Reboulet, S.Tafazoli, and K. Hashtrudi-Zaad, “A virtual excavator for controller development and evaluation,” in Proceedings of the 1998 IEEE International Conference on Robotics and Automation, Leuven, Belgium, 1998, pp. 52-58.

254

p

PA: 6 to 12 PB: 14 Normalized pressure response for Normalized Pressure Response steps in PA command

PA: 12 to 6 PB: 14

PA: 10 to 18 PB: 8

PA: 18 to 10 PB: 8

Plant Inputs PA [Mpa] Pa [MPa]

30 20 10 0 30 20 10 0 0 40 20 0 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9

1 0.8 0.6 0.4 0.2 0 0.9 1 1.1 1.2 1.3 1.4 1.5 Time [s]
PA: 8 PB: 10 to 18

1.6 1.7

1.8 1.9

2

Plant Response Q [lpm] Q [lpm]

Pb [MPa] PB [Mpa]

-20 0

Time[s] time [s]
PA: 8 PB: 18 to 10

PA: 14 PB: 6 to 12 Normalized pressure response for Normalized pressure response steps in PB command

PA: 14 PB: 12 to 6

1 0.8 0.6 0.4 0.2 0 0.9

Fig. 10: Slewing the excavator cab 45 degrees counterclockwise. Commands from ROI station are shown in dashed lines and the HIL response in solid lines. The dashed line in the Plant Response plot equals the commanded pump flow based on the slew velocity command from the operator.

1

1.1 1.2 Time [s]

1.3

1.4

Fig. 8: Normalized response for step changes in pressure command for PA (top) and PB (bottom). The responses are normalized such that 0 corresponds to the pressure command just before the step and 1 corresponds to the command just after the step, e.g. PA : 12 to 6; PB : 14 denotes a step from PA = 12 MPa to PA = 6 MPa with PB = 14 MPa. The step occurs at 1 s and QA = 60 lpm. Observe the discretization effects due to transmission of pressure measurements over CANbus.

PA Magnitude [dB]
0 -10 -20 -30
-1

PB

10

10

0

10

1

0

-50

-100

10

-1

10

0

10

1

Frequency [Hz]

Fig. 9: The steady-state frequency response for controlling PA and PB . QA = 60 lpm. The pressure being varied was given a sinusoidal reference of amplitude 1.5 MPa with an offset of 18 MPa; the other pressure was set to 8 MPa.

[3] H. I. Torres-Rodriguez, V Parra-Vega, and F.J. Ruiz-Sanchez, “Integration of force-position control and haptic interface facilities for a virtual excavator simulator,” in Proceedings of the 2005 International Conference on Advanced Robotics, Seattle, 2005, pp. 761-768. [4] M. E. Kontz, “Haptic control of hydraulic machinery using proportional valves.” Ph.D. dissertation, The Georgia Institute of Technology, G.W. Woodruff School of Mechanical Engineering, Atlanta, 2007. [5] D. W. Johnson, G. H. Lovell, and J. J. Murray, “Development of a coordinated motion controller for a front shovel excavator,” in Proceedings of the Seventh Topical Meeting on Robotics and Remote Systems, Augusta, GA, pp. 239-246, 1997. [6] P. D. Lawrence, S. E. Salcudean, et al. “Coordinated and forcefeedback control of hydraulic excavators,” in Proceedings of the International Symposium on Experimental Robotics, Stanford, CA, pp. 181-194, 1995. [7] S. E. Salcudean, K. Hashtrudi-Zaad, S. Tafazoli, S. P. DiMaio, and C. Reboulet, “Bilateral matched impedance teleoperation with applications to excavator control,” IEEE Control Systems Magazine, vol. 19, pp. 966-971, Dec 1999. [8] A.R. Enes, W. J. Book, “A hardware-in-the-loop simulation testbed For emulating hydraulic loads representing the complete dig cycle of a construction machine,” in Proceedings of the ASME International Mechanical Engineering Congress and Exposition, Boston, 2008. [9] J. L. Lahti, S. J. Andrasko, and J. J. Moskwa, “A transient hydrostatic dynamometer for single cylinder engine research with real time multi-cylinder dynamic simulation,” American Society of Mechanical Engineers, The Fluid Power and Systems Technology Division FPST, pp. 223-230, 2003. [10] A. J. C. Ramden, J. O. Palmberg, “Design and analysis of a load simulation for testing hydraulic valves,” in Fluid Power Engineering: Challenges and Solutions, Tenth Bath International Fluid Power Workshop, C. R. Burrows and K. A. Edge, Ed. England: Research Studies Press Ltd., 1998. [11] R. Zhang, D. E. Carter, and A. G. Alleyne, “Multivariable Control of an Earthmoving Vehicle Powertrain Experimentally Validated in an Emulated Working Cycle,” Proceedings of the ASME International Mechanical Engineering Congress and Exposition, Washington, DC, 2003. [12] H. E. Merrit, Hydraulic Control Systems, New York: Wiley, 1967. [13] F. Malaguti, “Soil machine interaction in digging and earthmoving automation,” in International Symposium on Automation and Robotics in Construction, Brighton, UK, pp. 187-191, 1994. [14] S.P. DiMaio, S. E. Salcudean, and C. Reboulet, “A virtual environment for the simulation and programming of excavator trajectories,” in Presence, vol. 10, no. 5, Cambridge: MIT Press, pp. 465-476, 2001.

Phase [degrees]

255