Machine vision system can be applied in real-time to precisely measure visual properties such as color, length, angle, position, orientation, etc. One advantage of machine vision system is the ability to perform contact-free measurement, which is important especially when contact is difficult. However, one of the difficulties in designing a machine vision system is how to be robust to noises present in an image, to scene disturbances caused by unwanted background and foreground objects, as well as to different illumination conditions. Various machine vision approaches had been applied to visual servo control  . Visual servo system must be capable of tracking image features in a sequence of images. Visual object tracking is used to identify the trajectory of moving objects in video frame sequences. Object tracking involves intensive computation in order to extract real-time information from high-volume video data with low power consumption. Real-time image processing tasks require not only high computing power but also high data bandwidth. Thus, prototypical FPGA based real-time image processing systems for computer vision and visual servo applications have been proposed    . An application can usually be divided into a pre- processing part located on a dedicated subsystem and a high-level part located on a host system. For instance, the pre-processing system is built on FPGA chips and the high-level part for robot vision applications runs on a PCI-based PC system  . In particular, an FPGA implementation of a multi-microrobot visual tracking system using CMOS vision sensors had been shown to achieve a 0.01 mm accuracy  . Their system was capable of tracking a single robot at a frequency of over 200 Hz. The motivation of this work is to develop a real-time machine vision feedback based robot motion ball-beam balance control system on FPGA chips.
Ball-and-beam control systems are well-known apparatus designed for demonstrating position control capabilities of unstable open-loop systems. Many nonlinear/linear control methods as well as system stability analysis of ball-beam systems have been presented in literatures    . In particular, for a non- linear ball-beam system model, a simple PD controller with asymptotic stability has been theoretically proven and experimentally tested  . However, accurately determining the beam angle and ball position in real-time is an important task. There are many approaches to applying beam inclination and ball position sensors; visual feedback is a common solution    . However, there are only few works for applying visual feedback to both beam inclination and ball position measurements. Moreover, there are also few works on the integration of a full-scale robot manipulator and vision feedback system in a ball-beam or a ball-plate balancing control system.
Other related works include ball-beam balance systems with feedbacks from a tablet  and from a smartphone  , a robotic soccer beam balance system  , and a ball-and-plate balance system  . A ball-beam control system with visual feedback sensor from a smart tablet had been designed in  . Additionally, augmented reality was integrated into an interactive touchscreen interface. A ball-beam system with wireless sensors on a smartphone had been demonstrated in  . A smartphone’s inertial and camera sensors were used to measure the angular orientation/velocity of the beam and translational ball position, and a local feedback linearizing controller was used to stabilize the ball-beam control system. Ryu and Oh had experimented with a soccer-and-beam balance system installed on the top of the end-effector of a redundant manipulator  . A force-torque sensor was attached to the end-effector to estimate the ball position through differentiation. Cheng and Tsai had designed a ball-and-plate balance system, which was attached as the end effector and maneuver by a two-DOF robotic wrist  . The ball’s position was captured by a PC-based video camera system and the orientation of the plate was controlled by a LQR algorithm.
Up to now, we have not observed that an automatic ball-beam balance system by a robot manipulator with more than 3-axes. This paper attempts to design an automatic robotic ball-beam balance system using an integrated machine vision system. The main goal of this machine vision based robot motion control system is to measure the ball position and beam angle in real-time, and to balance the ball at any desired point on the beam through the motion of a 5-axes robotic end-effector. This works also focuses on an entirely FPGA based control implementation with input from a CMOS image sensor.
2. Robotic Ball-Beam Control System
The robotic ball-beam control system setup is shown in Figure 1. The end-ef- fector of a 5-axes robot manipulator is connected to the left end of the beam which can be freely rotated about its center pivot. A blue ball is allowed to roll freely along the beam. Both the ball’s position and beam angle are measured by a real-time image processing system with input from a CMOS image sensor at a rate of 50 frame-per-second. The overall control system is implemented on two FPGA development boards, one for real-time image processing and the other one for all required control functions, including ball balance control and end- effector position control.
The ball-beam system is a two-dimensional system on the x-z plane and hence only three axes of the 5-axis robot manipulator, , and , are under independent-joint set-point PID control, the other two axes and are stationary and set to zero angle position. The robot’s end-effector is attached to the left-end of the beam, as shown in Figure 2. The ball-beam system itself is a sin-
Figure 1. The control system set-up of the vision feedback robotic ball- beam balance system.
Figure 2. The robot end-effector automatically balances the ball at any desired position on the x-z plane, where is the reference position.
gle-input (beam angle) and single-output (ball position) system.
To have the robot’s end-effector automatically drive and balance the ball at any position, a ball position PD controller and independent robot joint PID controllers play major parts in the overall control system as shown in Figure 3. Let and be the ball current position and target goal position, respectively. A simple discrete PD control law,
is applied to guide the robotic end-effector’s motion, where is the ball balance position error, and the control sampling time is Tc = 0.02 sec. The robotic end-effector’s motion command position and orientation are then set to
where R is half of the beam length as shown in Figure 2. Solving the inverse kinematic Equation (2), robot joint goal position commands , and are then obtained. Incremental digital PID control law,
, , (3)
are used to independently control the position of each robot joints. The servo control sampling time is Ts = 0.001 seconds.
3. Ball-Beam System Image Processing
Let be the image coordinate of the beam’s pivot joint, be the image coordinate of the blue ball, and be the image coordinate of the point where is the lowest pixel position of the beam along the vertical line , as shown in Figure 4. A blue ball is used here in order to simplify ball detection and to improve measurement accuracy. Let and be the image feedback output variables. When beam angle is small, we have , where L is a constant length; therefore, the ball position and beam angle can be expressed as
Figure 3. The system block diagram of the robotic ball-beam control system.
where scaling constant s = 1.5 mm/pixel and k = 0.24 deg/pixel.
The proposed ball-beam system visual feedback is built on the Altera DE2 FPGA development board with input from a CMOS image sensor camera (TRDB-D5M). The image processing module captures real-time images with a 640 × 480 resolution and at 50 fps. Figure 5 shows the functional block diagram of image processing module, in which G is the G-component of the RGB color image. The processing pipeline includes color space transformation, Sobel edge detection, binary threshold, erosion/dilation, and target object position detection.
Real-time image input is fed to the FPGA chip row by row. Up to 5 rows of an image are stored in FIFO line-buffers in a pipelined fashion. The image processing unit is clocked at 96 MHz. The raw image data is captured by a color
Figure 4. Ball-beam visual sensor system, where L depends on the distance between the camera and the beam.
Figure 5. The image processing pipeline.
camera with resolution at 50 fps (frames per second). The raw image is converted to a 24-bit RGB image (by down sampling of every scanned pixels). The center position of the blue ball is obtained as follows. First, RGB to color space transformation is performed : , where and . Color space transformation is followed by binary segmentation for the component with a threshold value of 130. An erosion operation in a window is performed next, followed by a dilation operation in a window. True pixels indicate potential ball position. Finally, the center of blue ball is calculated from the center of the true pixels. Beam angle detection is processed in a similar way. Since the G-component of the RGB color image is similar to the gray image. First, Sobel edge detection is applied to the G-component of the original RGB image with a threshold value of 384, followed by a dilation operation in window. For all pixels, is the lowest pixel point of the beam in column ( ) such that . Figure 6 shows the ball-beam visual feedback processing experimental results.
4. Experimental Results
4.1. Robot Joints Set-Point Control Experiments
The robot manipulator is running in set-point position control mode, in which each joint is under independent joint PID servo control. The digital PID control is implemented in relative form
Figure 6. Experimental results of the ball-beam visual feedback processing.
where , , is the current position error of joint i. PID gains for joint 2, 3 and 4 are , , ( -axis), , , ( -axis) and , , ( -axis). Figure 7 shows robot manipulator set-point control experiments for joints , and .
4.2. Ball-Beam PD Controller Tuning Experiment
To estimate the system transfer function and design the controller of the ball- beam system, a manual balance control test is performed. A human attaches his hand to the left-end of the beam and balances the ball at the center pivot joint position. Figure 8 shows the ball position and beam angle trajectories sampled at an interval 0.02 seconds. One can apply MATLAB’s ident system identification function with input/output data as shown in Figure 7, the system transfer function of the ball-beam system is estimated to be
Having the ball-beam system transfer function , one can then use MATLAB’s pidtool as shown in Figure 9 to tune a continuous-time PD controller based on Eq. (1), and obtains
Then a digital PD controller is implemented as shown below
Figure 7. Set-point control experimental results for robot joints , and .
Figure 8. Manual ball-beam balance control test trajectories.
Figure 9. Tuning the ball-beam PD controller with MATLAB’s pidtool.
where , and
4.3. Robotic Ball-Beam Balance Control Experiments
Several robotic ball-beam balance control experiments are performed for different ball target positions: center, right-side, left-side and multi-positions, as shown in Figures 10-13, respectively. The control performance for the center target position as shown in Figure 10 can be summarized as follows: rise time is 4.0 sec., settling time 8.0 sec., overshot 25% and steady-state error 2%. When compared with Figure 8, one can see that the control performance of PD controller is much better than that of a human in terms of less rise/settle, smaller overshoot, and less steady-state error. The control performance for the right-side target position as shown in Figure 11 can be summarized as follows: rise time 4.0 sec., settling time 7.0 sec., overshot 2% and steady-state error 1%. The control performance for the left-side target position as shown in Figure 12 can be summarized as follows: rise time 4.0 sec., settling time 8.0 sec., overshot 20% and steady-state error 1%. In summary, the robotic ball-beam balance control has less overshoot especially for smaller displacement and when the ball is rolling from left to right as shown from Figure 13.
In this work, we have implemented a visual feedback based robotic motion control system on two FPGA chips with efficient parallel architecture. We have experimentally verified that a robotic motion control system with an integrated
Figure 10. Ball-beam balance control experiment 1: center position.
Figure 11. Ball-beam balance control experiment 2: right-side position.
visual feedback system can automatically balance a ball-beam system with the end-effector of a 5-axis manipulator. This machine vision based robotic ball- beam set-point control system is able to balance the ball at any desired position on the beam efficiently in real-time. Although the proposed system is almost as good as a naive user manually balancing the ball to stay at any desired position,
Figure 12. Ball-beam balance control experiment 3: left-side position.
Figure 13. Ball-beam balance control experiment 3: left-side position.
the limitations of the current system are lower sampling time in visual control feedback and smaller angle range of the bean angle.
This work is supported by Taiwan Ministry of Science and Technology Grant MOST 105-2221-E-011-047.
 Jorg, S., Langwald, J. and Nickl, M. (2004) FPGA Based Real-Time Visual Sevoing. Proceedings of the 17th International Conference on Pattern Recognition, Cambridge, 26-26 August 2004, 749-753.
 Diederichs, C. (2011) Fast Visual Servoing of Multiple Microrobots Using an FPGA-Based Smart Camera System. IFAC Proceedings Volumes, 44, 14636-14641.
 Garc, G., Jara, C., Pomares, J., Alabda, A., Poggi L. and Torres, F. (2014) A Survey on FPGA-Based Sensor Systems: Towards Intelligent and Reconfigurable Low-Power Sensors for Computer Vision, Control and Signal Processing. Sensors, 14, 6247-6278.
 Hauser, J., Sastry, S. and Kokotovic, P. (1992) Nonlinear Control via Approximate Input-Output Linearization: The Ball and Beam Example. IEEE Transactions on Automatic Control, 37, 392-398.
 Petrovic, I., Brezak, M. and Cupec, R. (2002) Machine Vision Based Control of the Ball and Beam. IEEE 7th International Workshop on Advanced Motion Control, Maribor, 3-5 July 2002, 573-577.
 Ho, C. and Shih, C. (2008) Machine Vision Based Tracking Control of Ball Beam System. Key Engineering Materials, 381-382, 301-304.
 Frank, J., Gomez, J. and Vikram, K.V. (2015) Using Tablets in the Vision-Based Control of Ball and Beam Test-Bed. Proceedings of 12th International Conference on Informatics in Control, Automation and Robotics, Alsace, 21-23 July 2015, 92-102.
 Brill, A., Frank, J. and Kapola, V. (2016) Using Inertial and Visual Sensing from a Mounted Smartphone to Stabilize a Ball and Beam Test-Bed. Proceedings of American Control Conference, Boston, 6-8 July 2016, 1335-1340.
 Ryu, K. and Oh, Y. (2011) Balance Control Of Ball-Beam System Using Redundant Manipulator. Proceedings of the 2011 IEEE International Conference on Mechatronics, Beijing, 7-10 August 2011, 403-408.