In this paper, a mobile assistance-system is described which supports users in performing manual working tasks in the context of assembling complex products. The assistance system contains a head-worn display for the visualization of information relevant for the workflow as well as a video camera to acquire the scene. This paper is focused on the interaction of the user with this system and describes work in progress and initial results from an industrial application scenario. We present image-based methods for robust recognition of static and dynamic hand gestures in realtime. These methods are used for an intuitive interaction with the assistance-system. The segmentation of the hand based on color information builds the basis of feature extraction for static and dynamic gestures. For the static gestures, the activation of particular sensitive regions in the camera image by the user’s hand is used for interaction. An HMM classifier is used to extract dynamic gestures depending on motion parameters determined based on the optical flow in the camera image.
 Schwerdtfeger, B., Reif, R., Gunthner, W.A., Klinker, G., Hamacher, D., Schega, L., Bockelmann, I., Doil, F. and Tümler, J. (2009) Pick-by-Vision: A First Stress Test. 8th IEEE International Symposium on Mixed and Augmented Reality, Orlando, 19-22 October 2009, 115-124.
 Conradi, J. and Alexander, T. (2008) Display Devices for Virtual Environments: Impact on Performance, Workload, and Simulator Sickness. EGVE 2008 Symposium Eindhoven, The Netherlands, 29-30 May 2008, 103-109.
 Smith, S.P. and Hart, J. (2006) Evaluating Distributed Cognitive Resources for Way Finding in a Desktop Virtual Environment. Symposium on 3D User Interfaces (3DUI), Virginia, 25-29 March 2006, 3-10.
 Gabbard, J.L., Swan, J.E., Hix, D., Kim, S.-J. and Fitch, G. (2007) Active Text Drawing Styles for Outdoor Augmented Reality: A User-Based Study and Design Implications. Virtual Reality Conference (VR), Charlotte, 10-14 March 2007, 35-42.
 Tümler, J., Doil, F., Mecke, R., Paul, G., Schenk, M., Pfister, E.A., Huckauf, A., Bockelmann, I. and Roggentin, A. (2008) Mobile Augmented Reality in Industrial Applications: Approaches for Solution of User-Related Issues. 7th IEEE/ACM International Symposium on Mixed and Augmented Reality, Cambridge, 15-18 September 2008, 87-90.
 Park, M., Schmidt, L., Schlick, C. and Luczak, H. (2007) Design and Evaluation of an Augmented Reality Welding Helmet. Human Factors and Ergonomics in Manufacturing & Service Industries, 17, 317-330.
 Tümler, J., Scharf, C., Mecke, R.M., Schenk, M. and Paul, G. (2008) Incorporating User Preference to Represent Information for Manual Work Supported by Augmented Reality. SVR 2008 Symposium, João Pessoa, 13-16 May 2008, 196-203.
 Fröhlich, B., Hochstrate, J., Skuk, V. and Huckauf, A. (2006) The Globefish and the Globemouse: Two New Six Degree of Freedom Input Devices for Graphics Applications. ACM CHI Conference on Human Factors in Computing Systems, Montréal, 22-27 April 2006, 191-199.
 Cheddad, A., Condell, J., Curran, K. and Kevitt, P.M. (2009) A New Color Space for Skin Tone Detection. 16th IEEE Internation Conference on Image Processing (ICIP), Cairo, 7-10 November 2009, 497-500.
 Khan, R., Hanbury, A., Stöttinger, J. and Bais, A. (2012) Color Based Skin Classification. Pattern Recognition Letters, 33, 157-163. http://dx.doi.org/10.1016/j.patrec.2011.09.032
 Rashid, O., Al-Hamadi, A., Panning, A. and Michaelis, B. (2009) Posture Recognition Using Combined Statistical and Geometrical Feature Vectors Based on SVM. International Conference on Image, Signal and Vision Computing (ICISVC), London, 26-28 August 2009, 590-597.
 Suk, H.I., Sin, B.K. and Lee, S.W. (2010) Hand Gesture Recognition Based on Dynamic Bayesian Network Framework. Pattern Recognition, 43, 3059-3072.
 Lange, R., Dürr, F. and Rothermel, K. (2011) Efficient Real-Time Trajectory Tracking. The VLDB Journal, 20, 671-694. http://dx.doi.org/10.1007/s00778-011-0237-7
 Rekha, J., Bhattacharya, J. and Majumder, S. (2011) Hand Gesture Recognition for Sign Language: A New Hybrid Approach. The 2011 International Conference on Image Processing, Computer Vision, and Pattern Recognition (IPCV), Las Vegas, 18-21 July 2011, 80-86.
 Wagner, D., Langlotz, T. and Schmalstieg, D. (2008) Robust and Unobtrusive Marker Tracking on Mobile Phones. Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality, Cambridge, 15-18 September 2008, 121-124.
 Fuhrmann, A., Schmalstieg, D. and Purgathofer, W. (1999) Fast Calibration for Augmented Reality. Proceedings of the ACM Symposium on Virtual Reality Software and Technology, London, 20-22 December 1999, 166-167.
 Kato, H. and Billinghurst, M. (1999) Marker Tracking and HMD Calibration for a Video-Based Augmented Reality Conferencing System. 2nd IEEE and ACM International Workshop on Augmented Reality, San Francisco, 85-94.
 Tuceryan, M. and Navab, N. (2000) Single Point Active Alignment Method (SPAAM) for Optical See-Through HMD Calibration for AR. IEEE and ACM International Symposium on Augmented Reality, Munich, 5-6 October 2000, 149-158.
 Lam, E.Y. (2005) Combining Gray World and Retinex Theory for Automatic White Balancein Digital Photography. 9th International Symposium on Consumer Electronics (ISCE), Macau, 14-16 June 2005, 134-139.
 Phung, S.L., Bouzerdoum, A. and Chai, D. (2005) Skin Segmentation Using Color Pixel Classification: Analysis and Comparison. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27, 148-154.
 Rabiner, L.R. (1990) A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition. In: Waibel, A. and Lee, K.F., Eds., Readings in Speech Recognition, Morgan Kaufmann Publishers Inc., San Francisco, 267-296.