DFKI and Hitachi Jointly Develop AI Technology for Human Activity Recognition of Workers Using Wearable Devices

Deutsches Forschungszentrum für Künstliche Intelligenz (German Research Center for Artificial Intelligence, “DFKI“) and Hitachi, Ltd. announced the joint development of AI (artificial intelligence) technology for human activity recognition of workers using wearable devices.  The AI technology performs real-time recognition of workers' activities by integrating technology in eye-tracking glasses to recognize gazed objects with technology in armband devices to recognize action.  The recognition ability of each activity is achieved by having the AI understand the tools or parts used at the production site, as well as anticipated actions through Deep Learning.  DFKI and Hitachi will use this newly developed AI technology to help assist operations and prevent human error, and to help contribute to enhancing quality and efficiency on the front line of manufacturing.

In line with initiatives, such as Industrie 4.0 in Germany and Society 5.0 in Japan, the manufacturing industry is accelerating steps towards innovating production using AI and robotics, and the automation of menial tasks.  At the same time, IIoT technology is being called for to collect and recognize the condition or movement of all things, including people and equipment, to assist in operations, and prevent human error.  As a result, in recent years, monitoring systems using cameras have been developed for predictive diagnosis of inappropriate worker movement or equipment failure in production lines.

Researchers from the DFKI research department, Smart Data & Knowledge Services, and Hitachi developed AI for human activity recognition to recognize the activity of workers using various data collected through the wearable devices, not image data from cameras.  Features of the AI that has been developed include:

1. Technology to recognize gazed objects by using eye-tracking glasses

This technology is to recognize and distinguish between targeted objects, such as, for example a “screw” versus a “screwdriver” without being disturbed by its surrounding environment, such as background or other objects.  This technology extracts the data of gaze points from the movements of eyeballs of workers who wear the eye-tracking glasses and utilizes the image recognition technology by Deep Learning.

2. Technology to recognize basic human actions through an armband device

This technology is to recognize basic human actions that require arms movements, such as, for example, a “twist” or a “push”.  This technology extracts the data relating to body actions from the microscopic and instantaneous signals that are measured by sensors attached to the arms.

3. “Hierarchical activity-recognition model” that recognizes workers' activities by integrating gazed objects and human actions

This integrates the technologies to recognize gazed objects and recognize basic human actions to develop a “hierarchical activity-recognition model”, which recognizes activities, such as, for example, “twisting a screw.”  As a result, recognizing a variety of worker activities is possible if all the actions and objects involved in the activities are learned in advance.

Based upon these developments, the AI technology that can recognize activities, such as, for example, “twisting a screw” or “pressing a switch”, as part of, for example, an “inspection task” in real-time, was realized.  DFKI and Hitachi will help to promote these technological developments for assisting operations and preventing human error on the front line of manufacturing, where operation guidance and inadequate action detection are required, by utilizing this newly developed AI.

DFKI and Hitachi will exhibit part of this technology at “CeBIT 2017”, an exhibition of digital business to be held from March 20-24, 2017 in Hannover, Germany.

Keywords: Artificial Intelligence (AI), Wearable Devices, Industry 4.0, Society 5.0, Manufacturing Industry, Predictive Diagnosis, ARC Advisory Group.