A review on multi-modal human motion representation recognition and its application in orthopedic rehabilitation training_Journal of Biomedical Engineering

Authors：

XING Mengmeng , WEI Guohui , LIU Jing , ZHANG Junzhong , YANG Feng ,  CAO Hui

School of Science and technology, Shandong University of Traditional Chinese Medicine, Jinan 250355, P.R.China;

Corresponding?author：

Keywords：

action capture; action feature extraction; deep learning; action recognition; orthopedic rehabilitation training

DOI：

10.7507/1001-5515.201906053

Video：

Export PDF Favorites Scan Get Citation

Abstract Full text Figures/Tables Video References Cited by

Human motion recognition (HAR) is the technological base of intelligent medical treatment, sports training, video monitoring and many other fields, and it has been widely concerned by all walks of life. This paper summarized the progress and significance of HAR research, which includes two processes: action capture and action classification based on deep learning. Firstly, the paper introduced in detail three mainstream methods of action capture: video-based, depth camera-based and inertial sensor-based. The commonly used action data sets were also listed. Secondly, the realization of HAR based on deep learning was described in two aspects, including automatic feature extraction and multi-modal feature fusion. The realization of training monitoring and simulative training with HAR in orthopedic rehabilitation training was also introduced. Finally, it discussed precise motion capture and multi-modal feature fusion of HAR, as well as the key points and difficulties of HAR application in orthopedic rehabilitation training. This article summarized the above contents to quickly guide researchers to understand the current status of HAR research and its application in orthopedic rehabilitation training.

Citation： XING Mengmeng, WEI Guohui, LIU Jing, ZHANG Junzhong, YANG Feng, CAO Hui. A review on multi-modal human motion representation recognition and its application in orthopedic rehabilitation training. Journal of Biomedical Engineering, 2020, 37(1): 174-178, 184. doi: 10.7507/1001-5515.201906053 Copy

1.	Guerra J, Uddin J, Nilsen D, et al. Capture, learning, and classification of upper extremity movement primitives in healthy controls and stroke patients//2017 International Conference on Rehabilitation Robotics (ICORR).lodon, UK: IEEE, 2017: 547-554.
2.	Wang Yufan, Zhao Yuliang, Chan R H, et al. Volleyball skill assessment using a single wearable micro inertial measurement unit at wrist. IEEE Access, 2018, 6: 13758-13765.
3.	潘月浩, 宋執環, 杜往澤, 等. 適用于老年公寓的動作識別設計方法. 電子測量與儀器學報, 2017, 31(1): 29-35.
4.	金剛, 李德華, 周學泳. 表演動畫中的運動捕捉技術. 中國圖象圖形學報, 2000, 5(3): 88-91.
5.	朱紅蕾, 朱昶勝, 徐志剛. 人體行為識別數據集研究進展. 自動化學報, 2018, 44(06): 978-1004.
6.	Alt Murphy M, Murphy S, Persson H C, et al. Kinematic analysis using 3D motion capture of drinking task in People with and without upper-extremity impairments. J Vis Exp, 2018(133): e57228.
7.	Callejas-Cuervo M, Gutierrez R M, Hernandez A I. Joint amplitude MEMS based measurement platform for low cost and high accessibility telerehabilitation: elbow case study. J Bodyw Mov Ther, 2017, 21(3): 574-581.
8.	Palumbo F, Gallicchio C, Pucci R, et al. Human activity recognition using multisensor data fusion based on reservoir computing. J Ambient Intell Smart Environ, 2016, 8(2): 87-107.
9.	Ryoo M S, Chen C C, Aggarwal J K, et al. An overview of contest on semantic description of human activities (SDHA) 2010//2010 International Conference on Pattern Recognition (ICPP).Berlin, Germany, 2010: 270-285.
10.	Wang Jiang, Liu Zicheng, Wu Ying, et al. Mining actionlet ensemble for action recognition with depth cameras//2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).RI, USA: IEEE, 2012: 1290-1297.
11.	Chen C, Jafari R, Kehtarnavaz N. UTD-MHAD: a multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor/2015 IEEE International Conference on Image Processing(ICIP).QC, Canada: IEEE, 2015: 168-172.
12.	王劍, 李春雨, 李躍新. 結合區域HOF和字典學習的人體行為識別方法. 計算機應用研究, 2017, 34(9): 2863-2866.
13.	肖玉玲. 結合HOG/HOF級聯特征和多層分類器的人體行為識別. 計算機工程與設計, 2017, 38(9): 2567-2572.
14.	Yao Hongxian, Jiang Xinghao, Sun Tanfeng, et al. 3D human action recognition based on the spatial-temporal moving skeleton descriptor//2017 IEEE International Conference on Multimedia and Expo (ICME).Shenzhen, China: IEEE, 2017: 937-942.
15.	Shih T Y, Hsieh Y W, WU C Y, et al. Quantifying kinematic features of upper-extremity movement strategies during tooth brushing in patients with stroke and healthy controls. J Neurol Sci, 2017, 381: 995.
16.	Hassan M M, Uddin M Z, Mohamed A A. A robust human activity recognition system using smartphone sensors and deep learning. Future Generation Computer Systems, 2018, 81: 307-313.
17.	胡婷鴻, 萬雷, 劉太昂, 等. 深度學習在圖像識別及骨齡評估中的優勢及應用前景. 法醫學雜志, 2017, 33(6): 629-634, 639.
18.	Nweke H F, Teh Y W, Al-garadi M A, et al. Deep learning algorithms for human activity recognition using Mobile and wearable sensor networks: state of the art and research challenges. Expert Syst Appl, 2018, 105: 233-261.
19.	張瑞, 李其申, 儲珺. 基于3D卷積神經網絡的人體動作識別算法. 計算機工程, 2019, 45(01): 259-263.
20.	Pham H H, Khoudour L, Crouzil A, et al. Exploiting deep residual networks for human action recognition from skeletal data. Computer Vision and Image Understanding, 2018, 170:51-66.
21.	Li Honggui, Trocan M. Deep learning of smartphone sensor data for personal health assistance. Microelectronics J, 2019, 88: 164-172.
22.	胡雙杰, 秦建邦, 郭薇. 基于特征自動提取的跌倒檢測算法. 傳感技術學報, 2018, 31(12): 1842-1847.
23.	裴曉敏, 范慧杰, 唐延東. 時空特征融合深度學習網絡人體行為識別方法. 紅外與激光工程, 2018, 47(02): 55-60.
24.	Eitel A, Springenberg J T, Spinello L, et al. Multimodal deep learning for robust RGB-D object recognition//2015 IEEE International Conference on Intelligent Robots and Systems (IROS). Hamburg, Germany: IEEE, 2015: 681-687.
25.	Kim J H, Hong G S, Kim B G, et al. DeepGesture: deep learning-based gesture recognition scheme using motion sensors. Displays, 2018, 55: 38-45.
26.	李玉鵬, 劉婷婷, 張良. 基于深度學習的人體動作識別方法. 計算機應用研究. 2019, 37(1). DOI:10.19734/j.issn.1001-3695.2018.05.0499.
27.	羅會蘭, 童康, 孔繁勝. 基于深度學習的視頻中人體動作識別進展綜述. 電子學報, 2019, 47(05): 1162-1173.
28.	Fernandez-Cervantes V, Neubauer N, Hunter B, et al. VirtualGym: A kinect-based system for seniors exercising at home. Entertain Comput, 2018, 27: 60-72.
29.	陳朝暉, 張文娣. 理筋正骨手法結合運動訓練治療慢性頸痛的臨床觀察. 中國中西醫結合雜志, 2018, 38(11): 1327-1331.
30.	盛偉, 林敏. 中醫正骨手法聯合康復訓練改善頸型頸椎病患者CASCS評分、Berg平衡功能和疼痛程度效果觀察. 時珍國醫國藥, 2018, 29(06): 1391-1393.
31.	馮敏山, 高景華, 高春雨, 等. 坐位腰椎旋轉手法的運動學研究. 中國中醫骨傷科雜志, 2018, 26(07): 16-20.
32.	曹慧, 高凡茹, 魏德健, 等. 中醫正骨仿真系統中姿態檢測的應用. 時珍國醫國藥, 2017, 28(12): 3017-3019.
33.	周晨爍, 魏德健, 曹慧, 等. 中醫正骨仿真系統中正骨手法的評價. 中華中醫藥學刊, 2018, 36(08): 1826-1828.
34.	朱立國, 韓濤, 于杰, 等. 中醫骨傷科旋提手法規范化操作傳承模式初探. 中醫雜志, 2018, 59(11): 927-931.

1. Guerra J, Uddin J, Nilsen D, et al. Capture, learning, and classification of upper extremity movement primitives in healthy controls and stroke patients//2017 International Conference on Rehabilitation Robotics (ICORR).lodon, UK: IEEE, 2017: 547-554.
2. Wang Yufan, Zhao Yuliang, Chan R H, et al. Volleyball skill assessment using a single wearable micro inertial measurement unit at wrist. IEEE Access, 2018, 6: 13758-13765.
3. 潘月浩, 宋執環, 杜往澤, 等. 適用于老年公寓的動作識別設計方法. 電子測量與儀器學報, 2017, 31(1): 29-35.
4. 金剛, 李德華, 周學泳. 表演動畫中的運動捕捉技術. 中國圖象圖形學報, 2000, 5(3): 88-91.
5. 朱紅蕾, 朱昶勝, 徐志剛. 人體行為識別數據集研究進展. 自動化學報, 2018, 44(06): 978-1004.
6. Alt Murphy M, Murphy S, Persson H C, et al. Kinematic analysis using 3D motion capture of drinking task in People with and without upper-extremity impairments. J Vis Exp, 2018(133): e57228.
7. Callejas-Cuervo M, Gutierrez R M, Hernandez A I. Joint amplitude MEMS based measurement platform for low cost and high accessibility telerehabilitation: elbow case study. J Bodyw Mov Ther, 2017, 21(3): 574-581.
8. Palumbo F, Gallicchio C, Pucci R, et al. Human activity recognition using multisensor data fusion based on reservoir computing. J Ambient Intell Smart Environ, 2016, 8(2): 87-107.
9. Ryoo M S, Chen C C, Aggarwal J K, et al. An overview of contest on semantic description of human activities (SDHA) 2010//2010 International Conference on Pattern Recognition (ICPP).Berlin, Germany, 2010: 270-285.
10. Wang Jiang, Liu Zicheng, Wu Ying, et al. Mining actionlet ensemble for action recognition with depth cameras//2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).RI, USA: IEEE, 2012: 1290-1297.
11. Chen C, Jafari R, Kehtarnavaz N. UTD-MHAD: a multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor/2015 IEEE International Conference on Image Processing(ICIP).QC, Canada: IEEE, 2015: 168-172.
12. 王劍, 李春雨, 李躍新. 結合區域HOF和字典學習的人體行為識別方法. 計算機應用研究, 2017, 34(9): 2863-2866.
13. 肖玉玲. 結合HOG/HOF級聯特征和多層分類器的人體行為識別. 計算機工程與設計, 2017, 38(9): 2567-2572.
14. Yao Hongxian, Jiang Xinghao, Sun Tanfeng, et al. 3D human action recognition based on the spatial-temporal moving skeleton descriptor//2017 IEEE International Conference on Multimedia and Expo (ICME).Shenzhen, China: IEEE, 2017: 937-942.
15. Shih T Y, Hsieh Y W, WU C Y, et al. Quantifying kinematic features of upper-extremity movement strategies during tooth brushing in patients with stroke and healthy controls. J Neurol Sci, 2017, 381: 995.
16. Hassan M M, Uddin M Z, Mohamed A A. A robust human activity recognition system using smartphone sensors and deep learning. Future Generation Computer Systems, 2018, 81: 307-313.
17. 胡婷鴻, 萬雷, 劉太昂, 等. 深度學習在圖像識別及骨齡評估中的優勢及應用前景. 法醫學雜志, 2017, 33(6): 629-634, 639.
18. Nweke H F, Teh Y W, Al-garadi M A, et al. Deep learning algorithms for human activity recognition using Mobile and wearable sensor networks: state of the art and research challenges. Expert Syst Appl, 2018, 105: 233-261.
19. 張瑞, 李其申, 儲珺. 基于3D卷積神經網絡的人體動作識別算法. 計算機工程, 2019, 45(01): 259-263.
20. Pham H H, Khoudour L, Crouzil A, et al. Exploiting deep residual networks for human action recognition from skeletal data. Computer Vision and Image Understanding, 2018, 170:51-66.
21. Li Honggui, Trocan M. Deep learning of smartphone sensor data for personal health assistance. Microelectronics J, 2019, 88: 164-172.
22. 胡雙杰, 秦建邦, 郭薇. 基于特征自動提取的跌倒檢測算法. 傳感技術學報, 2018, 31(12): 1842-1847.
23. 裴曉敏, 范慧杰, 唐延東. 時空特征融合深度學習網絡人體行為識別方法. 紅外與激光工程, 2018, 47(02): 55-60.
24. Eitel A, Springenberg J T, Spinello L, et al. Multimodal deep learning for robust RGB-D object recognition//2015 IEEE International Conference on Intelligent Robots and Systems (IROS). Hamburg, Germany: IEEE, 2015: 681-687.
25. Kim J H, Hong G S, Kim B G, et al. DeepGesture: deep learning-based gesture recognition scheme using motion sensors. Displays, 2018, 55: 38-45.
26. 李玉鵬, 劉婷婷, 張良. 基于深度學習的人體動作識別方法. 計算機應用研究. 2019, 37(1). DOI:10.19734/j.issn.1001-3695.2018.05.0499.
27. 羅會蘭, 童康, 孔繁勝. 基于深度學習的視頻中人體動作識別進展綜述. 電子學報, 2019, 47(05): 1162-1173.
28. Fernandez-Cervantes V, Neubauer N, Hunter B, et al. VirtualGym: A kinect-based system for seniors exercising at home. Entertain Comput, 2018, 27: 60-72.
29. 陳朝暉, 張文娣. 理筋正骨手法結合運動訓練治療慢性頸痛的臨床觀察. 中國中西醫結合雜志, 2018, 38(11): 1327-1331.
30. 盛偉, 林敏. 中醫正骨手法聯合康復訓練改善頸型頸椎病患者CASCS評分、Berg平衡功能和疼痛程度效果觀察. 時珍國醫國藥, 2018, 29(06): 1391-1393.
31. 馮敏山, 高景華, 高春雨, 等. 坐位腰椎旋轉手法的運動學研究. 中國中醫骨傷科雜志, 2018, 26(07): 16-20.
32. 曹慧, 高凡茹, 魏德健, 等. 中醫正骨仿真系統中姿態檢測的應用. 時珍國醫國藥, 2017, 28(12): 3017-3019.
33. 周晨爍, 魏德健, 曹慧, 等. 中醫正骨仿真系統中正骨手法的評價. 中華中醫藥學刊, 2018, 36(08): 1826-1828.
34. 朱立國, 韓濤, 于杰, 等. 中醫骨傷科旋提手法規范化操作傳承模式初探. 中醫雜志, 2018, 59(11): 927-931.

Previous Article
Research advancements of motor imagery for motor function recovery after stroke
Next Article
Research progress of decellularized extracellular matrix hydrogel in regenerative medicine

Journal of Biomedical Engineering

A review on multi-modal human motion representation recognition and its application in orthopedic rehabilitation training

Abstract Full text Figures/Tables Video References Cited by

Previous Article

Next Article

Format

Content