MUMS a Measure of hUman Motion Similarity Francisco J Torres Reyes Problem and Challenges 3D ChainCode LABANotation Contribution 1: Comparison Analysis of ChainCode and FastDTW

Contribution 2: Enhanced LABANotation for Rehabilitation Contribution 3: System Architecture for HMTR Lessons Learned Future Direction Conclusion Ftorres/MUMS 4/24/14 Outline of the Talk

2 Ftorres/MUMS 4/24/14 Why Measuring Human Motion Similarity? 3 How is Human Motion Captured and Modeled? Ftorres/MUMS 4/24/14

Sports high speed camera for slow motion speed analysis VR real time data acquisition -> simulation time Modeling human body -> skeleton 4 Ftorres/MUMS 3D Curves shown by MUMS tool from a shoulder exercise. I developed this tool for Windows environment These two diagrams capture the accumulated tracks of two snapshots. It shows all sensors data including those from head, torso, abdomen, arms and legs.

4/24/14 Human Motion can be modeled as Sets of 3D Curves 5 Utilize Time3D data from HMTR Project Ftorres/MUMS Dr. Yunyu Wang Certified Movement Analysis, LABANotation Reconstructor and Teacher Dr. James Carollo Physical Medicine and Rehabilitation, Ortophaedics, Bioengineering, School of Engineering and Applied Science.

4/24/14 Four Key Rehabilitation Exercises were chosen and analyzed in HMTR project by 6 # Pts = 34 # Video Frames = Video Frame # 1 733.4042 2 720.8265

33 452.7758 34 666.9445 Video Frame # 3D data per marker Sampling rate Body position for markers 1238 1 1447.329 1322.164

1343.254 1325.737 2 1659.807 1659.62 Ftorres/MUMS MoCap Data 4/24/14 C3D File Content 38.04591

36.31926 7 Ftorres/MUMS Invariant under Translation or Rotation [Bribiesca, 2006] 4/24/14 Chain Code - Orthogonal Changes of Direction 8

Measuring similarity on chain codes S , T are two curves m,n are the lengths of S and T L(x) is the length of a curve P is a common sub curve between S and T (x, y) is a pseudo metric of accumulated direction) is a pseudo metric of accumulated direction Nes, net are the number of preceding elements to P 9 Measuring similarity on chain codes 10 J is the index of the set of common sub curve P found between curves A and B l is the number of the set of common sub curves

Adding Time into the Equation Ftorres/MUMS 4/24/14 The United States National Anthem 11 What is an equivalent representation for human motion? LABANotation LABANotation: a record of how one moves so it can be repeated. This notation includes a set of sy) is a pseudo metric of accumulated directionmbols

that are placed on a vertical staff, where its vertical dimension represents the sy) is a pseudo metric of accumulated directionmmetry) is a pseudo metric of accumulated direction of the body) is a pseudo metric of accumulated direction, and its horizontal one represents the time [Bouchard, 2008] 12 Spatial and Temporal Analysis Ftorres/MUMS Note that there are five sensors per leg. Therefore five corresponding chain code may) is a pseudo metric of accumulated direction be generated. 4/24/14

Movement on the right leg track can be encoded in chain code for motion analy) is a pseudo metric of accumulated directionsis. Measures of 3 beats display) is a pseudo metric of accumulated direction from bottom up. Different movements of body) is a pseudo metric of accumulated direction limbs are encoded with directions 13 Y2 X2 X4 Ftorres/MUMS

Finds the optimal alignment between two time series Use the value calculated based on the optimal alignment to represent the similarity. If two time series are the same, the similarity value is zero. 4/24/14 Dynamic Time Warping Y2 X2 X4 Similarity) is a pseudo metric of accumulated direction values contributed by) is a pseudo metric of accumulated direction subsequence pairs: d(Y2, X4)+d(Y3,X5)+d(Y4,X6)+d(Y5,X7)+d(Y6,X8)=0+0+0+0+0=0 > d(Y2, X2)+d(Y3,X3)+d(Y4,X4)+d(Y5,X5)+d(Y6,X5)=1+1+0+1+0+1=4

14 Slow Start vs. Fast Pace 3D ChainCode Slow start Idle at starting position Fast pace New Idle sy) is a pseudo metric of accumulated directionmbol

Idle at ending position 15 same 3D curves 4/24/14 Slow Start rotated 90o 3D ChainCode and FastDTW Ftorres/MUMS

16 4/24/14 Slow Start rotated 270o 3D ChainCode and FastDTW Ftorres/MUMS

17 Shoulder Elevation and Rotation Exercise Similarity) is a pseudo metric of accumulated direction values of arms time3D curves are close with 3D ChainCode Similarity) is a pseudo metric of accumulated direction Formula Similarity) is a pseudo metric of accumulated direction values of arms time3D curves are quite different with fastDTW results 18 Ftorres/MUMS

4/24/14 Standing Hip Abduction Exercise 19 Ftorres/MUMS 4/24/14 Mini Squat Exercise 20 Add movement precision by) is a pseudo metric of accumulated direction adding new sy) is a pseudo metric of accumulated directionmbols Minimize notation modifications and changes

Apply) is a pseudo metric of accumulated direction the new notation on improving the specification of key) is a pseudo metric of accumulated direction rehab exercises in the HMTR project. Ftorres/MUMS LABANotation is designed to describe dance. We studied its usage and suggested the enhancement for rehabilitation purposes. Focus on the 4/24/14 Contribution 2: Enhanced LABANotation for Rehabilitation

21 Ftorres/MUMS Start standing with equal weight distributed between right and left legs Place feet shoulder width apart Keep torso upright, avoid bending at the waist Slowly loser yourself by bending ankles, knees, and hips Return to standing

4/24/14 Enhanced LABANotation for Rehabilitation Mini Squats Exercise 22 Ftorres/MUMS Start standing with equal weight distributed between right and left legs Slowly, shift your weight to the left side Raise the leg out to the side ~ 12 Keep the right foot facing forward Keep the torso upright and avoid leaning

to the side 4/24/14 Enhanced LABANotation for Rehabilitation Standing Hip Abduction Exercise 23 LabanWriter (Mac version from Ohio State) LabanDancer (Windows version from Dance Bureau) Ftorres/MUMS We propose a Human Motion Tracking and Reasoning (HMTR) Software Architecture.

Evaluates Tools for HMTR System Design 4/24/14 Contribution 3: HMTR System Design and Tool Evaluation 24 Ftorres/MUMS 4/24/14 Proposed Human Motion Tracking and Reasoning (HMTR) Software Architecture

25 Suggested Enhancement for LabanDancer Software Ftorres/MUMS 4/24/14 [Wilke, Calvert, Ryman, 2005] 26 Lessons Learned The original 3D ChainCode dissimilarity algorithm is very slow

when applying to real exercise data. The steps are reexamined and improved for the time performance. Bribiescas group did not consider the idle situation and did not encode the elapsed of time. They are interested in shape of the curves, while we are interested in movement. Laban Dancer executable code works fine but the source code was compiled in different version of visual studio and even the original authors can not provide a working project. Ftorres/MUMS Use tool from Internet to extract into text form and feed them in chain code program. 4/24/14 C3D data is a binary data difficult to parse.

27 Use this framework in another areas, i.e. dancing, video games Improve run time for the analysis of human motion, i.e. using nVidia CUDA tools Getting additional data with semantics for further analysis of similarity Define additional properties such off-track, sustain and develop procedure for computing the values for the properties. Ftorres/MUMS 4/24/14

Future Research 28 Ftorres/MUMS Proposed a new model to represent human motion Used LABANotation to analyze human motion on spatial and temporal domains Suggested enhancement of LABANotation for rehabilitation Developed a software tool to perform the analysis of human motion similarity on motion capture sessions Proposed an HMTR software architecture Propose enhancements for LabanDancer software for rehabilitation purposes

The analysis of human motion is needed in different areas of study. Three papers were published. Will submit the work on the comparison of chain code and fastDTW. 4/24/14 Conclusion 29