-
Comparison of marker-less 2D image-based methods for infant pose estimation
Authors:
Lennart Jahn,
Sarah Flügge,
Dajie Zhang,
Luise Poustka,
Sven Bölte,
Florentin Wörgötter,
Peter B Marschik,
Tomas Kulvicius
Abstract:
In this study we compare the performance of available generic- and infant-pose estimators for a video-based automated general movement assessment (GMA), and the choice of viewing angle for optimal recordings, i.e., conventional diagonal view used in GMA vs. top-down view. We used 4500 annotated video-frames from 75 recordings of infant spontaneous motor functions from 4 to 26 weeks. To determine w…
▽ More
In this study we compare the performance of available generic- and infant-pose estimators for a video-based automated general movement assessment (GMA), and the choice of viewing angle for optimal recordings, i.e., conventional diagonal view used in GMA vs. top-down view. We used 4500 annotated video-frames from 75 recordings of infant spontaneous motor functions from 4 to 26 weeks. To determine which pose estimation method and camera angle yield the best pose estimation accuracy on infants in a GMA related setting, the distance to human annotations and the percentage of correct key-points (PCK) were computed and compared. The results show that the best performing generic model trained on adults, ViTPose, also performs best on infants. We see no improvement from using infant-pose estimators over the generic pose estimators on our infant dataset. However, when retraining a generic model on our data, there is a significant improvement in pose estimation accuracy. The pose estimation accuracy obtained from the top-down view is significantly better than that obtained from the diagonal view, especially for the detection of the hip key-points. The results also indicate limited generalization capabilities of infant-pose estimators to other infant datasets, which hints that one should be careful when choosing infant pose estimators and using them on infant datasets which they were not trained on. While the standard GMA method uses a diagonal view for assessment, pose estimation accuracy significantly improves using a top-down view. This suggests that a top-down view should be included in recording setups for automated GMA research.
△ Less
Submitted 26 March, 2025; v1 submitted 7 October, 2024;
originally announced October 2024.
-
Deep learning empowered sensor fusion boosts infant movement classification
Authors:
Tomas Kulvicius,
Dajie Zhang,
Luise Poustka,
Sven Bölte,
Lennart Jahn,
Sarah Flügge,
Marc Kraft,
Markus Zweckstetter,
Karin Nielsen-Saines,
Florentin Wörgötter,
Peter B Marschik
Abstract:
To assess the integrity of the developing nervous system, the Prechtl general movement assessment (GMA) is recognized for its clinical value in diagnosing neurological impairments in early infancy. GMA has been increasingly augmented through machine learning approaches intending to scale-up its application, circumvent costs in the training of human assessors and further standardize classification…
▽ More
To assess the integrity of the developing nervous system, the Prechtl general movement assessment (GMA) is recognized for its clinical value in diagnosing neurological impairments in early infancy. GMA has been increasingly augmented through machine learning approaches intending to scale-up its application, circumvent costs in the training of human assessors and further standardize classification of spontaneous motor patterns. Available deep learning tools, all of which are based on single sensor modalities, are however still considerably inferior to that of well-trained human assessors. These approaches are hardly comparable as all models are designed, trained and evaluated on proprietary/silo-data sets. With this study we propose a sensor fusion approach for assessing fidgety movements (FMs). FMs were recorded from 51 typically developing participants. We compared three different sensor modalities (pressure, inertial, and visual sensors). Various combinations and two sensor fusion approaches (late and early fusion) for infant movement classification were tested to evaluate whether a multi-sensor system outperforms single modality assessments. Convolutional neural network (CNN) architectures were used to classify movement patterns. The performance of the three-sensor fusion (classification accuracy of 94.5%) was significantly higher than that of any single modality evaluated. We show that the sensor fusion approach is a promising avenue for automated classification of infant motor patterns. The development of a robust sensor fusion system may significantly enhance AI-based early recognition of neurofunctions, ultimately facilitating automated early detection of neurodevelopmental conditions.
△ Less
Submitted 5 December, 2024; v1 submitted 13 June, 2024;
originally announced June 2024.
-
Mobile solutions for clinical surveillance and evaluation in infancy -- General Movement Apps
Authors:
Peter B Marschik,
Amanda KL Kwong,
Nelson Silva,
Joy E Olsen,
Martin Schulte-Ruether,
Sven Bolte,
Maria Ortqvist,
Abbey Eeles,
Luise Poustka,
Christa Einspieler,
Karin Nielsen-Saines,
Dajie Zhang,
Alicia J Spittle
Abstract:
The Prechtl General Movements Assessment (GMA) has become a clinician and researcher tool-box for evaluating neurodevelopment in early infancy. Given it involves observation of infant movements from video recordings, utilising smartphone applications to obtain these recordings seems like the natural progression for the field. In this review, we look back on the development of apps for acquiring ge…
▽ More
The Prechtl General Movements Assessment (GMA) has become a clinician and researcher tool-box for evaluating neurodevelopment in early infancy. Given it involves observation of infant movements from video recordings, utilising smartphone applications to obtain these recordings seems like the natural progression for the field. In this review, we look back on the development of apps for acquiring general movement videos, describe the application and research studies of available apps, and discuss future directions of mobile solutions and their usability in research and clinical practice. We emphasise the importance of understanding the background that has led to these developments while introducing new technologies, including the barriers and facilitators along the pathway. The GMApp and Baby Moves App were the first ones developed to increase accessibility of the GMA, with two further apps, NeuroMotion and InMotion, designed since. The Baby Moves app has been applied most frequently. For the mobile future of GMA, we advocate collaboration to boost the field's progression and to reduce research waste. We propose future collaborative solutions including standardisation of cross-sites data collection, adaption to local context and privacy laws, employment of user feedback, and sustainable IT structures enabling continuous software updating.
△ Less
Submitted 26 March, 2023;
originally announced March 2023.
-
Facilitating deep acoustic phenotyping: A basic coding scheme of infant vocalisations preluding computational analysis, machine learning and clinical reasoning
Authors:
Tomas Kulvicius,
Sigrun Lang,
Claudius AA Widmann,
Nina Hansmann,
Daniel Holzinger,
Luise Poustka,
Dajie Zhang,
Peter B Marschik
Abstract:
Theoretical background: early verbal development is not yet fully understood, especially in its formative phase. Research question: can a reliable, easy-to-use coding scheme for the classification of early infant vocalizations be defined that is applicable as a basis for further analysis of language development? Methods: in a longitudinal study of 45 neurotypical infants, we analyzed vocalizations…
▽ More
Theoretical background: early verbal development is not yet fully understood, especially in its formative phase. Research question: can a reliable, easy-to-use coding scheme for the classification of early infant vocalizations be defined that is applicable as a basis for further analysis of language development? Methods: in a longitudinal study of 45 neurotypical infants, we analyzed vocalizations of the first 4 months of life. Audio segments were assigned to 5 classes: (1) Voiced and (2) Voiceless vocalizations; (3) Defined signal; (4) Non-target; (5) Nonassignable. Results: Two female coders with different experience achieved high agreement without intensive training. Discussion and Conclusion: The reliable scheme can be used in research and clinical settings for efficient coding of infant vocalizations, as a basis for detailed manual and machine analyses.
△ Less
Submitted 14 March, 2023;
originally announced March 2023.
-
Infant movement classification through pressure distribution analysis
Authors:
Tomas Kulvicius,
Dajie Zhang,
Karin Nielsen-Saines,
Sven Bölte,
Marc Kraft,
Christa Einspieler,
Luise Poustka,
Florentin Wörgötter,
Peter B Marschik
Abstract:
Aiming at objective early detection of neuromotor disorders such as cerebral palsy, we proposed an innovative non-intrusive approach using a pressure sensing device to classify infant general movements (GMs). Here, we tested the feasibility of using pressure data to differentiate typical GM patterns of the ''fidgety period'' (i.e., fidgety movements) vs. the ''pre-fidgety period'' (i.e., writhing…
▽ More
Aiming at objective early detection of neuromotor disorders such as cerebral palsy, we proposed an innovative non-intrusive approach using a pressure sensing device to classify infant general movements (GMs). Here, we tested the feasibility of using pressure data to differentiate typical GM patterns of the ''fidgety period'' (i.e., fidgety movements) vs. the ''pre-fidgety period'' (i.e., writhing movements). Participants (N = 45) were sampled from a typically-developing infant cohort. Multi-modal sensor data, including pressure data from a 32x32-grid pressure sensing mat with 1024 sensors, were prospectively recorded for each infant in seven succeeding laboratory sessions in biweekly intervals from 4-16 weeks of post-term age. For proof-of-concept, 1776 pressure data snippets, each 5s long, from the two targeted age periods were taken for movement classification. Each snippet was pre-annotated based on corresponding synchronised video data by human assessors as either fidgety present (FM+) or absent (FM-). Multiple neural network architectures were tested to distinguish the FM+ vs. FM- classes, including support vector machines (SVM), feed-forward networks (FFNs), convolutional neural networks (CNNs), and long short-term memory (LSTM) networks. The CNN achieved the highest average classification accuracy (81.4%) for classes FM+ vs. FM-. Comparing the pros and cons of other methods aiming at automated GMA to the pressure sensing approach, we concluded that the pressure sensing approach has great potential for efficient large-scale motion data acquisition and sharing. This will in return enable improvement of the approach that may prove scalable for daily clinical application for evaluating infant neuromotor functions.
△ Less
Submitted 1 July, 2023; v1 submitted 26 July, 2022;
originally announced August 2022.
-
Open video data sharing in developmental and behavioural science
Authors:
Peter B Marschik,
Tomas Kulvicius,
Sarah Flügge,
Claudius Widmann,
Karin Nielsen-Saines,
Martin Schulte-Rüther,
Britta Hüning,
Sven Bölte,
Luise Poustka,
Jeff Sigafoos,
Florentin Wörgötter,
Christa Einspieler,
Dajie Zhang
Abstract:
Video recording is a widely used method for documenting infant and child behaviours in research and clinical practice. Video data has rarely been shared due to ethical concerns of confidentiality, although the need of shared large-scaled datasets remains increasing. This demand is even more imperative when data-driven computer-based approaches are involved, such as screening tools to complement cl…
▽ More
Video recording is a widely used method for documenting infant and child behaviours in research and clinical practice. Video data has rarely been shared due to ethical concerns of confidentiality, although the need of shared large-scaled datasets remains increasing. This demand is even more imperative when data-driven computer-based approaches are involved, such as screening tools to complement clinical assessments. To share data while abiding by privacy protection rules, a critical question arises whether efforts at data de-identification reduce data utility? We addressed this question by showcasing the Prechtl's general movements assessment (GMA), an established and globally practised video-based diagnostic tool in early infancy for detecting neurological deficits, such as cerebral palsy. To date, no shared expert-annotated large data repositories for infant movement analyses exist. Such datasets would massively benefit training and recalibration of human assessors and the development of computer-based approaches. In the current study, sequences from a prospective longitudinal infant cohort with a total of 19451 available general movements video snippets were randomly selected for human clinical reasoning and computer-based analysis. We demonstrated for the first time that pseudonymisation by face-blurring video recordings is a viable approach. The video redaction did not affect classification accuracy for either human assessors or computer vision methods, suggesting an adequate and easy-to-apply solution for sharing movement video data. We call for further explorations into efficient and privacy rule-conforming approaches for deidentifying video data in scientific and clinical fields beyond movement assessments. These approaches shall enable sharing and merging stand-alone video datasets into large data pools to advance science and public health.
△ Less
Submitted 22 July, 2022;
originally announced July 2022.
