Awesome Human Activity Recognition

Human Activity Recognition (HAR) is the field of recognizing human actions and activities from sensor data — including video, skeleton/mocap, wearable IMU, and multimodal egocentric inputs. This list covers datasets, frameworks, pretrained models, tutorials, papers, competitions, and tools for HAR research.

中文 | Deutsch | Español | Français | 日本語 | 한국어 | Português | Русский

Contents

• Which Dataset Should I Use
• Datasets
• Frameworks and Libraries
• Pretrained Models
• Tutorials and Courses
• Key Papers
• Competitions and Challenges
• Tools and Utilities
• Related Awesome Lists

Which Dataset Should I Use

Pick your modality and task, then follow the recommendation to the right section.

I have video and want to classify actions — Start with Kinetics-700 for pretraining, evaluate on UCF-101 or HMDB-51 for comparison with prior work. See Vision.

I need temporal action detection in untrimmed video — ActivityNet for proposals, AVA for spatio-temporal, MultiTHUMOS for dense multi-label. Also listed under Vision above.

I work with skeleton or motion capture data — NTU RGB+D 120 is the de facto standard. For text-motion alignment, use Babel or HumanML3D. See Skeleton and Emerging.

I have IMU or wearable sensor data — UCI-HAR for baselines, PAMAP2 for multi-sensor, CAPTURE-24 for real-world scale (151 subjects, 3883 hours). See Wearable.

I need egocentric or multimodal data — Ego4D for scale (3.3k hours), EPIC-Kitchens-100 for kitchen actions, Ego-Exo4D for cross-view (NEW, CVPR 2024). See Multimodal.

I want text-to-motion generation — HumanML3D for single-person, InterHuman for two-person, Motion-X++ for whole-body with face and hands. Also listed under Emerging above.

Datasets

Vision (RGB / Depth)

• Kinetics-700 - Large-scale pretraining benchmark with 650k YouTube clips across 700 action classes.
• UCF-101 - Classic action recognition benchmark with 13.3k clips across 101 classes.
• HMDB-51 - Diverse action recognition dataset with 6.8k clips from movies and web videos across 51 classes.
• ActivityNet - Temporal action detection benchmark with 20k untrimmed YouTube videos across 200 classes.
• AVA - Spatio-temporal action detection with 430 movie clips and 80 atomic action labels with bounding boxes.
• NTU RGB+D 120 - Multi-view 3D action recognition with 114k sequences across 120 classes using RGB, depth, and skeleton.
• Something-Something V2 - Fine-grained object interaction dataset with 220k clips across 174 labels requiring temporal reasoning.
• FineGym - Fine-grained gymnastics action recognition with 32k hierarchically labeled segments.
• Moments in Time - Extremely diverse event and action recognition dataset with 1M labeled 3-second video clips across 339 classes.
• Diving48 - Fine-grained diving action recognition with 18k clips across 48 classes requiring temporal reasoning.
• Toyota Smarthome - Daily living activity recognition with 16k multi-view clips across 31 classes using RGB, depth, and skeleton.
• MultiSports - Spatio-temporal action detection across 4 sports with 3.2k clips and 66 fine-grained action classes.
• MultiTHUMOS - Dense multi-label temporal action detection with 65 classes and 38k annotations.
• FineSports - Multi-person fine-grained sports understanding with 10k NBA videos and 52 action types from CVPR 2024.

Skeleton and Mocap

• NTU RGB+D 60 - Foundation dataset for skeleton-based action recognition with 57k sequences across 60 classes.
• AMASS - Unified SMPL motion capture parameters from 40+ datasets covering 16k minutes and 344 subjects.
• Human3.6M - De facto standard for 3D pose estimation with 3.6M frames from 11 professional actors.
• Babel - Motion-language alignment dataset with 43 hours and 3.7k sequences annotated with SMPL and text labels.
• TotalCapture - Multi-modal 3D pose estimation benchmark combining mocap, multi-view RGB, and IMU from 5 subjects.
• PKU-MMD - Multi-modality action detection benchmark with 20k instances across 51 classes.
• Skeletics-152 - Large-scale skeleton action recognition from estimated poses with 150k clips across 152 classes.

Wearable Sensors

• UCI-HAR - Classic smartphone IMU benchmark with 30 subjects and 6 activities, near-saturated.
• PAMAP2 - Wearable HAR standard with multi-IMU and heart rate from 9 subjects across 18 activities.
• WISDM - Phone and smartwatch sensor data mining with 51 subjects and over 1 million samples.
• OPPORTUNITY - Rich context-aware activity recognition with 72 wearable and ambient sensors from 4 subjects.
• HAPT - Smartphone IMU dataset with postural transition detection from 30 subjects across 12 activities.
• RealWorld HAR - In-the-wild activity recognition with multiple device placements from 60 subjects across 15 activities.
• mHealth - Body-worn sensors with ECG for mobile health monitoring from 10 subjects across 12 activities.
• UniMiB-SHAR - Smartphone accelerometer dataset for daily activities and fall detection from 30 subjects across 17 activities.
• Daphnet - Freezing of gait detection for Parkinson's patients using 3 wearable accelerometers from 10 subjects.
• Sussex-Huawei Locomotion - Large-scale locomotion mode recognition with 2800+ hours from 3 users with phone and watch sensors.
• HARTH - Professional video-annotated free-living accelerometer HAR from 22 subjects in real-world conditions.
• CAPTURE-24 - Largest free-living wrist accelerometer dataset with 151 subjects and 3883 hours from Nature Scientific Data 2024.
• WEAR - Outdoor sports dataset with smartwatch IMU and egocentric video from 22 subjects across 18 activities, published at IMWUT 2024.

Multimodal and Egocentric

• EPIC-Kitchens-100 - Long-term egocentric kitchen actions with audio spanning 700 hours across 90 kitchens.
• Ego4D - Largest egocentric dataset with multi-task benchmarks spanning 3.3k hours across 74 scenes.
• Charades - Indoor multi-label action recognition with scripted descriptions spanning 9.8k videos across 157 labels.
• NTU Mutual Actions - Two-person interactions from NTU RGB+D with skeleton data across 26 interaction classes.
• ActivityNet Captions - Dense video captioning and temporal grounding with 20k videos and 100k captions.
• How2Sign - Multimodal American Sign Language dataset with RGB, depth, and pose spanning 80 hours.
• EgoExo-Fitness - Ego and exo fitness action quality assessment with 31 hours and 6k+ actions from ECCV 2024.

Emerging and Frontier

• BEHAVE - RGB-D human-object interaction with 3D pose spanning 321 sequences from 20 subjects.
• Motion-X - Full-body and hand joint motion from multisensor mocap with 2M frames from 10 subjects.
• Ego-Exo4D - Cross-view action understanding with synchronized ego and exo video spanning 1.4k sequences.
• HumanML3D - Text-to-motion generation dataset with SMPL annotations spanning 14k+ motion sequences.
• InterHuman - Two-person interaction motion with SMPL-X and text descriptions spanning 6k+ sequences.
• HOI4D - Egocentric hand-object interaction with RGB-D and hand pose spanning 4k+ video clips.
• FineBio - Fine-grained biology lab action understanding with multi-step procedure annotations.
• HAA500 - Diverse fine-grained atomic action recognition with 10k clips across 500 classes.
• Motion-X++ - Whole-body motion generation with text and audio spanning 120k+ sequences.
• FLAG3D - 3D fitness activity understanding with multi-view RGB, skeleton, and text spanning 180k sequences from CVPR 2024.
• InterX - Comprehensive human-human interaction dataset with SMPL-X spanning 11k+ sequences from CVPR 2024.
• WiMANS - First WiFi-based multi-user activity sensing benchmark at a top venue from ECCV 2024.

Frameworks and Libraries

Video Action Recognition

• MMAction2 - OpenMMLab toolbox for video understanding supporting 20+ model architectures including SlowFast, TimeSformer, and VideoMAE.
• PySlowFast - Facebook Research library for video understanding with SlowFast, X3D, MViT, and AVA models.
• Video-Swin-Transformer - Pure-transformer backbone for video recognition achieving SOTA on Kinetics-400, Kinetics-600, and SSv2.
• TimeSformer - Facebook Research divided space-time attention for video classification from ICML 2021.
• VideoMAE - Self-supervised video pretraining with masked autoencoders achieving SOTA on multiple benchmarks.
• InternVideo2 - Foundation model for video understanding at scale supporting action recognition, retrieval, and captioning.

Skeleton Action Recognition

• CTR-GCN - Channel-wise topology refinement graph convolution for skeleton-based action recognition from ICCV 2021.
• ST-GCN - Seminal spatial-temporal graph convolution network that established the GCN approach for skeleton-based HAR.
• 2s-AGCN - Two-stream adaptive graph convolutional network for skeleton-based action recognition from CVPR 2019.
• HD-GCN - Hierarchically decomposed graph convolutional network for skeleton action recognition from AAAI 2024.
• MotionBERT - Unified pretraining for human motion analysis covering 3D pose estimation and action recognition.
• InfoGCN - Information-bottleneck graph convolutional network for skeleton action recognition from CVPR 2022.

Wearable Sensor HAR

• tsai - Deep learning library for time series and sequences built on fastai and PyTorch, widely used for sensor HAR.
• aeon - Unified Python toolkit for time series including classification, clustering, and anomaly detection.
• NNCLR-HAR - Self-supervised contrastive learning framework for wearable sensor HAR from IMWUT 2022.
• DeepConvLSTM - Reference implementation of the convolutional LSTM architecture for wearable activity recognition.
• Hang-Time HAR - Basketball activity recognition from a single wrist-worn inertial sensor using deep learning.

Motion Generation and Estimation

• MDM - Human motion diffusion model for text-