WWDC 21 Scholarship Demo Video - HumanScan
Repo: https://github.com/JustinFincher/WWDC...
HumanScan is an interactive playground for capturing, rigging and applying animations to peoples in AR space. It involves four stages:
Environment Scanning: HumanScan uses a lidar-enabled depth map to build up a point cloud around the person you wish to scan. Every ARFrame is processed with the Metal command queue to fill up the point cloud buffer. Then an updated SCNGeometry is built from the buffer.
Skeleton Tracking: HumanScan uses ARKit 4's human tracking to acquire joint hierarchy. The hierarchy is then reduced from 91 joints to 19 joints for simplicity, with minor positioning adjustments if needed to better match the previously scanned point cloud.
Runtime Skinning: HumanScan now has both mesh and skeleton information. The first it will do is build up a signed distance field based on the skeleton to filter points to isolate human point clouds from background point clouds. Then HumanScan will traverse each point (vertex) and examine the value of signed distance fields from different joints, essentially deciding which bones this particular vertex wants to follow. Finally, HumanScan creates a SCNSkinner from the calculated bone indices and weights so the skeleton can actually influence point clouds' position.
Animation Replay: With the same technique in skeleton tracking, I have already pre-recorded an animation clip of myself doing hand-waving in AR space. This clip is serialized as a JSON file within the playground and will be applied to the newly created human model. Now, as the skeleton replays the same movement, the scanned point cloud will follow, demonstrating that we have created a digital clone that can perform arbitrary animations.
Frameworks used in HumanScan are SceneKit, ARKit, MetalKit, SwiftUI, and Combine. Together, HumanScan achieved point cloud generation and runtime mesh skinning.
#wwdc #wwdcscholarship #swiftstudentchallenge #arkit #scenekit #lidar