SF Tech Week · San Francisco · 7 October 2026
Learning from human video: notes from the HackerSquad VLA panel
Disclosure
This article is published on both Kinetic Blocks and Humanoid.guide. Humanoid.guide holds a stake in Kinetic Blocks AS, which operates a marketplace for licensed robot training data, and the two publications are kept apart by an editorial firewall.
On Wednesday 7 October HackerSquad and Bright Data hosted the panel “Learning from Human Video & Scaling VLA Training” at Bright Data’s premises in San Francisco, where five people working on robots, data tooling and models discussed how video of humans can be turned into training data for vision-language-action (VLA) models, and why accuracy measured in the laboratory so often falls away when a model is moved to a real site. The article is in two parts, the first covering the panel discussion and the second covering the talk Chengjiang Long of ByteDance gave before it.
Part 1
The panel
Megan Smith, founder and CEO in robotics and physical AI, came to the subject from space robotics at Northrop Grumman, where she worked on lunar modules and the docking system for Orion, and from construction robots, and she is now building an autonomous robot that applies makeup with an airbrush guided by computer vision. Her contribution was the most practical of the evening, since she argued that probabilistic AI belongs in training while the robot that goes into service should run on deterministic systems, which lowers latency and compute cost and makes the robot dependable when it is deployed in numbers. She noted that developing a single robot action used to take three to six months and now takes around three hours with the help of language models, and her advice to founders was to stay with the customer’s problem and the return on investment and to avoid being pulled along by viral demonstration videos. A participant from surgical robotics added from the floor that systems in that field are entirely classical, because life-critical settings require full determinism.
Chengjiang Long, research engineer and tech lead at ByteDance, took the opposite position on the architecture question and argued for end-to-end training on the grounds that errors in separate components compound, so that even 99 per cent accuracy at each step leads to cascading failures over a long chain, while a model trained end to end can make its later layers robust to gaps in the earlier ones.
Sashank Aryal, principal engineer and technical lead at Voxel51, who previously worked on warehouse robots at Amazon Robotics, described FiftyOne, Voxel51’s open-source platform for indexing, sorting, finding and labelling robotics datasets. His starting point was that the quality of the data decides whether a policy succeeds, and that robotics lacks the mature data infrastructure that language model companies could build on. One of the capabilities he presented is segment mining, where relevant episodes are found by querying the dataset in natural language, and he described it as a problem that is still partly manual.
Anto Patrex, founder and CEO of CosmicBrain AI, with a background from xAI and NASA, described a market where vertical integrators such as Fanuc and Kuka own the whole chain from hardware to deployment, while physical AI has no plug-and-play teleoperation stack that works across robot brands and no generalist model across hardware comparable to what ChatGPT is for language. In his view specialised VLA models perform better in bounded domains such as warehouses and shipyard welding, and he told founders that collaboration matters and that working in isolation is a mistake.
Raz Kaplan, who leads AI go-to-market at Bright Data, works with companies on identifying their next data needs for VLA development. He described an index of more than two billion videos that can be searched for exact moments, such as two hundred people holding the same yoga pose, and said that one to two million new videos are added each day, which in his view means that web video is still a usable foundation for pre-training and far from exhausted.
Several points came up across the discussion without being tied to one speaker. Teleoperation data was described as the reference standard, because it records every actuator and tactile sensor in a way simulation cannot reproduce, while around 60 per cent of egocentric video was said to be unusable because hands leave the frame, the image is blurred or the object is not visible. The panel also pointed to models whose accuracy fell from 98 to 37.5 per cent when they were moved from controlled settings to real sites, to Waymo’s experience of two human supervisors per car over fourteen years for a single task, to data farms in China producing around 5,000 hours of teleoperation data a week, and to three egocentric data collection companies that had reportedly closed in the previous three weeks because their margins could not cover the volumes foundation model companies are asking for.
Part 2
Chengjiang Long on the four gaps between web video and a training example
Chengjiang Long has a background in active learning from his PhD and now works on human-centric modelling and digital humans at ByteDance, with an end-to-end system for tracking and animation that takes input from VR headsets and webcams. His talk started from the observation that web video provides long, uncut footage with titles, captions and narration, while a VLA needs skill clips with a start and an end, a verb stating which action is performed on which object, three-dimensional motion that can be transferred to another body, and knowledge of material, mass and contact. He organised this as four gaps, where each gap is a labelling step and the weakest step sets the quality of the whole dataset, and for each gap he pointed to his own published work as evidence.
| Gap | Web video gives | A VLA needs | My evidence |
|---|---|---|---|
| When | Long, uncut footage | Skill clips with a start and an end | T2SGrid, TAR |
| What | Titles, captions, narration | The verb: which action, on which object | Invert4TVG |
| How | 2D pixels of a human body | 3D motion that transfers to another body | PUMPS, human mesh recovery |
| Why it works | Appearance only | Material, mass, contact | PhysSplat |
Each gap is a labeling step. The quality of the weakest step sets the quality of the dataset.
When. T2SGrid (CVPR 2026) handles the timing problem by arranging the frames inside a sliding time window into a single grid image, so that a vision-language model can use its spatial attention to reason about what happens over time and locate the segment that matches a text query.
What. Invert4TVG (ICLR 2026) shows that models tuned for temporal IoU often lose the ability to recognise the action itself, and that three tasks derived from the same labels, verb completion, action recognition and video description, preserve that ability, with an improvement of 7.1 per cent on R1@0.7 on Charades-STA for a 3B model. The consequence he drew for fine-tuning a VLM into a VLA is that understanding tasks should stay in the loss and be derived from labels already available, so that the backbone does not forget what a verb is.
How. PUMPS (ICCV 2025) addresses the fact that motion data does not transfer between skeletons with different proportions or structure, by encoding motion as temporal point clouds that remove the skeleton, so that one pre-trained model can handle motion prediction, transitions and keyframe interpolation. For human video used as robot data this means lifting 3D motion out of monocular video first and storing it in a body-agnostic form, since pseudo-actions are only as good as that recovery step.
Why it works. PhysSplat (ICCV 2025) uses a multimodal language model to estimate the physical properties of objects in a reconstructed 3D scene zero-shot, well enough to drive a particle simulation without hand-set parameters, in under two minutes on a single GPU. Since video shows appearance and never force, a language prior over materials becomes a cheap way to annotate what the camera cannot see and to turn a captured scene into an interactive one for synthetic rollouts.
Where labels are missing altogether, he proposed generating video that carries them, with three approaches on his slides: video generated under an explicit 6-DoF camera pose and 20-joint hand parameters, so that the conditioning signal is close to an action label; EgoX, which re-renders third-person footage as the first-person view the wearer would have seen; and Gaussian Scene Graph, where rigid, SMPL and deformable nodes are optimised and rendered together into controllable simulation data with precise labels. The cost of this route is the gap between synthetic and real data, which then has to be measured.
He closed the talk with a lesson from his work on photo-real avatars, where quality was still rising at 10,000 identities, returns flattened at around 16 input views, 4,000 supervision examples did better than more subjects at the same budget, and 93.9 per cent of a 545-million-parameter model stayed frozen, with a note on the slide that the transfer to robot data is an analogy without its own measurement. His three positions for the panel followed from this: the bottleneck is moving from hours of video to the quality of supervision, so the scaling curve should be measured on each axis before money is spent on data; fine-tuning from VLM to VLA should keep understanding objectives in the loss; and the labelling pipeline should be benchmarked as carefully as the policy.
Sources
- Chen, Z., Lin, H., Nie, Y., Ma, F., Xu, X., Yu, F. and Long, C. Invert4TVG: A Temporal Video Grounding Framework with Inversion Tasks Preserving Action Understanding Ability. ICLR 2026. arXiv:2508.07388.
- Guo, C., He, Y., Nie, Y., Ma, F., Xu, X. and Long, C. T2SGrid: Temporal-to-Spatial Gridification for Video Temporal Grounding. CVPR 2026, pp. 3443 to 3454. arXiv:2603.06973.
- Mo, C. A., Hu, K., Long, C., Yuan, D., Siu, W.-C. and Wang, Z. PUMPS: Skeleton-Agnostic Point-based Universal Motion Pre-Training for Synthesis in Human Motion Tasks. ICCV 2025, pp. 14496 to 14506. arXiv:2507.20170.
- Zhao, H., Wang, H., Zhao, X., Fei, H., Wang, H., Long, C. and Zou, H. PhysSplat: Efficient Physics Simulation for 3D Scenes via MLLM-Guided Gaussian Splatting. ICCV 2025, pp. 5242 to 5252. arXiv:2411.12789.
TAR, EgoX, Gaussian Scene Graph and work on human mesh recovery (Nie et al., NeurIPS 2024 and ECCV 2024) were cited on the slides without full references.
The account of the panel is based on notes taken during the event and photographs from the room, and statements have not been reviewed by the speakers.