๐Ÿš€ Matrix Dataset Documentation

Last Updated: 2025-03-25

๐ŸŒ Project Page ๐Ÿ“„ Research Paper ๐Ÿ“ Dataset Docs ๐Ÿ“ฆGitHub

๐Ÿ“Š Matrix Dataset

The Matrix dataset was first introduced by the Matrix team in the paper "The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control." This dataset is specifically designed for training world models and comprises millions of video sequences accompanied by corresponding control signals.

๐ŸŽ๏ธ Forza Horizon 5

๐Ÿ–ผ๏ธ Visual Content

  • Multi-scene driving sequences
  • Desert/Ocean/Grassland biomes
  • Dynamic weather system

๐Ÿ“ฆ Dataset Specs

  • 1.2M video-control pairs
  • Standardized 6-second clips
  • Resolution: 2560ร—1600 @ 60FPS

๐ŸŒ† Cyberpunk 2077

๐Ÿ–ผ๏ธ Visual Content

  • Dense urban landscapes
  • Day-night cycle (1:3 ratio)
  • Indoor-outdoor transitions

๐Ÿ“ฆ Dataset Specs

  • 1.0M video-control pairs
  • Standardized 6-second clips
  • Resolution: 2560ร—1600 @ 60FPS

โš™๏ธ Data collection&processing

๐ŸŽฎ Forza Horizon 5 Data Collection & Processing

Platform Construction

  • Automation: Game launching, screen recording, and control signal generation
  • Telemetry Extraction: Real-time positions (XYZ), velocities, and accelerations
  • Stagnation Detection:
    • Trigger movement if player is stagnant for over 2 seconds
    • Teleport if player remains within an 80m radius for over 40 seconds

Data Collection

  • Control Signals: Randomized forward, left, and right movements
  • Biomes: Data collected from woods, grass, sea, field, river, etc. (distribution: 12%-15%-18%-16%-15%-9%-15%)
  • Screen Recording: Continuous at native resolution with GUI removed
  • Telemetry Logging: Collision detection and motion analysis

Data Processing

  • Preprocessing: Clip segmentation and labeling (scene type, motion type, biome)
  • Filtering:
    • Class balance adjustment
    • Collision detection using telemetry acceleration spikes
    • Stagnation removal and visual artifact filtering
๐ŸŽฎ Cyberpunk 2077 Data Collection & Processing

Platform Construction

  • Cloud Servers: Scalable recording environment with server image cloning
  • Visual Enhancement: Reshade-based improvements in lighting, shadow, and texture sharpness
  • Control Signal Tracking: Socket-based virtual input emulation

Data Collection

  • Manual Gameplay: Annotators using low-definition streams
  • Operations: Forward movement and camera adjustments (U/D/L/R keys)
  • NPC and Vehicle Mods: Disabled to reduce scene complexity
  • Coordinates Tracking: Cheat Engine for collision checking

Data Processing

  • Preprocessing: Clip segmentation and labeling (scene type: urban indoor/outdoor, day/night)
  • Filtering:
    • Stagnation detection
    • Visual artifact removal (lighting glitches, texture corruption)
    • Collision filtering based on coordinate jumps

๐Ÿ‘ฅ Research Team

Core Members

  • โ€ข Ruili Feng - Tongyi Lab
  • โ€ข Han Zhang - Tongyi Lab
  • โ€ข Zhantao Yang - Tongyi Lab
  • โ€ข Jie Xiao - Tongyi Lab
  • โ€ข Zhilei Shu - Tongyi Lab
  • โ€ข Zhiheng Liu - Tongyi Lab
  • โ€ข Andy Zheng - Waterloo Univ
  • โ€ข ShangWen Zhu - SJTU Univ
  • โ€ข Yukun Huang - HKU
  • โ€ข Yu Liu - Tongyi Lab
  • โ€ข Hongyang Zhang - Vector

๐Ÿ“š Publications

@misc{feng2024matrixinfinitehorizonworldgeneration,
                title={The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control},
                author={Ruili Feng and Han Zhang and Zhantao Yang and Jie Xiao and Zhilei Shu and Zhiheng Liu and Andy Zheng and Yukun Huang and Yu Liu and Hongyang Zhang},
                year={2024},
                eprint={2412.03568},
                url={https://arxiv.org/abs/2412.03568}
            }

๐Ÿ”” Contact: https://matrixteam-ai.github.io/