RoboSet Dataset

Introducing RoboSet, a large-scale real-world multi-task dataset collected across a range of everyday household activities in kitchen scenes. RoboSet consists of a mix of kinesthetic demonstrations as well as teleoperated demonstrations. The dataset consists of multi-task activities with four different camera views per frame, and variations in the scene for every demonstration.

Dataset composition

The dataset consists of various setups and features

1. Source of dataset

   • The dataset has been collected from multiple sources, such as kinesthetic demonstrations and teleoperation. Kinesthetic demonstrated data was collected by playing back a demo trajectory with a new scene obtained by re-arranging objects, for every rollout. The teleoperated data was collected using an Oculus Quest 2 controller. The teleoperator would use the controller to guide the robot to perform the task; teleoperation ensured that each rollout was unique. Overall we have 28,500 trajectories in the dataset, out of which 9,500 are collected through teleoperation, and 19,000 are collected through kinesthetic playback.

2. Camera viewpoints

   • We collected the dataset with 4 distinct camera views per frame. This could be helpful for downstream policy learning where the visual representations are not tied to a specific viewpoint, and are robust to viewpoint variations.

3. Different Tasks

   • We categorize all the tasks in terms of semantic activities, such that each activity consists of 4-6 tasks. This could be helpful in developing multi-task multi-step agents that can chain different skills together to solve the entire sequence of tasks in an activity.
   • Since each task is defined through a language instruction, there could be opportunities for language guided sequencing, and generalization with LLMs.

The chart below represents the composition of RoboSet where each bar represents the type of data collected (Teleoperation, Kinesthetic, Autonomous). Use the menu above or the buttons below to navigate to their respective pages to download the dataset.

RoboSet Data Composition

RoboSet Data Schema

              
                  import h5py
                  path = "Path/to/dataset.h5"
                  h5 = h5py.File(path, 'r')
                  h5.keys() #Outputs Trials per h5, Trial 0, 1, 2, ...
                  h5['Trial0'].keys() # outputs data, derived, and config
                  # to extract the data
                  h5["Trial0"]['data'][data_key] #where data_key is one of the cells from the data tab