meet legoGPT, the ‘chatGPT’ for your own LEGO designs

Meet LegoGPT, the ‘chatGPT’ for custom LEGO designs

Using text prompts, users can generate their own buildable LEGO designs à la OpenAI’s ChatGPT with LegoGPT. Designed by the researchers at Carnegie Mellon University, it can produce physically stable LEGO brick models based on text inputs. The AI language model uses a dataset of LEGO structures paired with descriptive captions. It predicts where to put the individual LEGO bricks using next-token prediction. After it gives the results, users just follow the ‘design plan’ to start building their generated models using their existing bricks or produce their custom bricks for the design. LegoGPT even uses a validity check to make sure that each LEGO brick is in the right place.

The ChatGPT-like dataset used to train LegoGPT, called StableText2Lego, includes more than 47,000 LEGO structures. These represent more than 28,000 3D objects. Then, each structure comes with a caption that describes the model in natural language. The ChatGPT for LEGO also includes a feature that assigns colors and surface textures to the buildable structures still based on the text prompts. Another feature noted by the researchers is that the project can be assembled by either a human or a robot (arm).

all images courtesy of the researchers from Carnegie Mellon University

researchers convert the real brick models into ‘digital’ formats

The ChatGPT for LEGO starts when the researchers convert and scan the real models into digital formats, so the computers can analyze and save them. They do this by turning each design into a sequence of text tokens. Basically, these are written descriptions of each LEGO brick, ordered from bottom to top like poetry stanzas. Then, these sequences are paired with descriptions of what the models represent. All this forms a training dataset, which is then used to teach a language model how to predict LEGO brick layouts.

When users give LegoGPT a prompt, it begins building the model one brick at a time. After placing each brick, it checks if the brick is valid, doesn’t overlap with others, and fits with the rest of the structure. It’s the phase when the system generates different structures, and each variation goes through stability analysis. Because of this, the ChatGPT for LEGO removes the designs that aren’t physically ‘stable’ from the dataset, such as parts that are weak and could fall apart in the process. The full dataset, trained models, and source code of LegoGPT are available for public use and download. The researchers – composed of Ava Pun, Kangle Deng, Ruixuan Liu, Deva Ramanan, Changliu Liu, and Jun‑Yan Zhu of Carnegie Mellon University – have also uploaded the demo site for the users to try.

the AI language model uses a dataset of LEGO structures paired with descriptive captions