OpenAI Tests AI-Designed Hardware as Engineering Benchmarks Expose Limits
OpenAI describes a device designed with GPT-6 Astra, while a separate Google Cloud benchmark shows how far agents remain from reliable end-to-end engineering.
Published
Last updated / Fact-checked on
OpenAI said on November 3 that GPT-6 Astra designed a device for talking to Codex, including its enclosure, circuit board and firmware, "without any direct human edits," in under a week. Making it manufacturing-ready still took several rounds of review. The same day, Google Cloud's new engineering benchmark found every frontier model it tested scored below 50%.
Photo: This_is_Engineering / Pixabay
At a glance
| Device design | Enclosure, PCB, firmware by Astra in under a week |
| Prototypes | Small batch built by a contract manufacturer in under a month |
| Known gap | Missed production limits, tolerances, assembly details |
| Hardware-team token use | Up 23x since February |
| Partners | Chip Ganassi Racing (scan-to-CAD), Mazda (aero, crash safety) |
| Google PLE benchmark | 85 tasks; best model 48.3% |
From generated designs to manufactured prototypes
According to OpenAI, Astra designed the device’s appearance, enclosure and internal structures. It also selected electrical components and designed the printed circuit board, including component placement and connections that fit inside the enclosure. Astra generated the user interface, firmware and software integrations connecting the device to Codex.
OpenAI said two team members worked with Astra to design the device in under a week as a side project. A contract manufacturer then built a small batch of prototypes in under a month, according to the company.
However, OpenAI said making the design manufacturing-ready required several rounds of review and iteration. Astra did not consistently account for the manufacturer’s production limits, tolerances or assembly details. That qualification distinguishes the company’s claim about generating the design without direct human edits from the subsequent work needed to manufacture it.
Engineering teams test broader workflows
OpenAI said model usage across its hardware teams, measured in tokens, has grown 23-fold since February. The company also reported progress on BenchCAD 2.0, which evaluates reconstructing three-dimensional parts and assemblies and identifying circuit-board connections.
On that benchmark, OpenAI said a newer internal model averaged 0.875, compared with 0.594 for GPT-6 Astra at its highest reasoning setting. Those figures describe performance on BenchCAD’s tasks, rather than the separate end-to-end engineering evaluation announced by Google Cloud.
OpenAI said it is expanding work with Chip Ganassi Racing on workflows that convert three-dimensional scans of existing parts into editable computer-aided design models and two-dimensional manufacturing drawings. CGR engineers review the outputs, according to OpenAI.
“The model has been useful for going from raw scan data to a solid body,” CGR research and development engineer Matthew Ortman said in OpenAI’s announcement.
OpenAI also announced a research collaboration with Mazda’s aerodynamics and crash-safety engineers involving airflow and structural simulations. According to OpenAI, Astra is modeling nose shapes to reduce drag; the company said it would share results in coming months.
Independent testing shows the remaining gap
Google Cloud’s announcement, authored by Burak Gokturk, Rujun Han and George Lee, describes PLE as a benchmark and open competition containing 85 long-horizon tasks: 30 mechanical, 27 electrical, 18 civil and 10 structural engineering tasks. Each task contains an average of 61 files.
Google Cloud said it released 29 public tasks and grading tools, alongside a leaderboard at ple-bench.com. Its evaluation tested 14 frontier models across seven harnesses, the software environments used to run agents, with the best private-test result reaching 48.3%.
The distinction between partial correctness and acceptable deliverables was particularly pronounced. Google Cloud said the top configuration satisfied 91% of individual constraints on co-authored tasks, but only 41.3% of its deliverables passed every hard safety and feasibility check. Designs earn points only after passing every hard safety check, according to Google Cloud.
For readers, the two announcements address different questions. OpenAI describes assistance across design and prototyping, including manufacturing revisions; Google Cloud measures whether agents complete engineering assignments under strict acceptance criteria. PLE offers independent context, not a reported score for OpenAI’s featured model. Together, the accounts distinguish useful assistance from consistently acceptable finished engineering work.
What to watch
- Mazda results. OpenAI says it will share progress in the coming months.
- Manufacturing gaps. Whether newer models handle tolerances and assembly details.
- PLE leaderboard. How models rank as submissions arrive at ple-bench.com.
- New collaborators. OpenAI is recruiting engineering firms and software vendors.
More coverage is on our Artificial Intelligence hub.
How we reported this
This article draws on OpenAI's and Google Cloud's own posts; all figures are the companies' and were not independently tested. PLE is a separate benchmark; OpenAI's featured model was not reported as scored on it.
Published and fact-checked November 3, 2026.
Sources
- OpenAI, "Accelerating engineering for the physical world," Nov. 3, 2026: https://openai.com/index/accelerating-engineering-for-the-physical-world
- Google Cloud, Burak Gokturk, Rujun Han and George Lee, "Introducing Physical-Engineering's Last Exam for agents," Nov. 3, 2026: https://cloud.google.com/blog/products/ai-machine-learning/introducing-physical-engineerings-last-exam-for-agents
DailyMech used AI tools to help research and draft this article; it was fact-checked by our editors.