Decentralized Trust for Spatial AI
Sky3 is building an independent benchmark for spatial AI. Engineers, hobbyists and researchers contribute edge-case data from the physical world—the hard cases models fail on—the community verifies it, and AI models are scored in public so vendors stop grading their own homework.
Why spatial AI needs a neutral referee
Models that reason about the physical world are moving from research into robots, factories and buildings. Today, the claims about how well they work mostly come from the people who built them, measured on benchmarks they chose.
Simulation does not capture everything. A model that scores well on synthetic scenes can still misread a cluttered shelf, a non-standard fastener or a structure that only holds together because of one hidden joint. These gaps show up when a system is deployed, and that is the most expensive place to find them.
Sky3 proposes a simple alternative: test models on real-world cases that the model's builders did not pick, and publish the results.
- Independent. Benchmarks are run by the network, not by the teams whose models are being scored.
- Grounded in the real world. Test cases come from physical scans, drawings and recordings, with a focus on rare and difficult cases.
- Verified. Every contributed case is checked by automated tests and by human reviewers before it is used for scoring.
- Reproducible. Metrics, scoring code and evaluation splits are published so anyone can check a result.
- Private by default. Submissions are stripped of metadata, faces and identifying information before they enter the archive.
- Community-governed. The test suite and the rules for changing it are governed by contributors and reviewers through an open process, stewarded by a non-profit foundation.
This is a draft. The protocol, metrics and governance model will change based on feedback from the beta.
What a public scorecard looks like
The leaderboard will rank spatial and physical-world models on verified real-world cases. The table below shows the format only—all model names and scores are fictional to illustrate how results will be presented.
All model names and scores are invented for illustration. No real model has been evaluated.
What you'd upload, and why it matters
The benchmark will be built from edge cases that break current models. If you work with spatial data, you likely have examples that models consistently fail on.
LiDAR and point clouds
Scans with reflective surfaces, glass facades, or extreme weather artifacts. Cases where reconstruction algorithms misread structure or merge unconnected parts.
CAD and 3D models
Assemblies with non-standard fasteners, interference fits, or parts that only work in one configuration. Structures that look plausible but can't hold load.
Robot trajectories and logs
Paths through cluttered spaces, grasps that fail on ambiguous geometry, or pick sequences that ignore collision. Failure cases from real deployments.
Geometry imagery
Photos of structures with hidden joints, transparent or mirrored materials, ambiguous depth cues, or lighting that breaks segmentation. Occlusion and multi-body contact.
Nothing is live yet. The upload pipeline is on the roadmap. Joining the waitlist means you'd be among the first to contribute data when the system opens, helping shape what the benchmark measures and how contributors are credited.
From real-world data to a public score
Contribute real-world edge cases
Contributors will submit spatial data that models find hard: LiDAR scans, CAD files, robot trajectories and images of real geometry. Rare and complex cases are worth more. Accepted submissions will earn points that record each contributor's share of the archive.
- Planned formats: point clouds (.las, .laz, .ply), CAD (.step, .stl, .obj), trajectories (.csv, .json), images (.jpg, .png).
- Files will be sanitized on intake: metadata stripped, faces and personal information removed.
Verify every case before it counts
Each submission will go through automated checks for format, duplication and quality, then review by independent human verifiers. A case will enter the benchmark only when reviewers agree on its labels. Reviewers will earn points for accurate reviews.
- Automated checks: file integrity, duplicate detection, label consistency.
- Human review: multiple independent reviewers per case, agreement required.
Score models in public
Verified cases will form held-out evaluation sets. Models will be scored on geometric accuracy, structural understanding and physical reasoning, and results will be published on the Spatial Leaderboard with the method used to produce them.
- Metrics include IoU, structural integrity, assembly tree accuracy and physical logic (defined in the leaderboard).
- Model developers will be able to submit through a sandbox API (planned).
Points are a record of contribution during the beta. They have no monetary value, and any future use will be decided through the governance process.
Be among the first to contribute
The upload pipeline isn't built yet. Join the waitlist and we'll email you when it opens. You'll be among the first to upload spatial data and help shape the benchmark.
You're on the list.
We'll email when the beta opens. No other emails before then.
Something went wrong saving your details. Please try again.