A shared 3D world where humans and AI agents collaborate through actions, not just conversation.
Explore SOLVI.help · Watch the demo · Project source
SOLVI.help is a browser-based, multiplayer 3D world where humans and AI-powered robot avatars explore, communicate, and build together. Instead of remaining inside a chat window, the agents have virtual bodies, head-mounted cameras, persistent task context, and tools for interacting with a physics-enabled environment.
The project explores embodied AI in simulation, powered by Meta's multimodal models. It is an experimental prototype, not a claim of fully reliable general-purpose autonomy.
- Meet the agents: Jess, Zuck, Suck, and Elox can converse with people and other robots, receive instructions, and act in the same shared realm.
- See through a robot's eyes: virtual head cameras capture actual 3D rendered observations, including 360° scans, for multimodal analysis.
- Give meaningful tasks: navigation, object interaction, pick-up/delivery, furniture placement, and multi-step construction experiments.
- Build a world: procedural homes, stairs, roads, fences, vegetation, screens, appliances, and interactive equipment can be placed and edited.
- Use interactive props: laptops have hinged screens and media playback; fridges open and store supported small objects; tools and office equipment can be configured as procedural primitives. While controlling an avatar, press I to pick up or drop a held tool, U to use it, and B to toggle Inventory. Equipped tools retain independently adjustable grips for each hand. Shovels can mark an excavation site with an optional tool-use animation; physical terrain deformation is not yet supported.
- Share and persist: multiplayer scene updates, saved realms, object presets, and server-side persistence let participants work in the same space.
- Explore in a browser: Three.js/WebGL rendering, Rapier physics, and experimental WebXR support.
- Perception: the app captures virtual camera frames and sends selected imagery and structured scene context to Meta's multimodal services.
- Reasoning: Meta Muse Spark (through the Meta Responses API) supports conversation, scene understanding, and proposed next actions.
- Action: SOLVI validates the proposed action against its allowlisted tools, permissions, and world state; the scene and physics engines carry it out.
Important distinction: Meta provides reasoning and vision. SOLVI supplies the 3D renderer, physics, pathfinding, state persistence, tool execution, and multiplayer networking. Model plans can fail or fall back to deterministic logic; autonomous work is still experimental.
| Area | Stack |
|---|---|
| Client | JavaScript, Three.js, WebGL, WebXR |
| Physics and navigation | Rapier, scene-aware pathfinding |
| AI and vision | Meta Muse Spark, Meta Responses API |
| Server and realtime | Python Flask, WebSockets |
| Storage | SQLite |
| Video and media | HTML video, HLS where supported |
| Hosting | Ubuntu, Nginx, Gunicorn |
| Validation | Node.js tests, Python test suite, versioned SHA-256 release manifests |
cv-meta-hackathon-solvi-help/
├── README.md # Public project introduction (this page)
├── app/
│ ├── README.md # App-scoped project overview
│ ├── ABOUT.md # What the project is and its current scope
│ ├── CHANGELOG.md # Historical release and development notes
│ ├── index.html # Browser app
│ ├── js/ # World, editor, networking, robots, AI tools
│ ├── backend/ # Flask API, SQLite, WebSockets, installer
│ ├── assets/ # Bundled models and interface icons
│ ├── docs/ # Specifications and technical handover
│ ├── tests/ # Browser-code regression tests
│ └── package.json
└── updates/ # Local watcher input; not application source
Requirements: Node.js, npm, Python 3, and a modern browser with WebGL support. The real Meta API integration requires credentials, supplied securely and never committed to source control.
cd app
npm install
npm run vendor
python3 -m venv .venv
source .venv/bin/activate
pip install -r backend/requirements.txt
export SOLVI_DATA_DIR="$(pwd)/.local-data"
python3 backend/server.pyOpen http://127.0.0.1:8765/. If the project is already running, do not start a second local server on the same port.
Run the JavaScript regression suite from app/:
npm testThe application server and browser are separate from the test harness. Some Python integration tests require additional dependencies and a properly isolated test environment. Passing automated tests alone does not establish end-to-end robot autonomy, media compatibility, or two-client production synchronisation.
- Live application: https://solvi.help/
- Hackathon demonstration: https://x.com/sylwestermiel/status/2118125153318027329
- About and limitations:
ABOUT.md - Release history:
CHANGELOG.md - Implementation specifications:
docs/
Current packaged source release: v0.2.213. The latest packaged release is not considered live until the existing deployment watcher verifies and deploys it.
This is an evolving research and creative prototype. AI can misunderstand images or instructions, navigation can be obstructed, external media and Meta services can be unavailable, and robots do not perform long-running independent work while the controlling browser is closed. World edits are permission-checked; deployments must preserve existing realms and data. Do not commit API keys, user content, or production databases.
SOLVI.help explores a future where humans and AI agents share a world and collaborate through actions, not just words.
When adding a primitive, tool, furniture item or saved preset, SOLVI searches for a visible, unobstructed floor position near the last object added. It avoids room walls and existing furniture, and asks the creator to adjust the camera if no suitable placement can be found. Manual transforms remain available.
In the avatar Inventory → Held, select Grip on an equipped tool (or Switch hand to move it to the other free hand). The editor displays per-hand bone, position, rotation and handle contact-point controls. The saved values follow that tool in multiplayer and in presets. Tools do not yet perform terrain excavation or other tool-specific world actions.