Add simple scene graph
This commit is contained in:
291
SCENE_SERIALIZATION.md
Normal file
291
SCENE_SERIALIZATION.md
Normal file
@@ -0,0 +1,291 @@
|
||||
# Scene Serialization Implementation Summary
|
||||
|
||||
## Overview
|
||||
Implemented a dual-format scene serialization system for GhostEngine:
|
||||
- **Binary format** for runtime (AOT-compatible, fast)
|
||||
- **JSON format** for editor (reflection-based, human-readable)
|
||||
|
||||
Both formats support automatic Entity reference remapping.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Two Serialization Paths
|
||||
|
||||
#### **Runtime Path (Ghost.Engine)** - AOT-Compatible
|
||||
```
|
||||
SceneManager → SceneBinarySerializer → SerializationContext
|
||||
```
|
||||
- Binary format using direct memory operations (`memcpy`)
|
||||
- No reflection, no System.Text.Json dependency
|
||||
- Fast, compact, suitable for shipping builds
|
||||
- Synchronous operations
|
||||
|
||||
#### **Editor Path (Ghost.Editor.Core)** - Reflection-Based
|
||||
```
|
||||
EditorSceneManager → SceneSerializer → EntityJsonConverter → SerializationContext
|
||||
```
|
||||
- JSON format using System.Text.Json with reflection
|
||||
- Human-readable, debuggable
|
||||
- Automatic Entity remapping via custom converter
|
||||
- Async operations
|
||||
|
||||
### Core Components
|
||||
|
||||
#### 1. **SerializationContext.cs** (Ghost.Engine/IO)
|
||||
- **Shared by both runtime and editor**
|
||||
- Thread-safe context using `AsyncLocal<T>` for managing Entity ID remapping
|
||||
- Maps file-local IDs (0, 1, 2...) to runtime Entity instances
|
||||
- Bidirectional mapping for both serialization and deserialization
|
||||
- Usage pattern:
|
||||
```csharp
|
||||
using var context = SerializationContext.Create();
|
||||
context.RegisterEntity(fileId, runtimeEntity);
|
||||
```
|
||||
|
||||
#### 2. **SceneBinarySerializer.cs** (Ghost.Engine/IO)
|
||||
- **Runtime binary serialization** - AOT-compatible
|
||||
- Static utility class with synchronous methods
|
||||
- **Serialize**: Writes entities to BinaryWriter using raw memory operations
|
||||
- Format: Magic number (0x47534345 "GSCE"), version, entity count, component data
|
||||
- Uses `memcpy` for component data - zero reflection
|
||||
- Implements Entity reference remapping for Hierarchy component
|
||||
- **Deserialize**: Two-pass loading strategy
|
||||
- **Pass 1**: Create all entities, build ID mapping
|
||||
- **Pass 2**: Read and copy component data, remap Entity references
|
||||
- **RemapEntityReferences**: Manual remapping for components with Entity fields (currently Hierarchy)
|
||||
|
||||
#### 3. **EntityJsonConverter.cs** (Ghost.Editor.Core/Serializer/Converters)
|
||||
- **Editor-only** custom `JsonConverter<Entity>`
|
||||
- Automatically remaps Entity references during JSON serialization/deserialization
|
||||
- During **serialization**: Writes file-local ID from SerializationContext
|
||||
- During **deserialization**: Reads file-local ID and translates to runtime Entity
|
||||
- Enables deep Entity reference remapping in nested components (e.g., Hierarchy)
|
||||
|
||||
#### 4. **SceneSerializer.cs** (Ghost.Editor.Core/Serializer)
|
||||
- **Editor-only** static utility class for JSON scene file I/O
|
||||
- **SaveSceneAsync**: Queries entities by SceneID, serializes components using reflection
|
||||
- **LoadSceneAsync**: Two-pass loading strategy with automatic Entity remapping
|
||||
- **Pass 1**: Create all entities, build ID mapping
|
||||
- **Pass 2**: Deserialize components with automatic Entity remapping via EntityJsonConverter
|
||||
- File format: JSON with entities array containing component type names and data
|
||||
|
||||
#### 5. **Scene.cs** (Ghost.Engine/Core)
|
||||
- Lightweight handle class with World reference, SceneID, and Name
|
||||
- No longer owns the World - respects "database pattern"
|
||||
- Constructor: `Scene(World world, string name)`
|
||||
- Implements IDisposable and IEquatable
|
||||
|
||||
#### 6. **SceneManager.cs** (Ghost.Engine/Services)
|
||||
- **Runtime scene lifecycle manager** - uses binary serialization
|
||||
- **LoadScene**: Synchronous, loads from binary file, supports Single/Additive modes
|
||||
- **SaveScene**: Synchronous, saves scene to binary file
|
||||
- **UnloadScene**: Efficiently destroys all entities with matching SceneID
|
||||
- Maintains registry of loaded scenes per World
|
||||
|
||||
#### 7. **EditorSceneManager.cs** (Ghost.Editor.Core/SceneGraph)
|
||||
- **Editor scene lifecycle manager** - uses JSON serialization
|
||||
- **SaveSceneAsync**: Asynchronous, saves scene to JSON file
|
||||
- Integrates with editor workflows and UI
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
### 1. Dual Serialization Formats
|
||||
- **Binary for Runtime**: Fast, compact, AOT-compatible for shipping builds
|
||||
- No reflection or System.Text.Json dependency in Ghost.Engine
|
||||
- Direct memory operations using unsafe pointers
|
||||
- Synchronous operations suitable for runtime loading
|
||||
- **JSON for Editor**: Human-readable, debuggable, reflection-based
|
||||
- Located in Ghost.Editor.Core (not in runtime path)
|
||||
- Async operations for editor workflows
|
||||
- Automatic Entity remapping via custom JsonConverter
|
||||
|
||||
### 2. World-Centric Architecture
|
||||
- World is the data container (the "database")
|
||||
- Scene is a lightweight handle/view into that data
|
||||
- SceneManager orchestrates the I/O and entity management
|
||||
- Respects separation of concerns: World doesn't know about scenes
|
||||
|
||||
### 3. Component Tagging
|
||||
- Uses `SceneID` component (currently IComponent, ready for ISharedComponent upgrade)
|
||||
- Each entity stores its scene membership
|
||||
- Enables efficient querying and batch operations
|
||||
|
||||
### 4. Entity Reference Remapping
|
||||
- "Smart Serializer" strategy with two-pass loading
|
||||
- File uses sequential IDs (0, 1, 2...)
|
||||
- Runtime creates new Entities with different IDs
|
||||
- SerializationContext handles the translation
|
||||
- **Binary format**: Manual remapping in `RemapEntityReferences` method
|
||||
- **JSON format**: Automatic remapping via `EntityJsonConverter`
|
||||
- Works for Hierarchy and any other component with Entity fields
|
||||
|
||||
### 5. AOT Compatibility
|
||||
- Ghost.Engine has zero reflection-based serialization
|
||||
- All JSON/reflection code isolated to Ghost.Editor.Core
|
||||
- Binary serializer uses only unsafe pointers and memcpy
|
||||
- Suitable for IL2CPP and NativeAOT compilation
|
||||
|
||||
## Binary Format Specification
|
||||
|
||||
```
|
||||
Header:
|
||||
4 bytes: Magic number (0x47534345 "GSCE")
|
||||
4 bytes: Version number (int32)
|
||||
4 bytes: Entity count (int32)
|
||||
|
||||
For each entity:
|
||||
4 bytes: File ID (int32)
|
||||
4 bytes: Component count (int32)
|
||||
|
||||
For each component:
|
||||
4 bytes: Component Type ID (int32)
|
||||
4 bytes: Component Size (int32)
|
||||
N bytes: Raw component data (memcpy from archetype)
|
||||
```
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Runtime Usage (Binary)
|
||||
```csharp
|
||||
// Create a world
|
||||
var world = World.Create();
|
||||
|
||||
// Load a scene additively (synchronous)
|
||||
var scene = SceneManager.LoadScene(world, "path/to/scene.bin", SceneLoadMode.Additive);
|
||||
|
||||
// Save the scene (synchronous)
|
||||
SceneManager.SaveScene(scene, "path/to/scene.bin");
|
||||
|
||||
// Unload the scene
|
||||
SceneManager.UnloadScene(scene);
|
||||
```
|
||||
|
||||
### Editor Usage (JSON)
|
||||
```csharp
|
||||
// In editor code
|
||||
var world = World.Create();
|
||||
|
||||
// Save scene to JSON (async)
|
||||
await EditorSceneManager.SaveSceneAsync(scene, "path/to/scene.json");
|
||||
|
||||
// JSON is human-readable and can be version-controlled
|
||||
```
|
||||
|
||||
## Future Optimizations
|
||||
|
||||
### When ISharedComponent is Available
|
||||
- Change `SceneID` from `IComponent` to `ISharedComponent`
|
||||
- Entities with same SceneID will be grouped in same chunks
|
||||
- Unloading becomes O(chunks) instead of O(entities)
|
||||
- Can free entire memory blocks instead of individual entities
|
||||
|
||||
### Entity Remapping Source Generator
|
||||
- Currently `RemapEntityReferences` in `SceneBinarySerializer` manually handles Hierarchy
|
||||
- Could implement a source generator to automatically detect Entity fields in all components
|
||||
- Would eliminate need for manual per-component remapping code
|
||||
- Pattern: `[SerializableEntity]` attribute on fields containing Entity references
|
||||
|
||||
### Compression
|
||||
- Binary format is uncompressed raw data
|
||||
- Could add optional compression (LZ4, Zstandard) for smaller file sizes
|
||||
- Trade-off: loading time vs disk space
|
||||
|
||||
## Files Modified/Created
|
||||
|
||||
### Created in Ghost.Engine (Runtime)
|
||||
- `Ghost.Engine/IO/SerializationContext.cs` - Shared ID remapping context
|
||||
- `Ghost.Engine/IO/SceneBinarySerializer.cs` - AOT-compatible binary serialization
|
||||
- `Ghost.Engine/Components/SceneID.cs` - Scene tagging component
|
||||
|
||||
### Created in Ghost.Editor.Core (Editor)
|
||||
- `Ghost.Editor.Core/Serializer/SceneSerializer.cs` - JSON serialization (moved from Ghost.Engine)
|
||||
- `Ghost.Editor.Core/Serializer/Converters/EntityJsonConverter.cs` - Entity remapping for JSON (moved from Ghost.Engine)
|
||||
|
||||
### Modified
|
||||
- `Ghost.Engine/Core/Scene.cs` - Refactored to lightweight handle
|
||||
- `Ghost.Engine/Services/SceneManager.cs` - Runtime scene lifecycle with binary serialization
|
||||
- `Ghost.Editor.Core/SceneGraph/EditorSceneManager.cs` - Editor scene lifecycle with JSON serialization
|
||||
|
||||
### Deleted
|
||||
- `Ghost.Engine/IO/SerializerRegistry.cs` - Obsolete ComponentSerializerRegistry
|
||||
- `Ghost.Editor.Core/Serializer/SceneNodeSerializer.cs` - Obsolete
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
### Binary Serialization Details
|
||||
- Uses `BinaryWriter`/`BinaryReader` for primitive types (int, etc.)
|
||||
- Component data copied with `Unsafe.CopyBlock` (memcpy equivalent)
|
||||
- Stackalloc buffer reused for zero-filled missing components (prevents stack overflow)
|
||||
- Entity remapping performed after all entities created (two-pass loading)
|
||||
|
||||
### JSON Serialization Details
|
||||
- Uses `System.Text.Json` with `JsonSerializerOptions`
|
||||
- `EntityJsonConverter` registered as custom converter
|
||||
- Automatic Entity field detection and remapping during deserialization
|
||||
- Human-readable format suitable for version control
|
||||
|
||||
### Thread Safety
|
||||
- `SerializationContext` uses `AsyncLocal<T>` for thread-safe context isolation
|
||||
- Binary serializer is not thread-safe (single-threaded runtime loading)
|
||||
- JSON serializer uses async methods but should not be called concurrently for same World
|
||||
|
||||
### Error Handling
|
||||
- Missing components write zero-filled data (graceful degradation)
|
||||
- Unknown component types in JSON are skipped with warning
|
||||
- Invalid Entity references remap to Entity.Null
|
||||
- File format version checked on load (future-proofing)
|
||||
|
||||
## Known Limitations
|
||||
|
||||
1. **Manual Entity Remapping in Binary Format**
|
||||
- Currently only Hierarchy component is remapped
|
||||
- Other components with Entity fields need manual handling
|
||||
- Solution: Implement source generator for automatic detection
|
||||
|
||||
2. **Component Size Limit**
|
||||
- Binary serializer uses 4KB stackalloc buffer for zero-fills
|
||||
- Components larger than 4KB will throw exception if missing
|
||||
- Solution: Increase MaxComponentSize constant if needed
|
||||
|
||||
3. **SceneNode Integration**
|
||||
- Legacy SceneNode class in Ghost.Editor.Core still exists
|
||||
- May need integration with new Scene/SceneSerializer system
|
||||
- Future work: Decide on SceneNode vs Scene unification
|
||||
|
||||
4. **No Compression**
|
||||
- Binary format is uncompressed
|
||||
- Large scenes may have bigger file sizes than necessary
|
||||
- Future optimization: Add LZ4/Zstandard compression layer
|
||||
|
||||
5. **Managed Components**
|
||||
- Current implementation assumes all IComponent types are unmanaged
|
||||
- ScriptComponent and ManagedEntity may need separate handling
|
||||
- Future work: Add managed reference serialization
|
||||
|
||||
## Testing Recommendations
|
||||
|
||||
1. **Binary Format Round-Trip**
|
||||
- Create entities with various components
|
||||
- Save to binary file
|
||||
- Load into new World
|
||||
- Verify all component data matches
|
||||
|
||||
2. **Entity Reference Remapping**
|
||||
- Create parent-child hierarchies
|
||||
- Serialize and deserialize
|
||||
- Verify parent/child Entity references updated correctly
|
||||
|
||||
3. **Additive Loading**
|
||||
- Load multiple scenes into same World
|
||||
- Verify SceneID tagging works correctly
|
||||
- Unload specific scenes and verify entities destroyed
|
||||
|
||||
4. **JSON Compatibility**
|
||||
- Save same scene to both JSON and binary
|
||||
- Verify both formats produce equivalent results when loaded
|
||||
- Test JSON editing by hand (human-readable requirement)
|
||||
|
||||
5. **AOT Compatibility**
|
||||
- Build Ghost.Engine with NativeAOT or IL2CPP
|
||||
- Verify no reflection or dynamic code generation warnings
|
||||
- Test binary serialization in AOT-compiled build
|
||||
Reference in New Issue
Block a user