GhostEngine/SCENE_SERIALIZATION.md

Scene Serialization Implementation Summary

Overview

Implemented a dual-format scene serialization system for GhostEngine:

• Binary format for runtime (AOT-compatible, fast)
• JSON format for editor (reflection-based, human-readable)

Both formats support automatic Entity reference remapping.

Architecture

Two Serialization Paths

Runtime Path (Ghost.Engine) - AOT-Compatible

SceneManager → SceneBinarySerializer → SerializationContext

• Binary format using direct memory operations (memcpy)
• No reflection, no System.Text.Json dependency
• Fast, compact, suitable for shipping builds
• Synchronous operations

Editor Path (Ghost.Editor.Core) - Reflection-Based

EditorSceneManager → SceneSerializer → EntityJsonConverter → SerializationContext

• JSON format using System.Text.Json with reflection
• Human-readable, debuggable
• Automatic Entity remapping via custom converter
• Async operations

Core Components

1. SerializationContext.cs (Ghost.Engine/IO)

• Shared by both runtime and editor
• Thread-safe context using AsyncLocal<T> for managing Entity ID remapping
• Maps file-local IDs (0, 1, 2...) to runtime Entity instances
• Bidirectional mapping for both serialization and deserialization
• Usage pattern:

  using var context = SerializationContext.Create();
  context.RegisterEntity(fileId, runtimeEntity);
  

2. SceneBinarySerializer.cs (Ghost.Engine/IO)

• Runtime binary serialization - AOT-compatible
• Static utility class with synchronous methods
• Serialize: Writes entities to BinaryWriter using raw memory operations
  • Format: Magic number (0x47534345 "GSCE"), version, entity count, component data
  • Uses memcpy for component data - zero reflection
  • Implements Entity reference remapping for Hierarchy component
• Deserialize: Two-pass loading strategy
  • Pass 1: Create all entities, build ID mapping
  • Pass 2: Read and copy component data, remap Entity references
• RemapEntityReferences: Manual remapping for components with Entity fields (currently Hierarchy)

3. EntityJsonConverter.cs (Ghost.Editor.Core/Serializer/Converters)

• Editor-only custom JsonConverter<Entity>
• Automatically remaps Entity references during JSON serialization/deserialization
• During serialization: Writes file-local ID from SerializationContext
• During deserialization: Reads file-local ID and translates to runtime Entity
• Enables deep Entity reference remapping in nested components (e.g., Hierarchy)

4. SceneSerializer.cs (Ghost.Editor.Core/Serializer)

• Editor-only static utility class for JSON scene file I/O
• SaveSceneAsync: Queries entities by SceneID, serializes components using reflection
• LoadSceneAsync: Two-pass loading strategy with automatic Entity remapping
  • Pass 1: Create all entities, build ID mapping
  • Pass 2: Deserialize components with automatic Entity remapping via EntityJsonConverter
• File format: JSON with entities array containing component type names and data

5. Scene.cs (Ghost.Engine/Core)

• Lightweight handle class with World reference, SceneID, and Name
• No longer owns the World - respects "database pattern"
• Constructor: Scene(World world, string name)
• Implements IDisposable and IEquatable

6. SceneManager.cs (Ghost.Engine/Services)

• Runtime scene lifecycle manager - uses binary serialization
• LoadScene: Synchronous, loads from binary file, supports Single/Additive modes
• SaveScene: Synchronous, saves scene to binary file
• UnloadScene: Efficiently destroys all entities with matching SceneID
• Maintains registry of loaded scenes per World

7. EditorSceneManager.cs (Ghost.Editor.Core/SceneGraph)

• Editor scene lifecycle manager - uses JSON serialization
• SaveSceneAsync: Asynchronous, saves scene to JSON file
• Integrates with editor workflows and UI

Key Design Decisions

1. Dual Serialization Formats

• Binary for Runtime: Fast, compact, AOT-compatible for shipping builds
  • No reflection or System.Text.Json dependency in Ghost.Engine
  • Direct memory operations using unsafe pointers
  • Synchronous operations suitable for runtime loading
• JSON for Editor: Human-readable, debuggable, reflection-based
  • Located in Ghost.Editor.Core (not in runtime path)
  • Async operations for editor workflows
  • Automatic Entity remapping via custom JsonConverter

2. World-Centric Architecture

• World is the data container (the "database")
• Scene is a lightweight handle/view into that data
• SceneManager orchestrates the I/O and entity management
• Respects separation of concerns: World doesn't know about scenes

3. Component Tagging

• Uses SceneID component (currently IComponent, ready for ISharedComponent upgrade)
• Each entity stores its scene membership
• Enables efficient querying and batch operations

4. Entity Reference Remapping

• "Smart Serializer" strategy with two-pass loading
• File uses sequential IDs (0, 1, 2...)
• Runtime creates new Entities with different IDs
• SerializationContext handles the translation
• Binary format: Manual remapping in RemapEntityReferences method
• JSON format: Automatic remapping via EntityJsonConverter
• Works for Hierarchy and any other component with Entity fields

5. AOT Compatibility

• Ghost.Engine has zero reflection-based serialization
• All JSON/reflection code isolated to Ghost.Editor.Core
• Binary serializer uses only unsafe pointers and memcpy
• Suitable for IL2CPP and NativeAOT compilation

Binary Format Specification

Header:
  4 bytes: Magic number (0x47534345 "GSCE")
  4 bytes: Version number (int32)
  4 bytes: Entity count (int32)

For each entity:
  4 bytes: File ID (int32)
  4 bytes: Component count (int32)
  
  For each component:
    4 bytes: Component Type ID (int32)
    4 bytes: Component Size (int32)
    N bytes: Raw component data (memcpy from archetype)

Usage Examples

Runtime Usage (Binary)

// Create a world
var world = World.Create();

// Load a scene additively (synchronous)
var scene = SceneManager.LoadScene(world, "path/to/scene.bin", SceneLoadMode.Additive);

// Save the scene (synchronous)
SceneManager.SaveScene(scene, "path/to/scene.bin");

// Unload the scene
SceneManager.UnloadScene(scene);

Editor Usage (JSON)

// In editor code
var world = World.Create();

// Save scene to JSON (async)
await EditorSceneManager.SaveSceneAsync(scene, "path/to/scene.json");

// JSON is human-readable and can be version-controlled

Future Optimizations

When ISharedComponent is Available

• Change SceneID from IComponent to ISharedComponent
• Entities with same SceneID will be grouped in same chunks
• Unloading becomes O(chunks) instead of O(entities)
• Can free entire memory blocks instead of individual entities

Entity Remapping Source Generator

• Currently RemapEntityReferences in SceneBinarySerializer manually handles Hierarchy
• Could implement a source generator to automatically detect Entity fields in all components
• Would eliminate need for manual per-component remapping code
• Pattern: [SerializableEntity] attribute on fields containing Entity references

Compression

• Binary format is uncompressed raw data
• Could add optional compression (LZ4, Zstandard) for smaller file sizes
• Trade-off: loading time vs disk space

Files Modified/Created

Created in Ghost.Engine (Runtime)

• Ghost.Engine/IO/SerializationContext.cs - Shared ID remapping context
• Ghost.Engine/IO/SceneBinarySerializer.cs - AOT-compatible binary serialization
• Ghost.Engine/Components/SceneID.cs - Scene tagging component

Created in Ghost.Editor.Core (Editor)

• Ghost.Editor.Core/Serializer/SceneSerializer.cs - JSON serialization (moved from Ghost.Engine)
• Ghost.Editor.Core/Serializer/Converters/EntityJsonConverter.cs - Entity remapping for JSON (moved from Ghost.Engine)

Modified

• Ghost.Engine/Core/Scene.cs - Refactored to lightweight handle
• Ghost.Engine/Services/SceneManager.cs - Runtime scene lifecycle with binary serialization
• Ghost.Editor.Core/SceneGraph/EditorSceneManager.cs - Editor scene lifecycle with JSON serialization

Deleted

• Ghost.Engine/IO/SerializerRegistry.cs - Obsolete ComponentSerializerRegistry
• Ghost.Editor.Core/Serializer/SceneNodeSerializer.cs - Obsolete

Implementation Notes

Binary Serialization Details

• Uses BinaryWriter/BinaryReader for primitive types (int, etc.)
• Component data copied with Unsafe.CopyBlock (memcpy equivalent)
• Stackalloc buffer reused for zero-filled missing components (prevents stack overflow)
• Entity remapping performed after all entities created (two-pass loading)

JSON Serialization Details

• Uses System.Text.Json with JsonSerializerOptions
• EntityJsonConverter registered as custom converter
• Automatic Entity field detection and remapping during deserialization
• Human-readable format suitable for version control

Thread Safety

• SerializationContext uses AsyncLocal<T> for thread-safe context isolation
• Binary serializer is not thread-safe (single-threaded runtime loading)
• JSON serializer uses async methods but should not be called concurrently for same World

Error Handling

• Missing components write zero-filled data (graceful degradation)
• Unknown component types in JSON are skipped with warning
• Invalid Entity references remap to Entity.Null
• File format version checked on load (future-proofing)

Known Limitations

1. Manual Entity Remapping in Binary Format

   • Currently only Hierarchy component is remapped
   • Other components with Entity fields need manual handling
   • Solution: Implement source generator for automatic detection

2. Component Size Limit

   • Binary serializer uses 4KB stackalloc buffer for zero-fills
   • Components larger than 4KB will throw exception if missing
   • Solution: Increase MaxComponentSize constant if needed

3. SceneNode Integration

   • Legacy SceneNode class in Ghost.Editor.Core still exists
   • May need integration with new Scene/SceneSerializer system
   • Future work: Decide on SceneNode vs Scene unification

4. No Compression

   • Binary format is uncompressed
   • Large scenes may have bigger file sizes than necessary
   • Future optimization: Add LZ4/Zstandard compression layer

5. Managed Components

   • Current implementation assumes all IComponent types are unmanaged
   • ScriptComponent and ManagedEntity may need separate handling
   • Future work: Add managed reference serialization

Testing Recommendations

1. Binary Format Round-Trip

   • Create entities with various components
   • Save to binary file
   • Load into new World
   • Verify all component data matches

2. Entity Reference Remapping

   • Create parent-child hierarchies
   • Serialize and deserialize
   • Verify parent/child Entity references updated correctly

3. Additive Loading

   • Load multiple scenes into same World
   • Verify SceneID tagging works correctly
   • Unload specific scenes and verify entities destroyed

4. JSON Compatibility

   • Save same scene to both JSON and binary
   • Verify both formats produce equivalent results when loaded
   • Test JSON editing by hand (human-readable requirement)

5. AOT Compatibility

   • Build Ghost.Engine with NativeAOT or IL2CPP
   • Verify no reflection or dynamic code generation warnings
   • Test binary serialization in AOT-compiled build