Refactor asset pipeline: new registry, loader, and runtime
Major overhaul of asset system: - Split assets into source, .gmeta (JSON), and cooked .imported binaries - Replaced Asset base class; added TextureAsset, TextureLoader - AssetManager now uses job-based, dependency-aware loading - Unified IAssetHandler API; removed legacy handler interfaces - Updated D3D12 allocator and graphics code for new resource model - Improved error handling, memory management, and GPU upload logic - Updated docs and removed obsolete code/interfaces
This commit is contained in:
1216
docs/specs/asset_loading_design.md
Normal file
1216
docs/specs/asset_loading_design.md
Normal file
File diff suppressed because it is too large
Load Diff
310
docs/specs/asset_registry_analysis.md
Normal file
310
docs/specs/asset_registry_analysis.md
Normal file
@@ -0,0 +1,310 @@
|
||||
# GhostEngine Asset Registry — Design Analysis & Recommendations
|
||||
|
||||
## 1. Your Current Design at a Glance
|
||||
|
||||
Your current approach is **Unreal-style packed binary** (`.gasset`):
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────────┐
|
||||
│ AssetMetadata (128 bytes, fixed) │
|
||||
│ FormatVersion ─ ID ─ TypeID ─ │
|
||||
│ HandlerVersion ─ DependencyCount ─ │
|
||||
│ DependenciesOffset ─ SettingsOffset/Size ─ │
|
||||
│ ContentOffset/Size │
|
||||
├──────────────────────────────────────────────┤
|
||||
│ Settings blob (struct → raw bytes) │
|
||||
├──────────────────────────────────────────────┤
|
||||
│ Content blob (e.g. ImageContentHeader + raw) │
|
||||
├──────────────────────────────────────────────┤
|
||||
│ Dependencies (Guid[]) │
|
||||
└──────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
The AssetRegistry maintains an in-memory GUID↔path index by reading the first 20 bytes of every `.gasset` on startup, with a `FileSystemWatcher` for live updates. A planned SQLite backend (`AssetRegistry.Backend.cs`) would persist this catalog.
|
||||
|
||||
---
|
||||
|
||||
## 2. Unreal vs Unity — The Trade-Off Matrix
|
||||
|
||||
| Dimension | Unreal (Packed Binary `.uasset`) | Unity (Raw File + `.meta` sidecar) |
|
||||
|---|---|---|
|
||||
| **Source control** | Opaque blobs — merges impossible, diffs useless | Raw files are human-readable; `.meta` is text YAML — mergeable |
|
||||
| **Import speed** | One file to open per asset | Two opens per asset (source + meta), but meta is tiny |
|
||||
| **Runtime loading** | One `seek+read` → done (no re-import step) | Must "import" (cook) before runtime loading; raw files are editor-only |
|
||||
| **Artist iteration** | Must re-import through editor | Can drop a PNG in Explorer & it auto-imports |
|
||||
| **Dependency tracking** | Embedded in the binary — self-contained | External DB (`.meta` GUIDs + Library/) — can desync |
|
||||
| **Asset settings versioning** | Binary struct layout is fragile | YAML/JSON → easy to add fields with defaults |
|
||||
| **Corruption resilience** | One corrupted byte → whole asset lost | Source file is unaffected; re-import fixes derived data |
|
||||
| **Build pipeline** | Already cooked (or close to it) | Separate cook step needed for builds |
|
||||
| **Team discoverability** | "What is this .gasset?" → need editor to inspect | "It's a PNG, I can open it anywhere" |
|
||||
|
||||
### Key Insight
|
||||
|
||||
> Unreal doesn't actually store source data inside `.uasset` for most asset types. Unreal stores the **cooked/processed** representation. The source data (FBX, PSD, etc.) lives outside the engine's asset system — artists use a separate "source art" folder. The `.uasset` is a **derived artifact**, not the source of truth.
|
||||
|
||||
Unity's insight was: **leave source files alone, store metadata beside them, and derive everything else into a Library/ cache.** The `.meta` sidecar is tiny (GUID + import settings in YAML), version-control-friendly, and the actual imported data lives in `Library/` (a local, regenerable cache).
|
||||
|
||||
---
|
||||
|
||||
## 3. Current Design — Issues Found
|
||||
|
||||
### 3.1 Binary Settings Are a Versioning Nightmare
|
||||
|
||||
```csharp
|
||||
// TextureAssetHandler — writes settings as raw struct bytes
|
||||
Unsafe.WriteUnaligned(ref address, settings.Basic);
|
||||
Unsafe.WriteUnaligned(ref Unsafe.Add(ref address, ...), settings.Advanced);
|
||||
```
|
||||
|
||||
**Problem:** Adding a single field to `BasicSettings`, `AdvancedSettings`, or `SamplerSettings` changes the struct layout. Every existing `.gasset` file becomes unreadable because the byte offsets shift. You have `HandlerVersion` in the metadata, but no migration logic — and you'd need one per handler per version.
|
||||
|
||||
> [!CAUTION]
|
||||
> This is the #1 pain point of the Unreal approach in practice. Epic has dedicated teams managing asset versioning with `FArchive` custom serialization + version tags. For a small team, this is a massive maintenance burden.
|
||||
|
||||
### 3.2 Source File Is Destroyed on Import
|
||||
|
||||
```csharp
|
||||
// OnFileSystemOp — line 224
|
||||
File.Delete(assetPath); // ← deletes the original source file!
|
||||
```
|
||||
|
||||
After import, the source `.png` is deleted and only the `.gasset` remains. If the user wants to change import settings (e.g. switch from BC7 to BC5 for a normal map), they need to find the original source file elsewhere and re-import.
|
||||
|
||||
### 3.3 Handler Discovery Is O(N × M) per Call
|
||||
|
||||
```csharp
|
||||
// GetAssetHandlerForExtension — line 326-338
|
||||
foreach (var handlerType in AppDomain.CurrentDomain.GetAssemblies()
|
||||
.SelectMany(assembly => assembly.GetTypes())
|
||||
.Where(type => typeof(IAssetHandler).IsAssignableFrom(type) ...))
|
||||
```
|
||||
|
||||
This scans **every type in every loaded assembly** on each call. It's called from `OnFileSystemOp` (FileSystemWatcher callback — frequent!) and `ImportAssetAsync`. The `_cachedHandler` dictionary helps for repeat loads, but the initial scan is expensive and runs every time a new extension is encountered.
|
||||
|
||||
### 3.4 `async void` in FileSystemWatcher Callback
|
||||
|
||||
```csharp
|
||||
private async void OnFileSystemOp(object sender, FileSystemEventArgs e)
|
||||
```
|
||||
|
||||
If `ImportAsync` throws, the exception is swallowed silently (unobserved). `FileSystemWatcher` callbacks should be synchronous (queue work to a channel/queue), or at minimum wrap the body in `try/catch`.
|
||||
|
||||
### 3.5 Race Conditions in Path Mapping
|
||||
|
||||
```csharp
|
||||
// ConcurrentDictionary + lock(_pathLock)
|
||||
_pathToGuid = new ConcurrentDictionary<...>(); // concurrent dict
|
||||
lock (_pathLock) { _pathToGuid[relativePath] = guid; } // but manually locked
|
||||
```
|
||||
|
||||
You're using `ConcurrentDictionary` but also taking a `Lock` for every access. These two strategies conflict — either use a plain `Dictionary<>` + lock, or use `ConcurrentDictionary` lock-free. Mixing them gives the worst of both: allocation overhead of `ConcurrentDictionary` with the contention of a lock.
|
||||
|
||||
### 3.6 Missing Content Hash for Cache Invalidation
|
||||
|
||||
The `TextureProcessor` hashes **settings** to build a cache key (`guid_settingsHash.dds`), but doesn't hash the **source content**. If you replace a PNG with a different image of the same name, the stale cache is served because only the settings hash changed (it didn't).
|
||||
|
||||
### 3.7 No Version Migration Path
|
||||
|
||||
The 128-byte `AssetMetadata` header reserves space for expansion — good! But there's no mechanism to detect "this `.gasset` was written by handler v1 and we're now at v3" and upgrade in place. Currently `HandlerVersion` is written but never read.
|
||||
|
||||
---
|
||||
|
||||
## 4. Recommendation: Hybrid Architecture
|
||||
|
||||
I recommend a **Unity-inspired hybrid** — keep source files untouched, use lightweight sidecar metadata, and produce a separate cooked cache. Here's the concrete design:
|
||||
|
||||
### 4.1 Three-Layer Architecture
|
||||
|
||||
```
|
||||
ProjectRoot/
|
||||
├── Assets/ ← Source files (PNG, FBX, HLSL, ...)
|
||||
│ ├── Textures/
|
||||
│ │ ├── hero_diffuse.png ← Source of truth (never modified)
|
||||
│ │ └── hero_diffuse.png.gmeta ← Sidecar: GUID + import settings (YAML/JSON)
|
||||
│ └── Models/
|
||||
│ ├── character.fbx
|
||||
│ └── character.fbx.gmeta
|
||||
│
|
||||
├── Library/ ← Derived data cache (local, .gitignore'd)
|
||||
│ ├── AssetDB.sqlite ← Fast GUID↔path + dependency index
|
||||
│ ├── Imports/ ← Cooked assets (DDS, compiled meshes, etc.)
|
||||
│ │ ├── <guid>.imported ← Binary cooked data (current .gasset content section)
|
||||
│ │ └── ...
|
||||
│ └── Thumbnails/
|
||||
│ └── <guid>.thumb
|
||||
│
|
||||
└── .ghostignore ← Patterns to exclude from asset scanning
|
||||
```
|
||||
|
||||
### 4.2 `.gmeta` Sidecar File
|
||||
|
||||
```yaml
|
||||
# hero_diffuse.png.gmeta
|
||||
guid: 0906f4eb-c3f0-431b-bcea-132c88ab0c3f
|
||||
handler: TextureAssetHandler
|
||||
handlerVersion: 1
|
||||
settings:
|
||||
textureType: Default
|
||||
textureShape: Texture2D
|
||||
isSRGB: true
|
||||
maxSize: 2048
|
||||
filterMode: Anisotropic
|
||||
wrapMode: Repeat
|
||||
generateMipmaps: true
|
||||
compressionLevel: Normal
|
||||
# ... full settings tree
|
||||
dependencies: []
|
||||
labels: [environment, hero] # optional user tags
|
||||
```
|
||||
|
||||
**Why this is better:**
|
||||
|
||||
| Concern | Current `.gasset` | Proposed `.gmeta` |
|
||||
|---|---|---|
|
||||
| Add a field | Binary layout breaks | YAML: missing keys → default values |
|
||||
| Merge conflict | Impossible (binary) | Text merge, trivial |
|
||||
| Inspect settings | Need editor | Open in any text editor |
|
||||
| Source file recovery | Destroyed | Untouched, always available |
|
||||
| Re-import | Need original file | `Library/` rebuild from source + `.gmeta` |
|
||||
| `git diff` | `Binary files differ` | Readable YAML diff |
|
||||
|
||||
### 4.3 SQLite Catalog (`Library/AssetDB.sqlite`)
|
||||
|
||||
Replace the in-memory `ConcurrentDictionary<string, Guid>` mapping with an SQLite database (you already planned this in `AssetRegistry.Backend.cs`):
|
||||
|
||||
```sql
|
||||
-- Core asset table
|
||||
CREATE TABLE assets (
|
||||
guid BLOB PRIMARY KEY, -- 16 bytes, exactly sizeof(Guid)
|
||||
path TEXT NOT NULL, -- relative path to .gmeta
|
||||
handler TEXT NOT NULL, -- handler type name
|
||||
content_hash TEXT, -- xxHash64 of source file bytes
|
||||
settings_hash TEXT, -- xxHash64 of import settings
|
||||
imported_at INTEGER, -- unix timestamp of last successful import
|
||||
UNIQUE(path)
|
||||
);
|
||||
|
||||
-- Dependency edges (forward: asset → dependency)
|
||||
CREATE TABLE dependencies (
|
||||
from_guid BLOB NOT NULL REFERENCES assets(guid),
|
||||
to_guid BLOB NOT NULL REFERENCES assets(guid),
|
||||
PRIMARY KEY (from_guid, to_guid)
|
||||
);
|
||||
|
||||
-- Reverse index for "what depends on me?" queries
|
||||
CREATE INDEX idx_dep_reverse ON dependencies(to_guid);
|
||||
|
||||
-- Full-text search on asset paths and labels
|
||||
CREATE VIRTUAL TABLE assets_fts USING fts5(path, labels);
|
||||
```
|
||||
|
||||
**Startup becomes:**
|
||||
1. Open SQLite DB → instant GUID↔path from indexed table
|
||||
2. Diff `Assets/` tree vs DB → find stale/new/deleted `.gmeta` files
|
||||
3. Queue incremental re-imports only for changed assets
|
||||
|
||||
This is **dramatically faster** than scanning every `.gasset` header on disk (your current `LoadExistingAssets`).
|
||||
|
||||
### 4.4 Import Pipeline
|
||||
|
||||
```
|
||||
Source File Changed
|
||||
│
|
||||
▼
|
||||
FileSystemWatcher
|
||||
│
|
||||
├─── No .gmeta exists? → Generate one (new GUID, default settings)
|
||||
│
|
||||
▼
|
||||
Hash source + settings
|
||||
│
|
||||
├─── Hash matches DB? → Skip (already imported)
|
||||
│
|
||||
▼
|
||||
Queue ImportJob to background channel
|
||||
│
|
||||
▼
|
||||
ImportWorker (background thread pool)
|
||||
│
|
||||
├── Read source file
|
||||
├── Run handler pipeline (e.g. NVTT compress)
|
||||
├── Write Library/Imports/<guid>.imported
|
||||
├── Update SQLite (content_hash, settings_hash, imported_at)
|
||||
└── Fire AssetChanged event on main thread
|
||||
```
|
||||
|
||||
### 4.5 Handler Registration — Build Once, Cache Forever
|
||||
|
||||
Replace the per-call assembly scan with a startup-once TypeCache approach (you already have this pattern in the engine):
|
||||
|
||||
```csharp
|
||||
// Startup: build lookup tables once
|
||||
Dictionary<string, Type> _extensionToHandler; // ".png" → typeof(TextureAssetHandler)
|
||||
Dictionary<Guid, Type> _typeIdToHandler; // TypeGuid → handler type
|
||||
|
||||
// Populated once via TypeCache / assembly attribute scan at editor startup
|
||||
foreach (var type in TypeCache.GetTypesWithAttribute<CustomAssetHandlerAttribute>())
|
||||
{
|
||||
var attr = type.GetCustomAttribute<CustomAssetHandlerAttribute>();
|
||||
_typeIdToHandler[new Guid(attr.ID)] = type;
|
||||
foreach (var ext in attr.SupportedExtensions)
|
||||
_extensionToHandler[ext] = type;
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. What to Keep from Your Current Design
|
||||
|
||||
Your design has several things done well:
|
||||
|
||||
| Element | Verdict |
|
||||
|---|---|
|
||||
| `AssetMetadata` fixed-size header with offsets | ✅ Keep for the cooked `.imported` files — great for O(1) seeks |
|
||||
| `Handle<GPUTexture>` on `TextureAsset` | ✅ Clean separation of asset data vs GPU resource handle |
|
||||
| `WeakReference<Asset>` cache in registry | ✅ Elegant — auto-evicts when nothing holds the asset |
|
||||
| `IAssetHandler` / `IImportableAssetHandler` split | ✅ Good separation (some assets are import-only, e.g. shaders compiled differently) |
|
||||
| `AssetReference` with internal/external encoding | ✅ Clever — keeps sub-asset refs compact |
|
||||
| `TextureProcessor` cache with settings hash | ✅ Great idea, just needs content hash too |
|
||||
| `Result<T>` return pattern | ✅ Consistent with the rest of GhostEngine |
|
||||
|
||||
---
|
||||
|
||||
## 6. Summary Recommendation
|
||||
|
||||
```
|
||||
┌────────────────────────────────────────────────────────────┐
|
||||
│ RECOMMENDED APPROACH │
|
||||
│ │
|
||||
│ Source files → untouched, checked into git │
|
||||
│ .gmeta sidecars → GUID + settings (YAML), in git │
|
||||
│ Library/ → derived cache, .gitignored │
|
||||
│ AssetDB.sqlite → fast GUID↔path index │
|
||||
│ Imports/*.imported → cooked binary (your AssetMetadata │
|
||||
│ header + content, no settings) │
|
||||
│ │
|
||||
│ Binary format → for cooked data only, not settings │
|
||||
│ Settings format → YAML/JSON in .gmeta (human + VCS) │
|
||||
│ Handler discovery → one-time TypeCache at startup │
|
||||
│ Watcher callbacks → queue to Channel<T>, no async void │
|
||||
└────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
This gives you:
|
||||
- **Unreal's runtime performance** (cooked binary in Library/ → single seek+read)
|
||||
- **Unity's artist workflow** (drop files in Assets/, settings are readable text)
|
||||
- **Clean version control** (text `.gmeta` files merge cleanly)
|
||||
- **Resilient re-import** (source is never touched; Library/ is regenerable)
|
||||
- **Zero startup cost** (SQLite index instead of scanning thousands of file headers)
|
||||
|
||||
---
|
||||
|
||||
## 7. Open Questions for You
|
||||
|
||||
1. **Do you want `.gmeta` in YAML, JSON, or a custom text format?** YAML is more compact and human-friendly, but adds a parser dependency. JSON is built into .NET but more verbose. A custom format is more work.
|
||||
|
||||
2. **Should the cooked `.imported` files keep the 128-byte `AssetMetadata` header?** It's useful for validation on load, but since SQLite already knows the GUID and handler, you could simplify the binary format.
|
||||
|
||||
3. **Do you want hot-reload of import settings?** (Changing `.gmeta` → auto re-import and refresh live asset in editor.) Your current `WeakReference<Asset>` + `RefreshAsync` already supports this.
|
||||
|
||||
4. **How do you want to handle the `Library/` on first clone?** Options: (a) full re-import from source, (b) share a pre-built Library via LFS, (c) asset server that caches imports.
|
||||
1625
docs/specs/asset_registry_design.md
Normal file
1625
docs/specs/asset_registry_design.md
Normal file
File diff suppressed because it is too large
Load Diff
1125
docs/specs/meshlet-architecture.md
Normal file
1125
docs/specs/meshlet-architecture.md
Normal file
File diff suppressed because it is too large
Load Diff
457
docs/specs/shader_pipeline_architecture.md
Normal file
457
docs/specs/shader_pipeline_architecture.md
Normal file
@@ -0,0 +1,457 @@
|
||||
# Shader Pipeline Architecture — Proposed Design
|
||||
|
||||
> Presented as a design walkthrough. Take what's useful, ignore what doesn't fit your vision.
|
||||
|
||||
---
|
||||
|
||||
## 1. System Topology
|
||||
|
||||
The first decision: **where does each responsibility live?**
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph EditorProcess["Ghost.Editor Process"]
|
||||
FW["FileWatcher<br/>(monitors .ghost DSL files)"]
|
||||
AR["AssetRegistry<br/>(GUID ↔ file path mapping)"]
|
||||
EP["Editor UI<br/>(status bar, material inspector)"]
|
||||
end
|
||||
|
||||
subgraph CompilerProcess["GhostShaderServer Process"]
|
||||
DSL["DSL Compiler<br/>(Ghost DSL → HLSL)"]
|
||||
DXC["DXC Compiler<br/>(HLSL → DXIL bytecode)"]
|
||||
MW["Manifest Writer<br/>(updates variant → hash mapping)"]
|
||||
end
|
||||
|
||||
subgraph RuntimeGraphics["Ghost.Graphics (Runtime)"]
|
||||
SL["ShaderLibrary<br/>(reads bytecode from cache)"]
|
||||
PL["PipelineLibrary<br/>(PSO creation + double-buffer)"]
|
||||
RGC["RenderGraphContext<br/>(binds PSO per draw call)"]
|
||||
BR["IShaderCompilationBridge<br/>(interface, 2 methods)"]
|
||||
end
|
||||
|
||||
subgraph SharedDisk["Shared Disk (ShaderCache/)"]
|
||||
MF["ShaderManifest.bin<br/>(GUID+variant → content hash)"]
|
||||
BC["Bytecode Files<br/>(content-addressed .bin blobs)"]
|
||||
end
|
||||
|
||||
FW -- "file changed event" --> AR
|
||||
AR -- "GUID + file path<br/>(named pipe)" --> CompilerProcess
|
||||
DSL --> DXC
|
||||
DXC -- "bytecode bytes" --> MW
|
||||
MW -- "write blob" --> BC
|
||||
MW -- "update entry" --> MF
|
||||
|
||||
SL -- "read blob" --> BC
|
||||
SL -- "read mapping" --> MF
|
||||
BR -- "status query<br/>(named pipe)" --> CompilerProcess
|
||||
|
||||
EP -- "poll status" --> BR
|
||||
|
||||
style CompilerProcess fill:#1a1a2e,stroke:#e94560,color:#eee
|
||||
style EditorProcess fill:#1a1a2e,stroke:#0f3460,color:#eee
|
||||
style RuntimeGraphics fill:#1a1a2e,stroke:#16213e,color:#eee
|
||||
style SharedDisk fill:#0f3460,stroke:#533483,color:#eee
|
||||
```
|
||||
|
||||
### Why a Separate Process?
|
||||
|
||||
| Concern | In-process compiler | Separate process |
|
||||
|---------|-------------------|------------------|
|
||||
| DXC crash | Editor dies | Server restarts, editor lives |
|
||||
| DXC memory leak | Editor bloats over time | Kill & restart server periodically |
|
||||
| Parallelism | Threads compete with editor UI | Fully independent CPU budget |
|
||||
| Build pipeline reuse | Need separate build-time path | Same server binary, different mode |
|
||||
| Complexity | Lower (one process) | Higher (IPC needed) |
|
||||
|
||||
> [!TIP]
|
||||
> If the separate process feels like overkill for your current stage, **start with in-process behind the `IShaderCompilationBridge` interface**, then swap the implementation to out-of-process later. The interface is the same either way.
|
||||
|
||||
---
|
||||
|
||||
## 2. Data Model — The Manifest
|
||||
|
||||
This is the most important data structure in the entire system. It decouples **identity** from **content**.
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
subgraph ShaderAsset["Shader Asset (on disk)"]
|
||||
GUID["Asset GUID<br/><i>e.g. 7f3a-...-c82b</i><br/>stable forever"]
|
||||
SRC["Source Code<br/><i>.ghost DSL file</i><br/>changes on edit"]
|
||||
end
|
||||
|
||||
subgraph Manifest["ShaderManifest"]
|
||||
E1["Entry:<br/>GUID=7f3a | Pass=0 | Variant=0x00<br/>→ ContentHash=0xABCD"]
|
||||
E2["Entry:<br/>GUID=7f3a | Pass=0 | Variant=0x01<br/>→ ContentHash=0x1234"]
|
||||
E3["Entry:<br/>GUID=7f3a | Pass=1 | Variant=0x00<br/>→ ContentHash=0x5678"]
|
||||
end
|
||||
|
||||
subgraph Cache["ShaderCache/ (content addressed)"]
|
||||
B1["AB/shader_cache_ABCD...bin"]
|
||||
B2["12/shader_cache_1234...bin"]
|
||||
B3["56/shader_cache_5678...bin"]
|
||||
end
|
||||
|
||||
GUID --> E1
|
||||
GUID --> E2
|
||||
GUID --> E3
|
||||
E1 --> B1
|
||||
E2 --> B2
|
||||
E3 --> B3
|
||||
|
||||
style ShaderAsset fill:#16213e,stroke:#0f3460,color:#eee
|
||||
style Manifest fill:#1a1a2e,stroke:#e94560,color:#eee
|
||||
style Cache fill:#0f3460,stroke:#533483,color:#eee
|
||||
```
|
||||
|
||||
### Manifest Entry Structure
|
||||
|
||||
```
|
||||
ManifestKey = Hash(AssetGUID + PassIndex + VariantKeywordMask)
|
||||
ManifestValue = ContentHash (= Hash of compiled bytecode)
|
||||
```
|
||||
|
||||
- **ManifestKey** is *structurally* derived — same shader, same pass, same keywords = same key, regardless of source changes.
|
||||
- **ContentHash** is *content-derived* — changes every time the source code changes.
|
||||
- When source changes: the ManifestKey stays the same, but the ContentHash it points to gets updated.
|
||||
|
||||
> [!IMPORTANT]
|
||||
> The `Shader` struct in runtime only needs to know the **AssetGUID**. It never stores or cares about content hashes. The `ShaderLibrary` uses the manifest to translate `(GUID, Pass, Variant) → ContentHash → File`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Compilation Flow — What Happens When You Save a Shader
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant User
|
||||
participant FileSystem
|
||||
participant Editor as Ghost.Editor
|
||||
participant Server as ShaderServer
|
||||
participant Cache as ShaderCache/
|
||||
|
||||
User->>FileSystem: Save "water.ghost"
|
||||
FileSystem-->>Editor: FileWatcher event
|
||||
|
||||
Editor->>Editor: Lookup GUID for "water.ghost"<br/>via AssetRegistry
|
||||
Editor->>Server: CompileRequest {<br/> guid: 7f3a-...,<br/> filePath: "water.ghost",<br/> defines: [...],<br/> platform: D3D12<br/>}
|
||||
|
||||
Note over Server: Mark status = Compiling<br/>for this GUID
|
||||
|
||||
Server->>Server: Read .ghost DSL file
|
||||
Server->>Server: DSL Compiler: DSL → HLSL
|
||||
|
||||
alt DSL has syntax errors
|
||||
Server->>Server: Mark status = Error
|
||||
Server-->>Editor: CompileResult {<br/> status: Error,<br/> errors: [...]<br/>}
|
||||
Editor->>Editor: Show errors in<br/>console/inspector
|
||||
else DSL is valid
|
||||
Server->>Server: For each (pass, variant):<br/>DXC Compile HLSL → DXIL
|
||||
|
||||
alt Any DXC error
|
||||
Server->>Server: Mark status = Error
|
||||
Server-->>Editor: CompileResult {<br/> status: Error,<br/> errors: [...]<br/>}
|
||||
else All variants compiled
|
||||
Server->>Cache: Write bytecode blobs<br/>(content-addressed)
|
||||
Server->>Cache: Update manifest entries:<br/>(GUID+pass+variant) → new hash
|
||||
Server->>Server: Mark status = Ready
|
||||
Server-->>Editor: CompileResult {<br/> status: Ready,<br/> variantCount: N<br/>}
|
||||
Editor->>Editor: Show ✓ in status bar
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
### Key Design Decision: Compile All Variants Upfront?
|
||||
|
||||
**No.** Only compile variants that are *currently referenced* by materials in the scene. The editor knows which materials reference which shader (via AssetRegistry), and which keyword combinations those materials use. Ship only what's needed.
|
||||
|
||||
For the edit-time hot-reload, you really only need the specific variants the viewport is currently rendering. The full permutation set is a build-time concern.
|
||||
|
||||
---
|
||||
|
||||
## 4. Runtime PSO Resolution — The Frame-by-Frame Flow
|
||||
|
||||
This is where most of the complexity lives. Here's what `SetActiveMaterial` does every frame:
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A["SetActiveMaterial(material)"] --> B["Compute ManifestKey<br/>= f(shader.GUID, passIndex, variantMask)"]
|
||||
B --> C{"PipelineLibrary<br/>has PSO for<br/>ManifestKey?"}
|
||||
|
||||
C -- "Yes (cache hit)" --> D["Bind existing PSO<br/>to command buffer"]
|
||||
D --> Z["Done ✓"]
|
||||
|
||||
C -- "No (cache miss)" --> E{"ShaderLibrary<br/>has bytecode for<br/>ManifestKey?"}
|
||||
|
||||
E -- "Yes (manifest hit)" --> F["Read bytecode<br/>from cache file"]
|
||||
F --> G["Create PSO from bytecode"]
|
||||
G --> H["Store in PipelineLibrary"]
|
||||
H --> D
|
||||
|
||||
E -- "No (manifest miss)" --> I{"Is this Editor<br/>or Runtime?"}
|
||||
|
||||
I -- "Runtime<br/>(shipped game)" --> J["Bind Fallback<br/>ERROR PSO ⚠️"]
|
||||
J --> K["Log error:<br/>missing shader"]
|
||||
K --> Z
|
||||
|
||||
I -- "Editor" --> L{"Query Bridge:<br/>IsCompiling?"}
|
||||
|
||||
L -- "Status = Compiling" --> M["Bind OLD PSO<br/>(keep previous frame's shader)"]
|
||||
M --> Z
|
||||
|
||||
L -- "Status = Error" --> N["Bind ERROR PSO<br/>(magenta)"]
|
||||
N --> Z
|
||||
|
||||
L -- "Status = Ready" --> O["The manifest was just updated.<br/>Re-read manifest entry."]
|
||||
O --> F
|
||||
|
||||
L -- "Status = NotAvailable" --> J
|
||||
|
||||
style A fill:#533483,stroke:#e94560,color:#eee
|
||||
style D fill:#16213e,stroke:#0f3460,color:#eee
|
||||
style J fill:#e94560,stroke:#1a1a2e,color:#eee
|
||||
style M fill:#0f3460,stroke:#533483,color:#eee
|
||||
style N fill:#e94560,stroke:#1a1a2e,color:#eee
|
||||
style Z fill:#16213e,stroke:#16213e,color:#eee
|
||||
```
|
||||
|
||||
### The "Keep Old PSO" Strategy — How It Works Mechanically
|
||||
|
||||
This is the part that makes the UX feel seamless. The trick:
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
subgraph PipelineLibrary
|
||||
direction TB
|
||||
K["ManifestKey 0xAABB"]
|
||||
K --> CURRENT["current: PSO_v2 ✓<br/>(what we render with)"]
|
||||
K --> PENDING["pending: null<br/>(set during recompilation)"]
|
||||
end
|
||||
|
||||
style CURRENT fill:#16213e,stroke:#0f3460,color:#eee
|
||||
style PENDING fill:#1a1a2e,stroke:#e94560,color:#eee
|
||||
```
|
||||
|
||||
When shader source changes and recompilation starts:
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
subgraph PipelineLibrary_During["During Recompilation"]
|
||||
direction TB
|
||||
K2["ManifestKey 0xAABB"]
|
||||
K2 --> CURRENT2["current: PSO_v2 ✓<br/>(still rendering with this)"]
|
||||
K2 --> PENDING2["pending: COMPILING<br/>(server is working...)"]
|
||||
end
|
||||
|
||||
style CURRENT2 fill:#16213e,stroke:#0f3460,color:#eee
|
||||
style PENDING2 fill:#e94560,stroke:#1a1a2e,color:#eee
|
||||
```
|
||||
|
||||
When recompilation finishes successfully:
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
subgraph PipelineLibrary_After["After Swap"]
|
||||
direction TB
|
||||
K3["ManifestKey 0xAABB"]
|
||||
K3 --> CURRENT3["current: PSO_v3 ✓<br/>(new shader, rendering now)"]
|
||||
K3 --> PENDING3["pending: null<br/>(swap complete)"]
|
||||
end
|
||||
|
||||
style CURRENT3 fill:#16213e,stroke:#0f3460,color:#eee
|
||||
style PENDING3 fill:#1a1a2e,stroke:#533483,color:#eee
|
||||
```
|
||||
|
||||
> [!NOTE]
|
||||
> The old `PSO_v2` is **not immediately destroyed**. It stays alive until the GPU is done with any in-flight frames referencing it (tracked by fence value). This prevents use-after-free on the GPU timeline.
|
||||
|
||||
---
|
||||
|
||||
## 5. Hot-Reload Sequence — The Complete Picture
|
||||
|
||||
Everything combined into one timeline:
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant User
|
||||
participant Editor
|
||||
participant Server as ShaderServer
|
||||
participant Cache as Disk Cache
|
||||
participant Runtime as RenderGraphContext
|
||||
participant GPU
|
||||
|
||||
Note over Runtime,GPU: Frame N: Rendering with PSO_v2
|
||||
|
||||
User->>Editor: Edit & save "water.ghost"
|
||||
Editor->>Server: CompileRequest(guid=7f3a)
|
||||
Server->>Server: status[7f3a] = Compiling
|
||||
|
||||
Note over Runtime,GPU: Frame N+1
|
||||
Runtime->>Runtime: SetActiveMaterial()
|
||||
Runtime->>Runtime: ManifestKey lookup → old hash still there
|
||||
Runtime->>Runtime: PipelineLibrary has PSO → use it
|
||||
Note over Runtime: Still rendering with PSO_v2<br/>(user sees no flicker)
|
||||
|
||||
Note over Server: Background: DSL→HLSL→DXC...
|
||||
|
||||
Note over Runtime,GPU: Frame N+2, N+3, ...
|
||||
Runtime->>Runtime: Same as N+1, no visible change
|
||||
|
||||
Server->>Cache: Write new bytecode files
|
||||
Server->>Cache: Update manifest:<br/>key(7f3a,0,0) → new_hash
|
||||
Server->>Server: status[7f3a] = Ready
|
||||
|
||||
Note over Runtime,GPU: Frame N+K (compilation done)
|
||||
Runtime->>Runtime: SetActiveMaterial()
|
||||
Runtime->>Runtime: Manifest read → NEW content hash
|
||||
Runtime->>Runtime: PipelineLibrary miss for new hash
|
||||
Runtime->>Cache: Read new bytecode
|
||||
Runtime->>GPU: Create PSO_v3
|
||||
Runtime->>Runtime: PipelineLibrary: current=PSO_v3
|
||||
Runtime->>Runtime: Bind PSO_v3
|
||||
|
||||
Note over Runtime,GPU: Frame N+K+1: Rendering with PSO_v3 ✓
|
||||
|
||||
Runtime->>Runtime: Defer release PSO_v2<br/>(after GPU fence)
|
||||
```
|
||||
|
||||
### What the User Sees
|
||||
|
||||
| Frame | Viewport | Status Bar |
|
||||
|-------|----------|------------|
|
||||
| N | Water renders normally | — |
|
||||
| N+1 | Water renders normally (old shader) | 🔄 Compiling water.ghost... |
|
||||
| N+2 | Water renders normally (old shader) | 🔄 Compiling water.ghost... |
|
||||
| N+K | Water renders with new shader | ✅ water.ghost compiled (2 variants) |
|
||||
|
||||
**Zero flicker. Zero blocking. Zero pink frames.**
|
||||
|
||||
---
|
||||
|
||||
## 6. How the Manifest Key Replaces Your Current Hash Problem
|
||||
|
||||
Here's a before/after of your `Shader` struct:
|
||||
|
||||
### Current Design (problematic)
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph Current["Current: Hash = f(source code)"]
|
||||
S1["Shader struct"] --> P1["Pass[0].Key = 0xABCD<br/><i>derived from source hash</i>"]
|
||||
P1 --> V1["ShaderVariantKey = f(0xABCD, keywords)"]
|
||||
V1 --> PK1["PipelineKey = f(variant, rtv, dsv)"]
|
||||
PK1 --> PSO1["PSO lookup in PipelineLibrary"]
|
||||
|
||||
EDIT["User edits source"] -.-> STALE["Pass[0].Key is now STALE ❌<br/>Still 0xABCD, but source changed"]
|
||||
STALE -.-> WRONG["Looks up OLD bytecode<br/>or worse, the old PSO"]
|
||||
end
|
||||
|
||||
style STALE fill:#e94560,stroke:#1a1a2e,color:#eee
|
||||
style WRONG fill:#e94560,stroke:#1a1a2e,color:#eee
|
||||
```
|
||||
|
||||
### Proposed Design (stable)
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph Proposed["Proposed: Key = f(GUID, pass index)"]
|
||||
S2["Shader struct<br/>assetGUID = 7f3a-..."] --> P2["Pass[0]: index=0<br/><i>no source hash stored</i>"]
|
||||
P2 --> MK["ManifestKey = f(7f3a, 0, keywords)"]
|
||||
MK --> MANIFEST["Manifest Lookup<br/>→ ContentHash = 0x9999"]
|
||||
MANIFEST --> SL2["ShaderLibrary<br/>→ read 99/shader_cache_9999.bin"]
|
||||
SL2 --> PSO2["Create or get PSO"]
|
||||
|
||||
EDIT2["User edits source"] -.-> RECOMP["Server recompiles<br/>→ new ContentHash = 0xBBBB"]
|
||||
RECOMP -.-> MUPD["Manifest updated:<br/>same key → 0xBBBB"]
|
||||
MUPD -.-> NEXT["Next frame: manifest read<br/>picks up 0xBBBB automatically"]
|
||||
end
|
||||
|
||||
style RECOMP fill:#0f3460,stroke:#533483,color:#eee
|
||||
style MUPD fill:#0f3460,stroke:#533483,color:#eee
|
||||
style NEXT fill:#16213e,stroke:#0f3460,color:#eee
|
||||
```
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **The `Shader` struct never changes.** No unload, no recreate, no generation counter bump. The manifest is the *only* mutable state, and it lives on disk, outside the runtime's object graph. The runtime just reads it.
|
||||
|
||||
---
|
||||
|
||||
## 7. The Two Interfaces That Make This Work
|
||||
|
||||
Only two abstractions are needed in `Ghost.Graphics` to support the full pipeline:
|
||||
|
||||
```mermaid
|
||||
classDiagram
|
||||
class IShaderCompilationBridge {
|
||||
<<interface>>
|
||||
+TryGetBytecode(manifestKey: ulong, out bytecode: ReadOnlyMemory~byte~) bool
|
||||
+IsCompiling(manifestKey: ulong) bool
|
||||
}
|
||||
|
||||
class RuntimeStub {
|
||||
+TryGetBytecode() → always from ShaderLibrary cache
|
||||
+IsCompiling() → always false
|
||||
}
|
||||
|
||||
class EditorImplementation {
|
||||
-NamedPipeClient _serverConnection
|
||||
+TryGetBytecode() → check manifest, read cache
|
||||
+IsCompiling() → query server status
|
||||
}
|
||||
|
||||
IShaderCompilationBridge <|.. RuntimeStub : "Shipped game"
|
||||
IShaderCompilationBridge <|.. EditorImplementation : "Editor mode"
|
||||
|
||||
class ShaderLibrary {
|
||||
-string _cacheDirectory
|
||||
+GetCache(contentHash: ulong) Result~bytes~
|
||||
+GetFromManifest(manifestKey: ulong) Result~bytes~
|
||||
}
|
||||
|
||||
EditorImplementation --> ShaderLibrary : reads cache
|
||||
RuntimeStub --> ShaderLibrary : reads cache
|
||||
```
|
||||
|
||||
> [!TIP]
|
||||
> `RenderGraphContext` doesn't talk to the bridge directly. It talks to `ShaderLibrary`, which internally consults the bridge on cache miss. This keeps the rendering code clean — it never sees compilation status. It just gets bytecode or it doesn't.
|
||||
|
||||
---
|
||||
|
||||
## 8. Build Pipeline — How Shipped Games Work
|
||||
|
||||
For completeness, here's how the same architecture handles builds:
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph BuildTime["Build Pipeline"]
|
||||
SCAN["Scan all materials<br/>in scenes/assets"] --> COLLECT["Collect all referenced<br/>(GUID, pass, variant) tuples"]
|
||||
COLLECT --> COMPILE["Compile all variants<br/>via ShaderServer"]
|
||||
COMPILE --> PACK["Package manifest +<br/>bytecode blobs into<br/>game data archive"]
|
||||
end
|
||||
|
||||
subgraph ShippedGame["Runtime (shipped game)"]
|
||||
LOAD["Load manifest +<br/>bytecode from archive"] --> LIB["ShaderLibrary<br/>(read-only, all variants pre-cached)"]
|
||||
LIB --> MISS{"Cache miss?"}
|
||||
MISS -- "Never<br/>(if build is correct)" --> OK["Create PSO normally"]
|
||||
MISS -- "Somehow yes<br/>(bug or modding)" --> ERR["Error PSO<br/>+ log warning"]
|
||||
end
|
||||
|
||||
BuildTime --> ShippedGame
|
||||
|
||||
style BuildTime fill:#1a1a2e,stroke:#0f3460,color:#eee
|
||||
style ShippedGame fill:#16213e,stroke:#533483,color:#eee
|
||||
```
|
||||
|
||||
The beauty: **the same `ShaderLibrary` and `PipelineLibrary` code runs in both editor and shipped game**. The only difference is whether `IShaderCompilationBridge` is the editor implementation or the runtime stub.
|
||||
|
||||
---
|
||||
|
||||
## Summary of Key Design Decisions
|
||||
|
||||
| # | Decision | Rationale |
|
||||
|---|----------|-----------|
|
||||
| 1 | Stable GUID identity, not content hash | Shader struct never needs recreation on edit |
|
||||
| 2 | Content-addressed cache | Deduplication, easy invalidation, git-friendly |
|
||||
| 3 | Manifest as the bridge | Decouples identity from compiled output cleanly |
|
||||
| 4 | Keep old PSO during recompile | Zero flicker, seamless UX |
|
||||
| 5 | Separate compiler process | Crash isolation, independent resource budget |
|
||||
| 6 | Two-method interface in runtime | Minimal coupling, easy to stub for shipped game |
|
||||
| 7 | Deferred PSO release via fence | Prevents GPU use-after-free |
|
||||
| 8 | Same code path for editor + shipped | Fewer bugs, one pipeline to maintain |
|
||||
Reference in New Issue
Block a user