Roam Bounty Program #018 - $8000

Overview

Roam is a research lab building its first social product: a mobile game builder that lets anyone create multiplayer games in minutes. We run targeted bounties to solve complex technical challenges that unlock scalability and extensibility in our stack.

This bounty is focused on creating an AI-driven system that automatically configures Unity shader and post-processing properties to match a desired art style from a text or image input. The budget is $8000.

Problem Statement

Our Unity projects use custom-written shaders (HLSL/ShaderLab) and post-processing effects that lack standardized structures. The process of mapping high-level artistic concepts (e.g., "painterly," "cel-shaded") to low-level, specific shader and post-processing parameter settings is a major bottleneck. The core challenge is to build a system that can:

1. Analyze: Understand the stylistic features from a user's text prompt or reference image.
2. Translate: Convert these high-level concepts into concrete shader property values.
3. Automate: Generate a compatible JSON configuration file for our shaders, post-processing effects and future ones, bypassing the non-differentiable nature of the Unity rendering pipeline.

Context and References

For Details on what is an art style, shader or post processing - check out the FAQs below.

The Non-Differentiable Problem

Unity’s rendering pipeline, which transforms shader properties into final pixel colors, is not differentiable. Our shaders and post processing are compiled and executed on the GPU without support for automatic differentiation. This prevents the use of standard gradient-based optimization techniques to iteratively adjust shader parameters to match a target style, as there is no direct way to calculate how a change in a property value affects the final rendered image.

Attempt 1: Large Reference Dataset + CLIP Similarity Matching

• Method: We generated a dataset of 1000+ images, each rendered with a shader using randomly sampled property values, and stored the corresponding JSON. An input prompt was converted to a CLIP embedding and matched against the pre-generated image embeddings to find the most similar one.
• Why it Failed: CLIP embeddings captured irrelevant image features like object geometry, pose, and lighting, rather than the specific stylistic nuances created by the shader. The similarity score was often dominated by factors we don't control, leading to inaccurate and repetitive stylistic matches.

Attempt 2: Direct Optimization using CLIP Loss