Wisp supports joint rendering of various pipelines (NeRF, SDFs, meshes)

[3a1b2c3]

Very curious if there are any examples of combining pipelines? Just add them to multiple scene_graph.neural_pipelines

[orperel]

Hi @3a1b2c3 , correct, adding objects to WispState's scene_graph.neural_pipelines should do the trick.
We'll add an example to showcase that soon.

Be aware that for now, the neural fields we optimize are assumed to exist in the normalized range of -1 to 1 (supporting per instance model matrix is on our short-term roadmap)

[3a1b2c3]

Great, what template would be adding a mesh for rendering? Does that mean mesh as sdf?

[orperel]

We don't have renderable meshes yet ;)

1. This is not mesh rendering per se, but a quick workaround -
   We have a PR under review for rendering kaolin's Structured Point Clouds as a neural field, which should be integrated very soon. That would entail converting your mesh to a SPC like so.

2. The correct way to go about it would be to implement a RasterizedRenderer for rendering meshes -> RenderBuffer.
   For example, kaolin's DIB-R rasterizer could be used as an actual implementation for such RasterizedRenderer.

[3a1b2c3]

Ugly code but got it in state

but optimizerApp wont allow multiple inputs for training function so curious to see an example. I am trying to move stuff around a bit to see what i can get

So DIB-R rasterizer does the camera space magic.
Can i use RasterizedRenderer and Nerf renderer at the same time (with these masks)? Like render a picture with a nerf and a mesh?

[3a1b2c3]

Would love to see a road map for this project at some point?

[3a1b2c3]

Small win: getting to train nerf and sdf. Still fighting transforms a a bit, would you normalize sdf in first place if thats not the final result i want? Can sdfs do textures?

[orperel]

Hi @3a1b2c3 ,
Great to see you've got some progress there!

For your questions -

1. "OptimizerApp wont allow multiple inputs for training function" - did you end up creating 2 separate trainers for nerf / sdf? If so you could try:

def joint_train_step():
   nerf_trainer.iterate()
   sdf_trainer.iterate()

and pass trainer_step_func=joint_train_step to the OptimizerApp.

2. "Can I use RasterizedRenderer and Nerf renderer at the same time?" - yup, that's the idea behind the renderers. They each represent an object in your scene graph so you can have multiple - a mesh and some nerf at the same time (even if one is ray traced and the other is rasterized). I've just added some explanations and diagrams about the renderers, you can check them out here: https://github.com/NVIDIAGameWorks/kaolin-wisp/tree/main/wisp/renderer

3. For roadmap of missing capabilities: right now we're still setting the course - but if you're worried about implementing something from our future plans, please don't be :) We're always happy to accept external contributions! (i.e. if you get support for meshes working).

4. Re: normalization - yes, the scale you set during optimization is assumed to be -1 to 1 (that's just convenient). What you're missing is an object transform that takes the rendered object and translates / rotates / scales it. I actually implemented, but I still need to iron out some bugs before I can push that one. See attachment:

wisp_transforms.mr.mp4

[3a1b2c3]

Having the transform exposed in the gui is way nicer than in the config like i did it :). I grabbed some code from pytorch3d for that. Transforming a nerf is slightly harder... Do you apply the transform to the grid or the resulting data in training?
Normalization again: training will fail when data is bigger than -1, 1 space?
Love your renderer drawings.

What is your take on best way to load several configs from commandline? I just hard coded the sdf one for now into my main script..

I have to check my code my code for the training details, basically i allow an array of hooks to be called OptimizationApp but your sugestion requires fewrr code changes. There is some problem with stopiteration exception now i think. Thanks for your help

[3a1b2c3]

Just trying the new version after being away to Europe ... lovely. Joint rendering would still be parallel more with no expectation of different pipelines occluding each other correctly etc?
Would it make sense to add it to the config system?

[3a1b2c3]

So I kind of could do a frankenrender (2 renders needed) vs frankennerf (Merge trained nerf and mesh use 1 renderer) approach. Might be worth trying both?

[3a1b2c3]

Nerfstudios renderer seems to be dealing with that scene slightly better (they also have controllable clipping planes)

[orperel]

Hi @3a1b2c3 !
Just to let you know that ObjectTransforms are now finally pushed:
https://github.com/NVIDIAGameWorks/kaolin-wisp/pull/65/files

How to use them:
You should see them right there in the gui.
You could also define a transform programmatically like so:

from wisp.renderer.core.api import add_to_scene_graph, remove_from_scene_graph
from wisp.core import ObjectTransform

t = ObjectTransform()
add_to_scene_graph(wisp_state, name=you_object_name, obj=you_object_pipeline, transform=t)
t.translate(...)
t.rotate(...)

Sorry for the delay, to address some of your questions:

• Transforms are applied in 2 different ways: ray traced objects (including NeRFs), actually apply the inverse transform on the rays. For rasterized objects, the model matrix from the transform object is applied in conjunction with the camera view matrix (for example, for data layers, that happens when the shaders are invoked, i.e. the PrimitivesPainter for data layers).

• Transforms are applied in a separate way to training. Training should always take place in normalized object coordinates (-1 to 1, for example). Unless you have a good reason, my suggestion would be to stick with those bounds and transform your object later for visualization purposes.

• Loading several configs: we've revamped the mains / config parser to make them easier to customize with your logic. This should be pushed soon, I plan to follow up with an example showing how to load objects via a main and view them side by side.

• Occlusions of different neural fields / pipelines: We handle it right now by predicting a depth channel, so assuming your SDF and NeRF both produce RenderBuffers with depth channels (should do that by default, according to the behavior of our default tracers), you should be fine. Some quality issues like "z fighting" or properly blending transparent areas from the 2 fields could be tricky to get right. The depth precision you had in wisp could, i.e. account for the differences to the scene quality via other software (tuning hyperparameters and using regluarizers could mitigate that).

• Merging together various object types: I'd argue this is still an open research problem. The solution we have for multiple pipelines is sort of a "frankenrender", it works but of course not optimal.