Individualkex also has a couple videos on the high level ideas: https://youtu.be/GQXDjzNWuPc?si=zlAN7dO9STGATKad
[1] https://github.com/playcanvas/supersplat
I remember a time when it was considered unpolite to ask a question without googling first. Is it still the case?
Yes
- https://www.meta.com/en-gb/experiences/gracia/25784099001234...
Any tips for an app to use on iOS to capture the necessary .ply data?
Scaniverse is a great app by Niantic that can do this on-device, but it isn't very customizable and can't export its raw scanning data (exported .plys do not have the data this editor requires).
Splats are sort of like byte code, they are the compiled and optimized representation of reflected light as semi-transparent guassians.
Or you can think of them as the PDF equivalent of a Google or Word Doc. All the logic is gone, and you just have final optimized results.
Generally when you edit PDFs, the results are not great and you cannot make major edits because the layout won't reflow, etc.
So while this is cool, I don't think it will take off unless there is another innovation in terms of either using AI to "reflow" the lighting and surfaces after an edit, or inferring more directly the underlying representations (true surface properties and the light sources.)
- SuperSplat dev :)
https://playcanv.as/e/p/cLkf99ZV/
Integrating AI is an interesting topic and something that certainly has potential.
- cleaning up noisy GuassianSplats is useful. There are often stragglers floating around in space that need to get deleted.
- compression/optimizing them is useful.
This being a cleanup and compression tool makes sense, but I guess I don't call that an "editor."
I guess I was more arguing against the idea that this is a viable "editor" where one can combine and manipulate in more radical ways Gaussian Splats. The current technological approach doesn't make this a feasible use case.
- Copy & Paste: e.g. delete a tree and fill the hole with a copied patch of grass
- Color Adjustments: tinting, brightness, etc.
If these aren't editing ops, I don't know what is. :) Sure, you _could_ go back and recapture photogrammetry or rerun training, but that's super costly in terms of time. SuperSplat lets you make simple edits quickly and easily.
In that model, we don't compile them, we train them; we don't run them, we sample/rasterize them.
This link came up on HN before and was a great refresher/expander on the math of Guassians which allow all this. [1].
Since Gaussians can be estimated, neural networks can model/generate them. Researchers are using this for 4D work and mesh extraction. The NNs run at lower frame rate informing the 3DGS running at interactive rates.
You are right that it is ephemeral and really a weird trick of the eye and we need new ways to edit/create it. Vectors/pixels have had a lot more time to grow tooling. People are working on it, just the toolbox is different. Very cool stuff will be coming up, I bet!
[1] https://news.ycombinator.com/item?id=41912160 I've also re-learned Fourier transforms to appreciate similar concepts.
EDIT: yep - https://gsgen3d.github.io/