📗 Neural networks is a class of functions parameterized by a set of weights (linear coefficients) that can approximate a function arbitrarily closely as the size of the network increases. For any \(y = f\left(x\right)\), \(y \approx N\left(x; w\right)\) where \(w\) can be estimated "efficiently" from samples of \(\left(x, y\right)\).
➩ \(x\) pixels of images, \(y \in \left\{\text{cat}, \text{dog}, \text{cow}\right\}\) labels, \(y \approx N\left(x\right)\) estimates the relationship between images and labels.
➩ \(x\) emails (numerical vectors representing the text), \(y \in \left\{-1, 1\right\}\) spam or ham, \(y \approx N\left(x\right)\) estimates the relationship between emails and labels.
➩ \(x\) states (configurations) of a game, \(y \in A\) set of possible actions, \(y \approx N\left(x\right)\) estimates the optimal action in state \(x\).
➩ \(x\) prompt (context) in a large language model, \(y \in V\) representing the next word in the vocabulary, \(y \approx N\left(x\right)\) predicts the next word based on the prompt.
📗 Neural networks are used in computer vision and image generation models.
➩ Pose estimation and face landmark tracking: \(y \approx N\left(x\right)\) where \(x\) is the image (or one frame of a video) and \(y\) are the key points (their 3D positions).
➩ Image segmentation: \(y \approx N\left(x\right)\) where \(x\) is the pixels and \(y\) is the group label of each pixel.
➩ Diffusion models (image generation, including texture generation): \(y \approx N\left(x\right)\) where \(x\) is a noisy image and \(y\) is the noise and \(x - y\) is the clean image with noise removed. Stable diffusion integrates this with a language model so \(x\) can be the prompt (numerical vectors representing the text) and \(y\) is the output image.
➩ Upscaling (super resolution): \(y \approx N\left(x\right)\) where \(x\) is the original image and \(y\) is the image with higher resolution.
➩ Neural Radiance Fields (NeRFs): \(y \approx N\left(x\right)\) where \(x\) is the image and \(y\) is the objects (points) in the scene. Gaussian splatting performs similar tasks but does it without using neural networks.
📗 Given photos with known camera direction, reconstruct a 3D scene, and render the scene from a new angle.
📗 Neural Radiance Fields (NeRFs) uses a neural network:
➩ Instead of learning the meshes, learn a neural network \(N\left(x, y, z, v\right)\) that outputs \(\left(\sigma, r, g, b\right)\), where \(v\) is the viewing direction, \(\sigma\) is the density (opacity \(\alpha = 1 - e^{- \sigma d}\) where \(d\) is the distance\(), and\)(r, g, b) is the color.
➩ When rendering, small \(\sigma\) means transparent (i.e. no object), and large \(\sigma\) means opaque (i.e. surface blocks visibility).
📗 Gaussian Splatting uses Gaussian primitives:
➩ Instead of learning the neural network representation, learn many 3D Gaussian ellipsoids with mean and variance \(\left(\mu, \Sigma\right)\), where \(\mu\) is the center (translation) and \(\Sigma\) is similar to the radius of the ellipsoid (rotation + scaling).
➩ When rendering, project the Gaussian ellipsoids to screen, which becomes ellipses (splats).
📗 Traditional graphics use primitives (geometry, material, etc) to represent objects in a scene.
📗 AI graphics use other representations (neural networks in NeRFs, Gaussian ellipsoids in Gaussian splats, latent space in diffusion models) to represent objects in a scene.