Abstract
This thesis sets out to bring improvements in the field of relightable video capture. Previous work in this area is limited to subjects with uniform colour regions or requires the subject to be captured under multiple illumination conditions. Capturing the subject under multiple illumination conditions gives more constraints to robustly fit a complex relightable model but limits the application to static subjects or capture with expensive and constraining high frequency cameras and lighting. This work seeks to bring improvements allowing high quality relighting without these limits. That is, to allow relighting of textured dynamic subjects captured under a single illumination condition. Taking inspiration from advances through the application of nonparametric statistics to computer vision problems, a diffuse reflectance labelling approach is developed that overcomes the limit of requiring the subject to consist of uniform colour regions. Through the use of a combination of invariant region descriptors a transfer map is calculated that is able to transfer diffuse reflectance from a static set to a target video. This allows a static reference set to be captured under multiple illumination conditions and used to label the target dynamic sequence. In developing this approach, methods for nearest neighbour search were evaluated. In investigating full search approaches a parallel graphics card accelerated approach was developed that gives an order of magnitude speed up over an optimised full search equivalent single threaded approach. In comparison non-full search approaches are found to be able to achieve near equivalent quality results more efficiently and with greater flexibility in the choice of similarity metric. The proposed reflectance labelling approach is validated by allowing the normal refinement stage of work previously limited to uniform colour regions to be applied on textured subjects. In investing improvements in surface normal refinement a first principles view is taken of the constraints available under known illumination. A novel normal refinement approach is derived based on this that contributes to detailed and natural relighting results. The proposed reflectance transfer method forms the basis of new work on material editing and mutual illumination correction. Using the proposed approach a material edit can be performed on a single frame and automatically applied to video with photo-realistic results. Photo realistic results are presented for relighting and material editing of performances with natural and rapid motion and of subjects with a range of textures.