Image stitching is the creation of a single image from a set of images. It is nowadays used in Google StreetView, satellite maps, or panorama creation apps. However, this task cannot be considered solved as current algorithms often produce images where objects are duplicated, cropped, or deleted. A recent paper on arXiv.org suggests an improved way to make composite images.
Editing files. Image credit: Magda Ehlers, free picture via Pexels
It proposes improvements to the seam finding step, which usually relies on Markov random field inference. Object detection algorithms are employed, and terms that penalize seams cutting through an object, as well as omission and duplication, are added.
The suggested method leads to more realistic and aesthetically pleasing results than currently used competitors like Photoshop. Furthermore, prioritizing some categories of objects (e. g., human) may be applied, thus preventing such artifacts as torn or duplicated humans in Google StreetView.
Image stitching is typically decomposed into three phases: registration, which aligns the source images with a common target image; seam finding, which determines for each target pixel the source image it should come from; and blending, which smooths transitions over the seams. As described in , the seam finding phase attempts to place seams between pixels where the transition between source images is not noticeable. Here, we observe that the most problematic failures of this approach occur when objects are cropped, omitted, or duplicated. We therefore take an object-centered approach to the problem, leveraging recent advances in object detection [2,3,4]. We penalize candidate solutions with this class of error by modifying the energy function used in the seam finding stage. This produces substantially more realistic stitching results on challenging imagery. In addition, these methods can be used to determine when there is non-recoverable occlusion in the input data, and also suggest a simple evaluation metric that can be used to evaluate the output of stitching algorithms.