Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model

ICLR 2023, Notable-Top-25%

Introduction

Most existing Image Restoration (IR) models are task-specific, which can not be generalized to different degradation operators. In this work, we propose the Denoising Diffusion Null-Space Model (DDNM), a novel zero-shot framework for arbitrary linear IR problems, including but not limited to image super-resolution, colorization, inpainting, compressed sensing, and deblurring. DDNM only needs a pre-trained off-the-shelf diffusion model as the generative prior, without any extra training or network modifications. By refining only the null-space contents during the reverse diffusion process, we can yield diverse results satisfying both data consistency and realness. We further propose an enhanced and robust version, dubbed DDNM+, to support noisy restoration and improve restoration quality for hard tasks. Our experiments on several IR tasks reveal that DDNM outperforms other state-of-the-art zero-shot IR methods. We also demonstrate that DDNM+ can solve complex real-world applications, e.g., old photo restoration.

Range-Null Space Decomposition (RND)

RND is a well-known concept in the field of linear algebra. For linear inverse problems, the lost information is exactly the Null-space while the kept information is exactly the range-space. If we know the degradation operator, the range-space can be analytically solved, by which we can yield fully consistent results. Here's an example.

**RND for colorization task**: y is the input image, xr is a raw result predicted by the network. It is obvious that xr is not consistent to y. But after a simple RND operation, we can yield a fully consistent colorization result.

RND for Diffusion Models

Diffusion models use a denoiser to remove the noise and generate the next state. This actually yields a clean image x0|t at each step.

Original DDPM reverse diffusion process.

We apply RND to each clean image x0|t to yield consistent intermediate results.

DDNM reverse diffusion process. We apply RND to each clean image x0|t.

By changing the degradation A, we can use a pretrained diffusion model to solve arbitrary linear inverse problems, without any training or optimization. Below we visualize the intermediate results of DDNM.

DDNM for Noisy Restoration Tasks

DDNM provides an elegant solution for general linear image restoration problems defined by y=Ax. However, it can not handle noisy situations where y=Ax+n. We provide a simple but effective solution. Since y contains additional noise n, we add parameters Σt and Φt to make sure the total noise in xt-1 is invariant while maximizing the range-space preservation.

DDNM+ reverse diffusion process. Σt and Φt make sure the total noise level in xt-1 is the same as the original.

DDNM+ for Real-World Applications

Due to the property of pseudo-inverse, we can flexibly composite operators for real-world applications. Besides, the denoising mechanism is robust to various types of global artifacts, e.g., the JPEG artifacts. Here are some demos.

Related Works

GAN Prior based Null-Space Learning for Consistent Super-Resolution
The source template of this page: SR3

Citation

For more details and additional results, read the full paper.


  @article{wang2023ddnm,
  
    title={Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model},

    author={Wang, Yinhuai and Yu, Jiwen and Zhang, Jian},

    journal={The Eleventh International Conference on Learning Representations},

    year={2023}

  
  }