Hypernetwork-Based Adaptive Image Restoration

Shai Aharon
Gil Ben-Artzi
Ariel University, Israel
Ariel University, Israel

2023 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2023)

Paper

Poster

Code



Highlights

  • Adaptive image restoration model to restore images with different degradation levels at inference time without the need to retrain the model.
  • Real time response time, with better accuracy than previous equivalent methods while reducing up to 75% of the number of parameters.
  • Supporting any number of degradation levels without the need to increase the model size.
  • Can be applied to various standard image restoration architectures without any change to the network structure.
  • The same model can be used for various image restoration tasks: denoising, DeJPEG and super-resolution.

Abstract

Adaptive image restoration models can restore images with different degradation levels at inference time without the need to retrain the model. We present an approach that is highly accurate and allows a significant reduction in the number of parameters. In contrast to existing methods, our approach can restore images using a single fixed-size model, regardless of the number of degradation levels. On popular datasets, our approach yields state-of-the-art results in terms of size and accuracy for a variety of image restoration tasks, including denoising, deJPEG, and super-resolution.

Performance



* Baseline is a dedicated model, trained on only a single noise level.





Method

Training: During training, the hypernetwork generates multiple main networks, each main network n i is optimized to restore a degraded image with a corresponding degradation level c i . The number of main networks (k) is fixed during the training process. The loss is averaged over all networks and the update is propagated to the hypernetwork.

Inference: Given a degraded image and an input degradation level c, we employ the learned weights of the hypernetwork θ h to generate the weights of a restoration network θ c . Each meta block generates the weights to match the degradation level.

The generation of the image restoration network for each degradation level is executed in a highly efficient manner, involving only a scalar multiplication. This allows the user to adjust the restoration level in real-time.

Open in Colab

Results

Super Resolution



Denoising


Paper


HYPERNETWORK-BASED ADAPTIVE IMAGE RESTORATION, S. Aharon, G. Ben-Artzi
Arxiv


Acknowledgements

The website template is from here.