Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency
ICLR 2024
Summary
The proposed algorithm effectively solves general inverse problems using pre-trained LDMs. To ensure hard data consistency, an optimization problem is solved during the reverse sampling process to align the results with measurement y. This paper proposes a ReSample method to remap measurement-consistent samples back onto the noisy data manifold, through which State-of-the-Art (SOTA) performance is achieved across various linear and nonlinear inverse problems.
Questions
Q: Why is the ReSample process necessary after optimization?
A: The sample obtained through optimization is a clean latent z_0 that aligns well with measurement y. Therefore, a process is required to map it back to a latent at timestep t to continue the reverse sampling process.
Q: Are the model parameters updated during the sampling process?
A: During the sampling process, the encoder, decoder, and diffusion model are all used in their pre-trained states. While the latent z is updated during optimization, all other parameters remain frozen during sample generation.
Paper
Summary
The proposed algorithm effectively solves general inverse problems using pre-trained LDMs. To ensure hard data consistency, an optimization problem is solved during the reverse sampling process to align the results with measurement y. This paper proposes a ReSample method to remap measurement-consistent samples back onto the noisy data manifold, through which State-of-the-Art (SOTA) performance is achieved across various linear and nonlinear inverse problems.
Questions
Q: Why is the ReSample process necessary after optimization?
A: The sample obtained through optimization is a clean latent z_0 that aligns well with measurement y. Therefore, a process is required to map it back to a latent at timestep t to continue the reverse sampling process.
Q: Are the model parameters updated during the sampling process?
A: During the sampling process, the encoder, decoder, and diffusion model are all used in their pre-trained states. While the latent z is updated during optimization, all other parameters remain frozen during sample generation.