Separable nonlinear least squares: the variable projection method and its applications

Analysis

Why This Paper Matters

This paper is a seminal review of the variable projection (VP) method, a technique that has profoundly impacted how separable nonlinear least-squares problems are solved. Since its introduction in the 1970s, the VP method has been a cornerstone for problems where the model is a linear combination of nonlinear functions—a structure that appears frequently in exponential fitting, signal processing, and inverse problems. The paper consolidates 30 years of theoretical developments and practical implementations, making it an essential reference for researchers and practitioners.

The significance lies in the method's ability to reduce the dimensionality of the optimization problem by analytically eliminating linear parameters. This not only simplifies the problem but also improves its conditioning, leading to faster and more robust convergence. The review highlights that the VP method is particularly effective for real and complex exponential models, which are notoriously difficult to solve with standard techniques. By providing a unified treatment, the paper has enabled broader adoption across diverse fields, from medical imaging to robotics.

Technical Contributions

The paper's key technical contributions include:

• Variable Projection Formulation: A rigorous derivation of the reduced objective function that depends only on nonlinear parameters, achieved by projecting out linear variables.
• Convergence Analysis: Theoretical results showing that the reduced problem is better conditioned and converges faster than the original when using the same optimization algorithm.
• Algorithmic Implementations: A historical account of various computer implementations, from early FORTRAN codes to more modern approaches, highlighting practical considerations.
• Application Spectrum: A comprehensive survey of applications, including exponential fitting in chemistry, image reconstruction in medical imaging, and parameter estimation in robotics and vision.
• Extensive Bibliography: A curated list of references that serves as a roadmap for researchers entering the field.

Results

As a review paper, it does not present new experimental results. However, it synthesizes findings from numerous studies that demonstrate the VP method's advantages. For instance, it reports that the VP method consistently outperforms full-space optimization in terms of convergence speed and numerical stability. The paper also notes that the method is particularly effective for ill-conditioned exponential fitting problems, where standard approaches often fail. The extensive bibliography provides evidence of the method's widespread success across applications.

Significance

The variable projection method has had a lasting impact on the field of optimization and inverse problems. It has become the method of choice for separable least-squares problems, especially in exponential fitting, which is pervasive in science and engineering. The paper's comprehensive review has helped standardize the approach and has inspired further research into extensions and variants. Its influence extends beyond the original domains, with applications in machine learning, where separable structures appear in neural network training and dictionary learning. The method's ability to reduce problem complexity and improve conditioning remains highly relevant in modern computational practice, making this paper a timeless reference.