# Optimizers#

Check out How to specify algorithms and algorithm specific options to see how to select an algorithm and specify algo_options when using maximize or minimize.

## Optimizers from scipy#

estimagic supports most scipy algorithms and scipy is automatically installed when you install estimagic.

## Own optimizers#

We implement a few algorithms from scratch. They are currently considered experimental.

## Optimizers from the Toolkit for Advanced Optimization (TAO)#

We wrap the pounders algorithm from the Toolkit of Advanced optimization. To use it you need to have petsc4py installed.

## Optimizers from the Numerical Algorithms Group (NAG)#

We wrap two algorithms from the numerical algorithms group. To use them, you need to install each of them separately:

• pip install DFO-LS

• pip install Py-BOBYQA

## PYGMO2 Optimizers#

Please cite [algo_18] in addition to estimagic when using pygmo. estimagic supports the following pygmo2 optimizers.

## The Interior Point Optimizer (ipopt)#

estimagic’s support for the Interior Point Optimizer ([algo_34], [algo_35], [algo_36], [algo_37]) is built on cyipopt, a Python wrapper for the Ipopt optimization package.

To use ipopt, you need to have cyipopt installed (conda install cyipopt).

## The Fides Optimizer#

estimagic supports the Fides Optimizer. To use Fides, you need to have the fides package installed (pip install fides>=0.7.4, make sure you have at least 0.7.1).

## The NLOPT Optimizers (nlopt)#

estimagic supports the following NLOPT algorithms. Please add the appropriate citations in addition to estimagic when using an NLOPT algorithm. To install nlopt run conda install nlopt.

## The SimOpt Optimizers (simopt)#

estimagic supports the following SimOpt algorithms. Please add the appropriate citations in addition to estimagic when using a SimOpt algorithm. To install simopt run pip install simoptlib==1.0.1.

## References#

algo_1(1,2)

Dieter Kraft. A software package for sequential quadratic programming. Technical Report, DLR German Aerospace Center – Institute for Flight Mechanics, Köln, Germany, 1988. URL: http://degenerateconic.com/wp-content/uploads/2018/03/DFVLR_FB_88_28.pdf.

algo_2

Fuchang Gao and Lixing Han. Implementing the nelder-mead simplex algorithm with adaptive parameters. Computational Optimization and Applications, 51(1):259–277, 2012.

algo_3(1,2,3,4,5,6,7,8)

Jorge Nocedal and Stephen Wright. Numerical optimization. Springer Science & Business Media, 2006.

algo_4

M Powell. A direct search optimization method that models the objective and constraint functions by linear interpolation, pages 51–67. Kluwer Academic, Dordrecht, 1994.

algo_5

Michael JD Powell. Direct search algorithms for optimization calculations. Acta numerica, pages 287–336, 1998.

algo_6

Michael JD Powell. A view of algorithms for optimization without derivatives. Mathematics Today-Bulletin of the Institute of Mathematics and its Applications, 43(5):170–174, 2007.

algo_7

Marucha Lalee, Jorge Nocedal, and Todd Plantenga. On the implementation of an algorithm for large-scale equality constrained optimization. SIAM Journal on Optimization, 8(3):682–706, 1998.

algo_8

AR Conn, NI Gould, and PL Toint. Nonlinear equations and nonlinear fitting. In Trust region methods, volume 1, pages 749–774. Siam, 2000.

algo_9

Richard H Byrd, Mary E Hribar, and Jorge Nocedal. An interior point algorithm for large-scale nonlinear programming. SIAM Journal on Optimization, 9(4):877–900, 1999.

algo_10

Ernst R. Berndt, Bronwyn Hall, Robert Hall, and Jerry Hausman. Estimation and inference in nonlinear structural models. Annals of Economic and Social Measurement, pages 653–665, 1974.

algo_11

Halbert White. Maximum likelihood estimation of misspecified models. Econometrica, 50(1):1–25, 1982.

algo_12(1,2)

S Benson, LC McInnes, JJ Moré, T Munson, and J Sarich. Tao user manual (revision 3.7). Technical Report, Technical Report ANL/MCS-TM-322, Argonne National Laboratory, 2017. URL: http://web.mit.edu/tao-petsc_v3.7/tao_manual.pdf.

algo_13(1,2)

Stefan M. Wild. Solving derivative-free nonlinear least squares problems with pounders. Technical Report, Argonne National Laboratory, 2015. URL: https://doi.org/10.1137/1.9781611974683.ch40.

algo_14(1,2,3)

Coralia Cartis, Jan Fiala, Benjamin Marteau, and Lindon Roberts. Improving the flexibility and robustness of model-based derivative-free optimization solvers. 2018. arXiv:1804.00154.

algo_15(1,2,3)

Michael JD Powell. The bobyqa algorithm for bound constrained optimization without derivatives. Cambridge NA Report NA2009/06, University of Cambridge, Cambridge, pages 26–46, 2009.

algo_16(1,2)

Coralia Cartis, Jan Fiala, Benjamin Marteau, and Lindon Roberts. Improving the flexibility and robustness of model-based derivative-free optimization solvers. 2018. arXiv:1804.00154.

algo_17(1,2,3)

Coralia Cartis, Lindon Roberts, and Oliver Sheridan-Methven. Escaping local minima with derivative-free methods: a numerical investigation. 2018. arXiv:1812.11343.

algo_18

Francesco Biscani and Dario Izzo. A parallel global multiobjective framework for optimization: pagmo. Journal of Open Source Software, 5(53):2338, 2020. URL: https://doi.org/10.21105/joss.02338, doi:10.21105/joss.02338.

algo_19

Martin Schlüter, Jose A. Egea, and Julio R. Banga. Extended ant colony optimization for non-convex mixed integer nonlinear programming. Computers & Operations Research, 36(7):2217–2229, jul 2009. doi:10.1016/j.cor.2008.08.015.

algo_20

Dervis Karaboga and Bahriye Basturk. A powerful and efficient algorithm for numerical function optimization: artificial bee colony (ABC) algorithm. Journal of Global Optimization, 39(3):459–471, apr 2007. doi:10.1007/s10898-007-9149-x.

algo_21

Marjan Mernik, Shih-Hsi Liu, Dervis Karaboga, and Matej Črepinšek. On clarifying misconceptions when comparing variants of the artificial bee colony algorithm by offering a new implementation. Information Sciences, 291:115–127, jan 2015. doi:10.1016/j.ins.2014.08.040.

algo_22

Rainer Storn and Kenneth Price. Differential evolution – a simple and efficient heuristic for global optimization over continuous spaces. Journal of Global Optimization, 11(4):341–359, 1997. URL: https://link.springer.com/article/10.1023/A:1008202821328.

algo_23(1,2)

Pietro S. Oliveto, Jun He, and Xin Yao. Time complexity of evolutionary algorithms for combinatorial optimization: a decade of results. International Journal of Automation and Computing, 4(3):281–293, jul 2007. doi:10.1007/s11633-007-0281-3.

algo_24

Janez Brest, Sao Greiner, Borko Boskovic, Marjan Mernik, and Viljem Zumer. Self-adapting control parameters in differential evolution: a comparative study on numerical benchmark problems. IEEE Transactions on Evolutionary Computation, 10(6):646–657, 2006. doi:10.1109/TEVC.2006.872133.

algo_25(1,2,3)

Saber M. Elsayed, Ruhul A. Sarker, and Daryl L. Essam. Differential evolution with multiple strategies for solving cec2011 real-world numerical optimization problems. In 2011 IEEE Congress of Evolutionary Computation (CEC), 1041–1048. 2011. doi:10.1109/CEC.2011.5949732.

algo_26(1,2)

Nikolaus Hansen. The CMA evolution strategy: a comparing review. In Towards a New Evolutionary Computation, pages 75–102. Springer Berlin Heidelberg, 2006. doi:10.1007/3-540-32494-1_4.

algo_27

A. Corana, M. Marchesi, C. Martini, and S. Ridella. Minimizing multimodal functions of continuous variables with the “simulated annealing” algorithm—corrigenda for this article is available here. ACM Trans. Math. Softw., 13(3):262–280, September 1987. URL: https://doi.org/10.1145/29380.29864, doi:10.1145/29380.29864.

algo_28(1,2)

Riccardo Poli, James Kennedy, and Tim Blackwell. Particle swarm optimization. Swarm Intelligence, 1(1):33–57, aug 2007. doi:10.1007/s11721-007-0002-0.

algo_29

David J. Wales and Jonathan P. K. Doye. Global optimization by basin-hopping and the lowest energy structures of lennard-jones clusters containing up to 110 atoms. The Journal of Physical Chemistry A, 101(28):5111–5116, jul 1997.

algo_30

Tobias Glasmachers, Tom Schaul, Sun Yi, Daan Wierstra, and Jürgen Schmidhuber. Exponential natural evolution strategies. In Proceedings of the 12th annual conference on Genetic and evolutionary computation - GECCO \textquotesingle 10. ACM Press, 2010. doi:10.1145/1830483.1830557.

algo_31

Seyedali Mirjalili, Seyed Mohammad Mirjalili, and Andrew Lewis. Grey wolf optimizer. Advances in Engineering Software, 69:46–61, mar 2014. doi:10.1016/j.advengsoft.2013.12.007.

algo_32

Tamara G. Kolda, Robert Michael Lewis, and Virginia Torczon. Optimization by direct search: new perspectives on some classical and modern methods. SIAM Review, 45(3):385–482, jan 2003. doi:10.1137/s003614450242889.

algo_33

M. Mahdavi, M. Fesanghary, and E. Damangir. An improved harmony search algorithm for solving optimization problems. Applied Mathematics and Computation, 188(2):1567–1579, may 2007. doi:10.1016/j.amc.2006.11.033.

algo_34(1,2)

Andreas Wächter and Lorenz T. Biegler. Line search filter methods for nonlinear programming: local convergence. SIAM Journal on Optimization, 16(1):32–48, jan 2005. doi:10.1137/s1052623403426544.

algo_35(1,2)

Andreas Wächter and Lorenz T. Biegler. Line search filter methods for nonlinear programming: motivation and global convergence. SIAM Journal on Optimization, 16(1):1–31, jan 2005. doi:10.1137/s1052623403426556.

algo_36(1,2)

Andreas Wächter and Lorenz T. Biegler. On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming. Mathematical Programming, 106(1):25–57, apr 2005. doi:10.1007/s10107-004-0559-y.

algo_37(1,2)

Jorge Nocedal, Andreas Wächter, and Richard A. Waltz. Adaptive barrier update strategies for nonlinear interior methods. SIAM Journal on Optimization, 19(4):1674–1693, jan 2009. doi:10.1137/060649513.

algo_38(1,2)

N Chiang, C Petra, and V Zavala. Structured nonconvex optimization of large-scale energy systems using pips-nlp. In Proceedings of the 18th power systems computation conference (PSCC). Wroclaw, Poland, 2014.

algo_39

Thomas F. Coleman and Yuying Li. On the convergence of interior-reflective newton methods for nonlinear minimization subject to bounds. Mathematcial Programming, 67(1-3):189–224, oct 1994. doi:10.1007/bf01582221.

algo_40

Thomas F. Coleman and Yuying Li. An interior trust region approach for nonlinear minimization subject to bounds. SIAM Journal on Optimization, 6(2):418–445, may 1996. doi:10.1137/0806023.

algo_41(1,2)

C. G. Broyden. A class of methods for solving nonlinear simultaneous equations. Mathematics of Computation, 19(92):577–577, 1965.

algo_42(1,2)

Jorge Nocedal and Stephen J. Wright, editors. Numerical Optimization. Springer-Verlag, 1999. doi:10.1007/b98874.

algo_43

J Nelder and R Mead. A simplex method for function minimization. The Computer Journal, 7:308–313, 1965.

algo_44

Richard Brent. Algorithms for minimization without derivatives. 1972.

algo_45

T Rowan. Functional stability analysis of numerical algorithms. 1990.

algo_46

M Powell. The newuoa software for unconstrained optimization without derivatives. In Proc. 40th Workshop on Large Scale Nonlinear Optimization. Erice, Italy, 2004.

algo_47

R Dembo and T Steihaug. Truncated newton algorithms for large-scale optimization. Math. Programming, 26:190–212, 1983. doi:10.1007/BF02592055.

algo_48

J Nocedal ; D and J Liu. On the limited memory bfgs method for large scale optimization. Math. Comput, 35:503–528, 1989.

algo_49

J. Nocedal. Updating quasi-newton matrices with limited storage. Math. Comput, 35:773–782, 1980.

algo_50(1,2)

Krister Svanberg. A class of globally convergent optimization methods based on conservative convex separable approximations. SIAM J. Optim, 12(2):555–573, 2002.

algo_51

J Vlcek and L Luksan. Shifted limited-memory variable metric methods for large-scale unconstrained minimization. J. Computational Appl. Math, 186:365–390, 2006.

algo_52

Dieter Kraft. Algorithm 733: tomp-fortran modules for optimal control calculations. ACM Transactions on Mathematical Software, 20(3):262–281, 1994.

algo_53

D Jones, C Perttunen, and B Stuckmann. Lipschitzian optimization without the lipschitz constant. J. Optimization Theory and Applications, 79:157, 1993.

algo_54

J Gablonsky and C Kelley. A locally-biased form of the direct algorithm. J. Global Optimization, 21(1):27–37, 2001.

algo_55

C Da Silva, M Santos, H Goncalves, and Hernandez-Figueroa. Designing novel photonic devices by bio-inspired computing. IEEE Photonics Technology Letters, 22(15):1177–1179, 2010.

algo_56

C Da Silva and Santos. Parallel and bio-inspired computing applied to analyze microwave and photonic metamaterial strucutures. 2010.

algo_57

H.-G Beyer and H.-P Schwefel. Evolution strategies: a comprehensive introduction. Journal Natural Computing, 1(1):3–52, 2002.

algo_58

W. Vent. Rechenberg, ingo, evolutionsstrategie — optimierung technischer systeme nach prinzipien der biologischen evolution. 170 s. mit 36 abb. frommann-holzboog-verlag. stuttgart 1973. broschiert. Feddes Repertorium, 86(5):337–337, 1975.

algo_59

Thomas Philip Runarsson and Xin Yao. Search biases in constrained evolutionary optimization. IEEE Trans. on Systems, Man, and Cybernetics Part C: Applications and Reviews, 35(2):233–243, 2005.

algo_60

P Thomas, Xin Runarsson, and Yao. Stochastic ranking for constrained evolutionary optimization. IEEE Trans. Evolutionary Computation, 4(3):284–294, 2000.

algo_61

P Kaelo and M Ali. Some variants of the controlled random search algorithm for global optimization. J. Optim. Theory Appl, 130(2):253–264, 2006.

algo_62

W Price. A controlled random search procedure for global optimization, pages 71–84. Volume 2. North-Holland Press, Amsterdam, 1978.

algo_63

W Price. Global optimization by controlled random search. J. Optim. Theory Appl, 40(3):333–348, 1983.