{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Which optimizer to use\n",
    "\n",
    "This is a very very very short and oversimplifying guide on selecting an optimization algorithm based on a minimum of information. \n",
    "\n",
    "\n",
    "To select an optimizer, you need to answer two questions:\n",
    "\n",
    "1. Is your criterion function differentiable?\n",
    "\n",
    "2. Do you have a nonlinear least squares structure (i.e. do you sum some kind of squared residuals at the end of your criterion function)?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define some inputs\n",
    "\n",
    "Again, we use versions of the sphere function to illustrate how to select these algorithms in practice"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import estimagic as em\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sphere(params):\n",
    "    out = {\n",
    "        \"value\": params @ params,\n",
    "        \"root_contributions\": params,\n",
    "    }\n",
    "    return out\n",
    "\n",
    "\n",
    "def sphere_gradient(params):\n",
    "    return params * 2\n",
    "\n",
    "\n",
    "start_params = np.arange(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Differentiable criterion function\n",
    "\n",
    "Use `scipy_lbfsgsb` as optimizer and provide the closed form derivative if you can. If you do not provide a derivative, estimagic will calculate it numerically. However, this is less precise and slower. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res = em.minimize(\n",
    "    criterion=sphere,\n",
    "    params=start_params,\n",
    "    algorithm=\"scipy_lbfgsb\",\n",
    "    derivative=sphere_gradient,\n",
    ")\n",
    "res.n_criterion_evaluations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that this solves a 5 dimensional problem with just 3 criterion evaluations. For higher dimensions, you will need more, but it scales very well to dozens and hundreds of parameters. \n",
    "\n",
    "If you are worried about being stuck in a local optimum, use multistart optimization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Not differentiable, only scalar output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use `nag_pybobyqa`. Note that for this you need to install the `PyBOBYQA` package if you do not already have it:\n",
    "    \n",
    "`pip install Py-BOBYQA`\n",
    "\n",
    "Then you select the algorithm as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "33"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res = em.minimize(\n",
    "    criterion=sphere,\n",
    "    params=start_params,\n",
    "    algorithm=\"nag_pybobyqa\",\n",
    ")\n",
    "res.n_criterion_evaluations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Not differentiable, least squares structure\n",
    "\n",
    "Use `nag_dfols`. To use `nag_dfols`, you need to install it via:\n",
    "\n",
    "`pip install DFO-LS`\n",
    "\n",
    "\n",
    "This optimizer will only work if your criterion function returns a dictionary that contains the entry `root_contributions`. This needs to be a numpy array or pytree that contains the residuals of the least squares problem. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res = em.minimize(\n",
    "    criterion=sphere,\n",
    "    params=start_params,\n",
    "    algorithm=\"nag_dfols\",\n",
    ")\n",
    "res.n_criterion_evaluations"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}