Optimization Tools

Optimization Functions

Functional wrapper around the pygmo, nlopt and scipy libraries.

estimagic.optimization.optimize.maximize(criterion, params, algorithm, criterion_kwargs=None, constraints=None, general_options=None, algo_options=None, gradient_options=None, logging='logging.db', log_options=None, dashboard=False, dash_options=None)[source]

Maximize criterion using algorithm subject to constraints and bounds.

Each argument except for general_options can also be replaced by a list of arguments in which case several optimizations are run in parallel. For this, either all arguments must be lists of the same length, or some arguments can be provided as single arguments in which case they are automatically broadcasted.

Parameters
  • criterion (callable or list of callables) –

    Python callable that takes a pandas DataFrame with parameters as the first argument. Supported outputs are:

    • scalar floating point

    • np.ndarray: contributions for the tao Pounders algorithm.

    • tuple of a scalar floating point and a pd.DataFrame:

      In this case the first output is the criterion value. The second output are the comparison_plot_data.

  • params (pd.DataFrame or list of pd.DataFrames) – See The params Argument.

  • algorithm (str or list of strings) – Specifies the optimization algorithm. See The algorithm Argument.

  • criterion_kwargs (dict or list of dicts) – Additional keyword arguments for criterion.

  • constraints (list or list of lists) – List with constraint dictionaries. See The constraints Argument.

  • general_options (dict) –

    Additional configurations for the optimization. Keys can include:

    • keep_dashboard_alive (bool): Do not terminate the dashboard process

      after the optimization(s) finish(es).

  • algo_options (dict or list of dicts) – Algorithm specific configurations for the optimization.

  • gradient_options (dict) – Options for the gradient function.

  • logging (str or pathlib.Path or list) – Path(s) to (an) sqlite3 file(s) which typically has the file extension .db. If the file does not exist, it will be created. See The logging and log_options Arguments for details.

  • log_options (dict or list of dict) – Keyword arguments to influence the logging. See The logging and log_options Arguments for details.

  • dashboard (bool) – Whether to create and show a dashboard, default is False. See The dashboard and dash_options Arguments for details.

  • dash_options (dict or list of dict, optional) –

    Options passed to the dashboard. Supported keys are:

    • port (int): port where to display the dashboard.

    • no_browser (bool): whether to display the dashboard in a browser.

    • rollover (int): how many iterations to keep in the convergence plots.

Returns

Each tuple consists of the harmonized result info dictionary and the params DataFrame with the minimizing parameter values of the untransformed problem as specified of the user.

Return type

results (tuple or list of tuples)

estimagic.optimization.optimize.minimize(criterion, params, algorithm, criterion_kwargs=None, constraints=None, general_options=None, algo_options=None, gradient_options=None, logging='logging.db', log_options=None, dashboard=False, dash_options=None)[source]

Minimize criterion using algorithm subject to constraints and bounds.

Each argument except for general_options can also be replaced by a list of arguments in which case several optimizations are run in parallel. For this, either all arguments must be lists of the same length, or some arguments can be provided as single arguments in which case they are automatically broadcasted.

Parameters
  • criterion (callable or list of callables) –

    Python callable that takes a pandas DataFrame with parameters as the first argument. Supported outputs are:

    • scalar floating point

    • np.ndarray: contributions for the tao Pounders algorithm.

    • tuple of a scalar floating point and a pd.DataFrame:

      In this case the first output is the criterion value. The second output are the comparison_plot_data.

  • params (pd.DataFrame or list of pd.DataFrames) – See The params Argument.

  • algorithm (str or list of strings) – Specifies the optimization algorithm. See The algorithm Argument.

  • criterion_kwargs (dict or list of dicts) – Additional keyword arguments for criterion.

  • constraints (list or list of lists) – List with constraint dictionaries. See The constraints Argument.

  • general_options (dict) –

    Additional configurations for the optimization. Keys can include:

    • keep_dashboard_alive (bool): Do not terminate the dashboard process

      after the optimization(s) finish(es).

  • algo_options (dict or list of dicts) – Algorithm specific configurations for the optimization.

  • gradient_options (dict) – Options for the gradient function.

  • logging (str or pathlib.Path or list) – Path(s) to (an) sqlite3 file(s) which typically has the file extension .db. If the file does not exist, it will be created. See The logging and log_options Arguments for details.

  • log_options (dict or list of dict) – Keyword arguments to influence the logging. See The logging and log_options Arguments for details.

  • dashboard (bool) – Whether to create and show a dashboard, default is False. See The dashboard and dash_options Arguments for details.

  • dash_options (dict or list of dict, optional) –

    Options passed to the dashboard. Supported keys are:

    • port (int): port where to display the dashboard.

    • no_browser (bool): whether to display the dashboard in a browser.

    • rollover (int): how many iterations to keep in the convergence plots.

Returns

Each tuple consists of the harmonized result info dictionary and the params DataFrame with the minimizing parameter values of the untransformed problem as specified of the user.

Return type

results (tuple or list of tuples)

Processing constraints

Process the user provided pc for use during the optimization.

The main purpose of this module is to convert the user provided pc into inputs for fast reparametrization functions. In the process, the pc are checked and consolidated. Consolidation means that redundant pc are dropped and other pc are collected in meaningful bundles.

To improve readability, the actual code for checking and consolidation are in separate modules.

The calls to the check functions are not in one place but scattered over the module. This is because we want to do each check as soon as it becomes possible, which allows to write error messages that are closely tied to the pc as written by the users and not some transformed version thereof. However, some checks can only be done after the consolidation.

The challenge in making this module readable is that after each function is applied the list of pc and the params dataframe will be slightly different, but it is not possible to reflect all of the changes in meaningful names because thy would get way too long. We chose the following conventions:

As soon as a list of pc or a params DataFrame is different from what the user provided they are called pc (for pc) and pp (for processed params) respectively. If all pc of a certain type, say linear, are collected in a this collection is called pc_linear.

If only few columns of processed params are used in a function, it is better to pass them as Series, to make the flow of information more explicit.

estimagic.optimization.process_constraints.process_constraints(constraints, params)[source]

Process, consolidate and check constraints.

Parameters
  • constraints (list) – List of dictionaries where each dictionary is a constraint.

  • params (pd.DataFrame) – see The params Argument.

Returns

A processed version of those constraints

that entail actual transformations and not just fixing parameters.

pp (pd.DataFrame): Processed params. A copy of params with additional columns:
  • _internal_lower: Lower bounds for the internal parameter vector. Those are derived from the original lower bounds and additional bounds implied by other constraints.

  • _internal_upper: As _internal_lower but for upper bounds.

  • _internal_free: Boolean column that is true for those parameters over which the optimizer will actually optimize.

  • _pre_replacements: The j_th element indicates the position of the internal parameter that has to be copied into the j_th position of the external parameter vector when reparametrizing from_internal, before any transformations are applied. Negative if no element has to be copied.

  • _post_replacements: As pre_replacements, but applied after the transformations are done.

  • _internal_fixed_value: Contains transformed versions of the fixed values that will become equal to the external fixed values after the kernel transformations are applied.

  • _is_fixed_to_value: True for parameters that are fixed to a value

  • _is_fixed_to_other: True for parameters that are fixed to another parameter

Return type

pc (list)

Enforcing constraints by reparametrizations and bounds

Handle pc by reparametrizations.

estimagic.optimization.reparametrize.reparametrize_to_internal(processed_params, processed_constraints)[source]

Convert a params DataFrame into a numpy array of internal parameters.

Parameters
  • processed_params (DataFrame) – A processed params DataFrame. See The params Argument.

  • processed_constraints (list) – Processed and consolidated pc.

Returns

1d numpy array of free reparametrized parameters.

Return type

internal_params (np.ndarray)

estimagic.optimization.reparametrize.reparametrize_from_internal(internal, fixed_values, pre_replacements, processed_constraints, post_replacements, processed_params)[source]

Convert a numpy array of internal parameters to a params DataFrame.

Parameters
  • internal (np.ndarray) – 1d numpy array with internal parameters

  • fixed_values (np.ndarray) – 1d numpy array with internal fixed values

  • pre_replacements (np.ndarray) – 1d numpy array with positions of internal parameters that have to be copied before transformations are applied. Negative if no value has to be copied.

  • processed_constraints (list) – List of processed and consolidated constraint dictionaries. Can have the types “linear”, “probability”, “covariance” and “sdcorr”.

  • post_replacments (np.ndarray) – 1d numpy array with parameter positions.

  • processed_params (pd.DataFrame) – See The params Argument

Returns

Copy of pp with replaced values.

Return type

updated_params (pd.DataFrame)

Utility functions for parameter handling

estimagic.optimization.utilities.cov_params_to_matrix(cov_params)[source]

Build covariance matrix from 1d array with its lower triangular elements.

Parameters

cov_params (np.array) – 1d array with the lower triangular elements of a covariance matrix (in C-order)

Returns

a covariance matrix

Return type

cov (np.array)

estimagic.optimization.utilities.sdcorr_params_to_matrix(sdcorr_params)[source]

Build covariance matrix out of variances and correlations.

Parameters

sdcorr_params (np.array) – 1d array with parameters. The dimensions of the covariance matrix are inferred automatically. The first dim parameters are assumed to be the variances. The remainder are the lower triangular elements (excluding the diagonal) of a correlation matrix.

Returns

a covariance matrix

Return type

cov (np.array)

estimagic.optimization.utilities.number_of_triangular_elements_to_dimension(num)[source]

Calculate the dimension of a square matrix from number of triangular elements.

Parameters

num (int) – The number of upper or lower triangular elements in the matrix.

Examples

>>> number_of_triangular_elements_to_dimension(6)
3
>>> number_of_triangular_elements_to_dimension(10)
4
estimagic.optimization.utilities.dimension_to_number_of_triangular_elements(dim)[source]

Calculate number of triangular elements from the dimension of a square matrix.

Parameters

dim (int) – Dimension of a square matrix.

estimagic.optimization.utilities.propose_algorithms(requested_algo, algos, number=3)[source]

Propose a a number of algorithms based on similarity to the requested algorithm.

Parameters
  • requested_algo (str) – From the user requested algorithm.

  • algos (dict(str, list(str))) – Dictionary where keys are the package and values are lists of algorithms.

  • number (int) – Number of proposals.

Returns

List of proposed algorithms.

Return type

proposals (list(str))

Example

>>> algos = {"scipy": ["L-BFGS-B", "TNC"], "nlopt": ["lbfgsb"]}
>>> propose_algorithms("scipy_L-BFGS-B", algos, number=1)
['scipy_L-BFGS-B']
>>> propose_algorithms("L-BFGS-B", algos, number=2)
['scipy_L-BFGS-B', 'nlopt_lbfgsb']
estimagic.optimization.utilities.robust_cholesky(matrix, threshold=None, return_info=False)[source]

Lower triangular cholesky factor of matrix.

Parameters
  • matrix (np.array) – Square, symmetric and (almost) positive semi-definite matrix

  • threshold (float) – Small negative number. Diagonal elements of D from the LDL decomposition between threshold and zero are set to zero. Default is minus machine accuracy.

  • return_info (bool) – If True, also return a dictionary with ‘method’. Method can take the values ‘np.linalg.cholesky’ and ‘Eigenvalue QR’.

Returns

Cholesky factor of matrix info (float, optional): see return_info.

Return type

chol (np.array)

Raises

np.linalg.LinalgError if an eigenvalue of matrix is below threshold.

In contrast to a regular cholesky decomposition, this function will also work for matrices that are only positive semi-definite or even indefinite. For speed and precision reasons we first try a regular cholesky decomposition. If it fails we switch to more robust methods.

estimagic.optimization.utilities.namedtuple_from_dict(field_dict)[source]

Filled namedtuple generated from a dictionary.

Example

>>> namedtuple_from_dict({'a': 1, 'b': 2})
NamedTuple(a=1, b=2)
estimagic.optimization.utilities.namedtuple_from_kwargs(**kwargs)[source]

Filled namedtuple generated from keyword arguments.

Example

>>> namedtuple_from_kwargs(a=1, b=2)
NamedTuple(a=1, b=2)
estimagic.optimization.utilities.namedtuple_from_iterables(field_names, field_entries)[source]

Filled namedtuple generated from field_names and field_entries.

Example

>>> namedtuple_from_iterables(field_names=['a', 'b'], field_entries=[1, 2])
NamedTuple(a=1, b=2)

The Scipy Algorithms

estimagic.optimization.scipy.minimize_scipy_np(func, x0, bounds, algo_name, algo_options=None, gradient=None)[source]

Interface for scipy.

Parameters
  • func (callable) – Objective function.

  • x0 (np.ndarray) – Starting values of the parameters.

  • bounds (Tuple[np.ndarray]) – A tuple containing two NumPy arrays where the first corresponds to the lower and the second to the upper bound. Unbounded parameters are represented by infinite values. The arrays have the same length as the parameter vector.

  • algo_name (str) – One of the optimizers of the scipy package which receives the same inputs as the "method" keyword of the original function.

  • algo_options (dict) – Options for the optimizer.

  • gradient (callable) – Gradient function.

Returns

Dictionary with processed optimization results.

Return type

results (dict)

The Pygmo Algorithms

estimagic.optimization.pygmo.minimize_pygmo_np(func, x0, bounds, origin, algo_name, algo_options, gradient=None)[source]

Minimize a function with pygmo.

Parameters
  • func (callable) – Objective function.

  • x0 (np.ndarray) – Starting values of the parameters.

  • bounds (Tuple[np.ndarray]) – A tuple containing two NumPy arrays where the first corresponds to the lower and the second to the upper bound. Unbounded parameters are represented by infinite values. The arrays have the same length as the parameter vector.

  • origin ({"nlopt", "pygmo"}) – Either an optimizer from NLOPT or pygmo.

  • algo_name (str) – One of the optimizers of the pygmo package.

  • algo_options (dict) – Options for the optimizer.

Returns

Dictionary with processed optimization results.

Return type

results (dict)

The Pounders Algorithm

Wrapper for pounders in the tao package.

estimagic.optimization.pounders.minimize_pounders_np(func, x0, bounds, gatol=1e-08, grtol=1e-08, gttol=1e-10, init_tr=None, max_iterations=None, n_errors=None)[source]

Minimize a function using the Pounders algorithm.

Pounders can be a useful tool for economists who estimate structural models using indirect inference, because unlike commonly used algorithms such as Nelder-Mead, Pounders is tailored for minimizing a non-linear sum of squares objective function, and therefore may require fewer iterations to arrive at a local optimum than Nelder-Mead.

The criterion function func() should return an array of the errors NOT an array of the squared errors or anything else.

Parameters
  • func (callable) – Objective function.

  • x0 (np.ndarray) – Starting values of the parameters.

  • bounds (Tuple[np.ndarray]) – A tuple containing two NumPy arrays where the first corresponds to the lower and the second to the upper bound. Unbounded parameters are represented by infinite values. The arrays have the same length as the parameter vector.

  • gatol (float) – Stop if relative norm of gradient is less than this. If set to False the algorithm will not consider gatol. Default is 1e-8.

  • grtol (float) – Stop if norm of gradient is less than this. If set to False the algorithm will not consider grtol. Default is 1e-8.

  • gttol (float) – Stop if norm of gradient is reduced by this factor. If set to False the algorithm will not consider grtol. Default is 1e-10.

  • init_tr (float) – Sets the radius for the initial trust region that the optimizer employs. If None the algorithm uses 100 as initial trust region radius. Default is None.

  • max_iterations (int) – Alternative Stopping criterion. If set the routine will stop after the number of specified iterations or after the step size is sufficiently small. If the variable is set the default criteria will all be ignored. Default is None.

  • n_errors (int or None) – The number of outputs of func are necessary to pre-allocate the results array. If the argument is None, evaluate the function once. This might be undesirable during dashboard optimizations.

Returns

Dictionary with processed optimization results.

Return type

results (dict)