Optimization Tutorial

Details on each optimizer.

Additional details about optimization in JSim.

The JSim Optimizer GUI is reached from the Model RunTime GUI by clicking on the Optimizer TAB. The Optimizer GUI has three main parts:

(1) Config (for configuring the optimization),

(2) Graphs (for watching the progression of the optimization and displaying the residuals at the end of optimization), and

(3) Report (giving details of the optimization in progress and final summary).

Please read the notes inside the models. They are accessible from the RunTime page.

Model 1: Getting acquainted with the optimizers in JSim.

Model 2: What the optimization report means

Model 3: Problems with constrained optimization.

Model 4: What about data outliers?

The Config Gui has four main subdivisions:

(1) Choose the optimization algorithm and set its termination controls.

(2) Calculate the covariance matrix and confidence limits.

(3) Choose parameters to vary and set limits for constrained optimization.

(4) Choose the data to match, the model curve to match, and set point weights and curve weights.

Each parameter is listed along with termination controls that are specific to that optimizer choice and other pertinent details.

Some of the optimization termination controls apply to almost every optimizer and are described here. Max # runs is the maximum number of model evaluations to be performed. Min RMS error is the sum of the squares of the weighted residuals. Min par step is the minimum parameter relative step size (sometimes required of all parameters, otherwise required of any one single parameter).

There are seven choices for the algorithm which are in two major classes. The first four are constrained optimizers, which means that the minimum and maximum values for every parameter being optimized need to be specified.

(1) simplex (the simplex algorithm);

(2) gridsearch (a grid searching algorithm) # grid points is the number of divisions to divide the min and max range of parameters for model evaluation; Max # iter is the maximum number of times to repeat process. The number of model evaluations is approximately

Optimizing five parameters with five grid points for three iterations requires 9,232 model evaluations. Running gridsearch more than once gives the same estimated values for parameters unless the user changes the range being searched. If the parameter space needs to be exhaustively search, simanneal (simulated annealing) may be a better choice.

(3) sensop (weighted nonlinear least squares optimizer using a variant of Levenberg-Marquardt) Min gradient is the squareroot of the sum of the squares of the parameter displacements.

(4) simanneal (simulated annealing algorithm) Init temp controls the probability that an “uphill” move will be accepted. An uphill move is moving in the direction of a worse fit. As the temperature decreases, the probability of accepting an uphill move decreases. The temperature decreases as the optimization progresses. max # runs should be set to a fairly large value (>1000 usually) for effectively searching the parameter space. If the user wishes to look for parameter values close to the values they already are, the user should narrow the min and max ranges and lower the temperature so that fewer uphill values will be accepted.

The second group contains the unconstrained optimizers. They can require models to evaluate ranges of parameters that include negative values.

(5) ggopt (a variant of the Glad-Goldstein algorithm for smooth functions with added noise) Additional termination controls are Max # iter (see gridsearch) and Min gradient (see sensop). The optimization is terminated if the relative change in the sum of the squares of the weighted residuals is less than five times the relative error .

(6) neldermead (no additional termination parameters)

(7) nl2sol (no additional termination parameters)

Covariance mat is a check box. When this box is checked, the normalized covariance matrix of the parameters being optimized will be calculated and displayed on the optimization report along with symmetric confidence limits for the parameters. A user can tell if a set of optimized parameters contain “redundant” parameters. This is indicated by normalized covariance values approaching 1 or -1. When two or more parameters have high positive or negative covariance, some of them should be fixed, and omitted from the optimization. Be aware however that fixing one of the parameters will automatically determine the other parameter. If the parameters have a covariance of plus or minus 1, the confidence limits become NaN's (Not a Number) as it is no longer possible to invert the matrix giving the result. If N parameters are being optimized, it takes N model evaluations to calculate the covariance matrix. These N additional runs are not counted against “nmax # runs.” If an optimization is cancelled prematurely, the covariance matrix and confidence limits are not calculated. The covariance matrix and confidence limits are displayed in the optimization Report.

Parameters to optimize are displayed by a left-click on the downward arrow to the right of the parameter box. Each optimizer will display the data items that need to be filled in for the optimizer chosen. If step is displayed, do not enter zero (0) in the step boxes. Step is an absolute value, not a relative value. Choose values which are proportional to your parameter. The check box at the right of each parameter spedification turns the parameter “on” (parameter will be optimized) and off (blank box), parameter will not be optimized.

The Data to match allows the user to specify multiple data sets for optimization. Individual curves are specified from the triangle to the left of the curve box. Par/Expr allows the user to specify the model curve which will fit the data, as well as programming functions of the model curve.

Examples:

To fit a data curve with a weighted sum of two model curves:

ModelCurve1+2*ModelCurve2

To fit part of the data with one curve and another part with a different curve

if(t<t1) ModelCurve1 else ModelCurve2

Point Weights: Point weights are specified in the boxes in the column under Pwgt. The user must insure that any formula used for point weights does not produce NaNs. A zero weight is acceptable. The number of non-zero weights must exceed the number of parameters being optimized.

Enter 1 in Pwgt to weight all data equally.

Point weights using formulas. The formulas are entered in the Pwgt box and are written using the MML syntax except they are not terminated with a semi-colon.

Enter exp(-t/(1 sec)) in Pwgt for exponentially decaying weights when unit conversion is on and the realDomain variable t has units of seconds.

Enter if((10<=t) and (t<=20)) 1 else 0 in Pwgt to fit just the data between 10 and 20 seconds.

Enter if(fgen_1>0) fgen_1^(-1) else 0 in Pwgt when fgen_1 is set to DataCurve and the dataSet and name selected in fgen_1 match the data being optimized and logarithmic weighting of the data is wanted. This is helpful when the data ranges over orders of magnitude. The “if” prevents a NaN (not a number) being used as a weight. Any name used for the function generator can be used, e.g. fgen_2, concdata, etc. Note that the expression

Point weights calculated by model code:

Enter value in Pwgt where value (any name may be used) is defined by the following or similar model code:

// Data weights
extern real aCurve(t);
real power=0;
real value(t);
value= if(aCurve>0) aCurve^power else 0;

From the RunTime page, select a function generator for the input variable, aCurve. From the function generator page, select the last function, DataCurve. Load the appropriate dataSet and curve name to match the data being optimized. Any function can be specified for the weight. The above example happens to be the power function. On the RunTime page set power to 0 for equal weights (excludes data points with zero weights); set power to -0.7 (combination of equal and logarithmic weights); set power to -1.0 for logarithmic weights. Equal weights favor fitting the peaks, logarithmic weights favor fitting the logarithm of the data.

A useful piece of advice: If the parameter being optimized is used in an expression where the sensitivity is exponential instead of linear, the use of surrogate parameters is suggested. For example, optimizing for pH is preferred to hydrogen ion concentration because pH values between 5 and 9 are easier to optimized than hydrogen ion content values between 1e-5 and 1e-9 because of the greater sensitivity.

Curve Weights: Curve weights are used when multiple data curves are being optimized.

Check box: The check box is used to include (checked) or exclude (unchecked) curves in the optimization.

Optimization produces a set of graphs. The graphs can be viewed as the optimization proceeds allowing the user to monitor the progress of the optimizer.

Parameters (values of parameters as a function of model evaluations)

Normalized Parameters (same as above, but all normalized by their max and min and plotted on same scale.)

RMS error (as a function of model evaluations)

Data to Match Residuals (unweighted)(should appear to be randomly distributed around ordinate 0.

Residuals (weighted)

Point Weights

An optimization report is generated and updated while the optimization is running. The information in the report is mostly self-explanatory. For details about the covariance matrix and confidence limits, please see Section One: Config Gui: (2) Calculate the covariance matrix and confidence limits.

ggopt

1. Glad T. and A. Goldstein, Optimization of functions whose values are subject to small error, BIT 17 (1977), pp160-169.

2. Bassingthwaighte, J.B., I.S. Chan, A.A. Goldstein and I.B. Russak GGOPT, an unconstrained non-linear optimizer, Comput. Meth. Prog. Biomed. 26 (1988), pp192-202.

neldermead

1. Nelder and R. Mead, Computer Journal, 1965, vol 7, pp 308-313.

sensop

Chan I.S., Goldstein A.A., and Bassingthwaighte J.B. SENSOP: A derivative-free solver for non-linear least squares with sensitivity scaling. Ann Biomed Eng. 1993;21:621-631.

simanneal

# S. Kirkpatrick and C. D. Gelatt and M. P. Vecchi, Optimization by Simulated Annealing, Science, Vol 220, Number 4598, pages 671-680, 1983.

# V. Cerny, A thermodynamical approach to the travelling salesman problem: an efficient simulation algorithm. Journal of Optimization Theory and Applications, 45:41-51, 1985