- batch file creation and obtaining performance data
- running all performance tools automatically
- comparing solver outcomes of solvers
- comparing resource times of two solvers
- terminating solvers at resource time limit
- performance profiles for solver comparisons

An overview of each of the tools is given below.

Try our PAVER Server for automated performance analysis and batch file creation.

**Note:**
Performance Tools 1.x has been superseded by PAVER 2.

- pprocess.gms - performs all performance tools on a given set of trace files and combines output in a summary HTML page.
- pprofile.gms - performance profile routine for solver comparisons for n solvers
- plotprof.gms - performance profile plotting routine using Gnuplotxy (Windows only)
- restime.gms - Resource time comparison utility for 2 solvers
- schulz.gms - termination routine to ensure solvers terminate at resource time limit
- square.gms - solver outcome comparison utility for 2 solvers
- timings.gms - creates resource time table for n different solvers

- A.trc - sample trace file for solver A
- B.trc - sample trace file for solver B
- C.trc - sample trace file for solver C

- readtrace.gms

- multiple models
- multiple solvers
- multiple options

The routine

Details on using

For example, suppose you have results using CONOTP1, CONOPT2, and CONOPT3 on an instance of the COPS models. It would be quite cumbersome to run all Performance Tools on all combinations of solvers, i.e. (CONOPT1, CONOPT2), (CONOPT1, CONOPT3), and (CONOPT2, CONOPT3) for all tools. This utility automates this task and summarizes the results in a single file.

Output is in the form of an HTML file, where each square entry is a link to the associated models.

Intput consists of two or more (up to 8) tracefiles, where each tracefile contains model run results for a single solver.

A sample pprocess.gms output: Details on using

Intput consists of two tracefiles, where each tracefile contains model run results for a single solver.

A sample square.gms output: Details on using

The thresholds for faster and much faster are 10% and 50% respectively. That is, if two solvers have resource times within 10% of eachother, they are considered the same, if one of them performs between 10% and 50% faster than the other, then it is faster, and if it performed more than 50% faster it is considered much faster. If different thresholds are desired, these can be specified on input.

The resource time utility creates two output files in HTML format (wth a text file format optional). The first compares only those models that were solved optimally by both solvers. The second compares all models listed in the tracefiles.

Intput consists of two tracefiles, where each tracefile contains model run results for a single solver.

Sample restimelp.gms output:

- out_resA.htm: compares only models solved optimally by both solvers
- out_resB.htm: compares all models listed in the two tracefiles

Several freeware utilities are needed for Windows users: Pslist and Pskill. Both are available for free from Sysinternals for non-commercial use. Download and place into your GAMS system directory.

UNIX users do not need to download, as equivalent utilities exist on UNIX systems.

- schulz.gms
- PsList from Sysinternals (only for Windows, UNIX users need not download)
- PsKill from Sysinternals (only for Windows, UNIX users need not download)

Initial performance comparisons were developed in [1] by Billups, Dirkse, and Ferris. In [2] Dolan and More expanded this approach to compare solver performance using

The profile computation routine is pprofile.gms. The user inputs different trace files, where each trace file contains information for one solver only. Users can input a maximum of 8 different trace files. The GAMS routine creates a text file containing a table of the performance profiles, which can be used to plot using a variety of software packages, such as Gnuplot, Excel or others.

Under Windows users can make use of the plotting routine Gnuplotxy created by Bruce Mccarl and Uwe Schneider. We have a performance profile plotting routine available called plotprof.gms which automates profile computation and plotting in one step. Examples are given below for users to download.

- pprofile.gms: creates performance profile text file
- plotprof.gms: plots performance profiles (currently only under Windows using gnuplotxy)
- Sample performance profile plot in .gif format
using Gnuplot. Created with
`plotprof.gms`. Actual solver name shave been changed to generic SolverA, SolverB, and SolverC to hide proprietary data. - Sample performance profile plot using Excel. Actual solver name shave been changed to generic SolverA, SolverB, and SolverC to hide proprietary data.
- gnuplotxy: GAMS/Gnuplot interface for plotting under Windows

- [1] Billups, Stephen C., Dirkse, Steven P., and Ferris, Michael C., "A comparison of large scale mixed complementarity problem solvers", Computational Optimizatoin and Applications, 7, pp. 3--25, 1997.
- [2] Dolan, Elizabeth D. and More, Jorge J., "Benchmarking optimization software with performance profiles", Mathematical Programming, Volume 91 (2), pp. 201-213, 2002.