GAMS-Readable Data Files

University of Colorado

* This research supported by the GAMS Applied General Equilibrium Research Fund. The software described here operates only with GAMS 2.25.089 or later on the PC. The author remains responsible for any bugs which exist in this software. This software is not officially supported by GAMS Corporation.

There had been a bug in the documentation. zip.exe and unzip.exe must be placed in the GAMS system directory in order to use the GAMS2ZIP and ZIP2GAMS routines. (Thanks to Randy Wigle for pointing out this error.)

A minor inconsistency with the latest release of GAMS has been corrected. We now save data with 13 digits following the decimal point rather than 14.

The six GAMS routines listed below can be used to write out parameters or level values in GAMS-readable form to an external file. This document specifies calling syntax and provides some examples of their use. Contents:

#### Installation

#### Syntax for

`gams2txt`#### Syntax for

`gams2prm`#### Syntax for

`gams2zip`#### Syntax for

`zip2gams`#### General Comments

#### Example 1: generic invocation

#### Example 2: Changing the name of a parameter.

#### Example 3: Invoking gams2txt within an if block.

#### Example 4: Reading data files with potentially empty arrays.

#### Example 5: Generating labeled output from multiple runs.

#### Example 6: Printing Zeros

#### Example 7: Using GAMS2ZIP and ZIP2GAMS .

#### Example 8: Using GAMS2ZIP and ZIP2GAMS to Save and Retrieve a Basis.

#### Example 9: Using GAMS2ZIP to archive results from multiple runs.

#### Appendix: How to Use Basis Information in GAMS

A number of libinclude routines must be installed in the GAMS include library subdirectory (LDIR) if you want to have these tools as a standard component of your GAMS installation.

Download `inclib.pck` into your GAMS
system directory, and run GAMSINST.

Download the appropriate INFO-ZIP executables, unzip these files into
a separate directory, and then copy `zip.exe` and `unzip.exe` into your
GAMS system directory (or another directory on the path). See the Info-Zip home page
for an overview of this software. They produce executables for a
large number of platforms, including:

N.B. The Unzip distribution file is a self-extracting archive. **Do
not rename this file!** Just execute
it and it will expand to produce a file named `unzip.exe`.

The file `inclib.pck` contains all of the libinclude programs
which have been developed at the University of Colorado during the
past couple of years. These include files for report writing (GAMS2TBL), spreadsheet interface , plotting (GNUPLOT), and interactive modeling
(GAMSCGI).
Henceforth I will be
keeping libinclude source files for all of these applications in a
single archive in order to simplify maintenance. You will need to
download additional batch and executable files in order to install
these other packages.

FILESome additional information:unit/name/; PUTunit; [$SETGLOBAL prefix "string"] [$SETGLOBAL timestamp "yesorno"] [$SETGLOBAL zeros "yesorno"] $LIBINCLUDE gams2txt [ident1[.suffix][ident2[.suffix]...]]

(i) GAMS2TXT works for parameters, variables, equations and sets.

(ii) The first invocation must be outside of a loop or if block.

(iii) Use a blank invocation (without arguments) to initialize.

(iv) Explanatory text are written as the "data" for sets.

(v) Values of EPS are written as zero on the output file.

(vi) Global environment variable "prefix" can specify a prefix to the output label (see Examples 5 and 9).

(vii) Global environment variable "timestamp" controls whether header comments are written to the output file. (Default is "yes".)

(vii) Global environment variable "zeros" controls whether zeros are written to the output file. (Default is "no". See Example 6.)

FILESome additional information:unit/name/; PUTunit; [$SETGLOBAL prefix "string"] [$SETGLOBAL timestamp "yesorno"] [$SETGLOBAL zeros "yesorno"] $LIBINCLUDE gams2prm [ident[.suffix] [name["description"]] ]

(i) GAMS2PRM works for parameters, variables, equations and sets.

(ii) Outputs only one item at a time.

(iii) The first invocation must be outside of a loop or if block.

(iv) Use a blank invocation (without arguments) to initialize.

(v) A name is required if the output item has a suffix.

(vi) Explanatory text are written as the "data" for SETs.

(vii) Values of EPS are written as zero on the output file.

(viii) Global environment variable "prefix" can specify a prefix to the output label (see Examples 5 and 9).

(ix) Global environment variable "timestamp" controls whether header comments are written to the output file. (Default is "yes".)

(x) Global environment variable "zeros" controls whether zeros are written to the output file. (Default is "no". See Example 6.)

$libinclude gams2zip [-m]Some additional information:archive[ident1[.suffix1] [-m] [-nname1] [ident2[.suffix2] [-m] [-nname2] ...]]

(i) Requires INFO-ZIP shareware to be in the GAMS directory or on your path.

(ii) GAMS2ZIP works for parameters, variables (level values or marginals), and equations (level values or marginals).

(iii) Output one or more items into a compressed archive.

(iv) The first invocation must be outside of a loop or if block.

(iv) Use a blank invocation (without arguments) to initialize.

(v) The item is written to the archive with a different name by including ("-n
*name*") following the item.

(vi) When an archive already exists, the default operation is "replacement".
Either one transfer or a single array may be entered with a "merge" by including an
"**-m**" flag at the start of the command (for the entire set), or following an
item (for a single array).

*N.B.* Specifying the **-m** flag for GAMS2ZIP is not
equivalent to applying a -m flag for ZIP.EXE. In the GAMS2ZIP call,
the -m flag indicates that the array is to be added (merged) to an
existing zip file.

$libinclude zip2gamsSome additional information:archive[ident1[-nname] [ident2...]]

(i) Requires INFO-ZIP shareware to be in the GAMS system directory.

(ii) Retrieves parameters, level values or marginals at compile time from an existing ZIP archive, as written by GAMS2ZIP.

(iii) Can retrieve multiple items at a time.

(iv) May only be invoked in a context which permits a parameter declaration. This means that every invocation must be outside of a loop or if block.

(iv) An item may be assigned values from an archive array which has a
different name through use of the "**-n**" flag. When this is
specified, the subsequent word defines the name of the archive array
to be assigned to the item.

(1) For the output routines `gams2txt` and `gams2prm`,
data is written to the currently active file.
*The user program must declare a file and make it current with a
PUT statement prior to invoking either routine*.

(2) The output may be a parameter, a variable or an equation. A variable or equation may only be output with a suffix, such as id.L (indicating level value), id.M (mararginal), id.LO (lower bound), or id.UP (upper bound).

(3) These routines may be used within a loop or if block only
if they are first initialized with blank invocations ("`$LIBINCLUDE
gams2prm`" or "`$LIBINCLUDE gams2txt`")
in a context where set and parameter declarations are permitted
(See Examples 3 and 5).

(4) The following names are used within these routines and may not be used in the calling program:

u__1,u__2,u__3,u__4,u__5,u__6,u__7,u__8,u__9,u__10, gms2txt1,gms2txt2,gms2txt3,gms2txt4,gms2txt5, gms2txt6,gms2txt7,gms2txt8,gms2txt9,gms2txt10

(5) To improve portability of GAMS code between PC and Unix
platforms, use a lower case reference to the batch file, `gams2txt`,
`gams2prm`, `gams2zip` or `zip2gams`.

(6) The routine `gams2prm` includes a generic `parameter`
statement with no reference to the domain or dimension of the
item. A program reading this data must provide an explicit domain
declaration if the parameter is to be domain-checked.

(7) Some care is needed when saving empty parameters. If you
save an empty (all zeroes) parameter to a file using `gams2prm`
and subsequently read the data file into a different GAMS
program, you must include the `$ONEMPTY` compiler
directive before reading the data file. Otherwise, GAMS
generates a compile-time error when it encounters an empty
parameter data range. See Example 4.

(8) The global environment variable 'prefix' can be used to add some characters to the output label. (See Examples 5 and 9.)

(9) `gams2zip` and `zip2gams` are a pair of routines
designed for storing a bunch of parameters or level values on an
external file for subsequent retrieval.

set i /i1*i3/; parameter x(i); x(i) = uniform(0,1); file kout /x.gms/; put kout; $libinclude gams2prm x

Example1 writes the following lines to `x.gms`:

parameter x / *=>gams2prm x * Called from D:\TFR\GAMS\GAMS2TXT\EXAMPLE1.GMS, line 70 * 01/08/98 16:15:19 I1 1.71747132000000E-01 I2 8.43266708000000E-01 I3 5.50375356000000E-01 /;

This data may be read into a subsequent GAMS program with the statements:

set i /i1*i3/; parameter x(i); $include x.gms

or simply:

$include x

The input code which declares set `i` and gives `x`
an explicit domain before reading `x.gms` is preferred
because it induces domain-checking on `x`.

This example is identical to the previous, except that the name of the output parameter is modified when it is written to disk.

set i /i1*i3/; parameter x(i); x(i) = uniform(0,1); file kout /y.gms/; put kout; $libinclude gams2prm x y

Example2 writes the following lines to `y.gms`:

parameter y / *=>gams2prm x y * Called from D:\TFR\GAMS\GAMS2TXT\EXAMPLE2.GMS, line 70 * 01/08/98 16:16:26 I1 1.71747132000000E-01 I2 8.43266708000000E-01 I3 5.50375356000000E-01 /;

Note that this example uses `gams2prm` within an if
block, so an initialization call is required.

set i /i1*i3/; parameter x(i); x(i) = uniform(0,1); file kout /x.gms/; put kout; $libinclude gams2prm if (x("i1") gt 10, $libinclude gams2prm x );

set i /i1*i2/; parameter x(i); $onempty $include x

The following batch file and GAMS program
illustrate how to solve a sequence of scenarios. We use the DOS `echo`
statement to write the file `scenario.gms`. This file is then
read by `example5.gms` which then appends output to `results.gms`
using the `gams2txt`.

This technique permits the solution of a sequence of scenarios without having the failure of one scenario interrupt the remaining cases. At the end, all the results are stored in a single GAMS-readable file, results.gms.

**EXAMPLE5.BAT**:

: A DOS batch file for solving a sequence of cases. if exist results.gms del results.gms echo run("sc1") = yes; >scenario.gms call gams example5 echo run("sc2") = yes; >scenario.gms call gams example5 echo run("sc3") = yes; >scenario.gms call gams example5

**EXAMPLE5.GMS**:

* This program for solves a specified scenarios and appends * the results to a GAMS-readable results file: set i /i1*i3/, sc set of all scenarios to solve / sc1*sc3/, run(sc) identifies the current scenario; parameter x(i) data parameter (solution values for current case); * Read identifier for the current scenario: $include scenario * Open the file in append mode and make it current: file kout /results.gms/; kout.ap=1; put kout; * Initialize prior to calling gams2txt from within a loop: $libinclude gams2txt * Use the scenario label as the prefix: $setglobal prefix "sc.tl'.'" loop(sc$run(sc), * Get the solution (this could be a solve statement): x(i) = uniform(0,1); * Write to the output file: $libinclude gams2txt x );

Example 5 writes the following lines to `results.gms`:

*=>gams2txt x * Called from D:\TFR\GAMS\GAMS2TXT\EXAMPLE5.GMS, line 287 * 01/08/98 16:00:19 SC1.I1 1.71747132000000E-01 SC1.I2 8.43266708000000E-01 SC1.I3 5.50375356000000E-01 *=>gams2txt x * Called from D:\TFR\GAMS\GAMS2TXT\EXAMPLE5.GMS, line 287 * 01/08/98 16:00:20 SC2.I1 1.71747132000000E-01 SC2.I2 8.43266708000000E-01 SC2.I3 5.50375356000000E-01 *=>gams2txt x * Called from D:\TFR\GAMS\GAMS2TXT\EXAMPLE5.GMS, line 287 * 01/08/98 16:00:20 SC3.I1 1.71747132000000E-01 SC3.I2 8.43266708000000E-01 SC3.I3 5.50375356000000E-01

Because the data was written with `gams2txt` instead of
`gams2prm`, the output file does not include a `PARAMETER`
statement. It may still be read into a GAMS program
as follows:

set i /i1*i3/; set sc /sc1*sc2/; parameter x(sc,i); parameter x / $include x /;

A global environment variable "zeros" specifies whether
`gams2txt` should write explicit zeros, as illustrated in this
example:

set i /i1,i2/, j /j1,j2/, k /k1,k2,k3/; table x(i,j,k) k1 k2 k3 i1.j1 1 i2.j2 2; file xd /xd.txt /; put xd; $libinclude gams2txt x

is written (under defaults):

I1.J1.K1 1.00000000000000E+00 I2.J2.K2 2.00000000000000E+00

When "$setglobal zeros yes", x is written:

I1.J1.K1 1.00000000000000E+00 I1.J1.K2 0.00000000000000E+00 I2.J2.K1 0.00000000000000E+00 I2.J2.K2 2.00000000000000E+00Note in this example that rows which are entirely zero (I1.J2, I2.J1), and columns which are entirely zero (K3) are not written.

$title ziptest1.gms -- Illustrates syntax for libinclude GAMS2ZIP * Generate some parameters with data to export: set i/1*100/; parameter a(i), b; a(i) = uniform(0,1); b(i) = uniform(0,1); * Also save level values for a variable: variable v(i); v.l(i) = uniform(0,1); * Create a zip archive named test.zip. * Parameter A saved with no change in name * (default). * Parameter B saved as a parameter named "BB" * through specification of an optional id. * Variable level values V.L: $libinclude gams2zip test a b -n bb v.l display a, b, v.l;

Here is a second program which reads the archive produced in the previous example. Note that domain checking is invoked for parameter B because it is declared with an explicit domain, but parameters A and V0 are not declared and therefore not checked.

$title ziptest2.gms -- Illustrates syntax for libinclude ZIP2GAMS set i/1*100/; parameter b(i); * Retrieve three items from the archive. Retrieve archive * array v.l into a parameter named v0: $libinclude zip2gams test a b v0 -n v.l display a, b, v0;Finally, here is a program which illustrates how ZIP2GAMS reports an error in the event that an item is retrieved which does not exist in the archive:

$title ziptest3.gms -- Illustrates error reporting * Attempt to retrieve a parameter which is not in the archive: $libinclude zip2gams test aaWhen an error code is generated by UNZIP, ZIP2GAMS first produces an error message on the log file and then requests UNZIP prints an archive list:

Error retrieving item aa from ZIP archive test.zip Archive: test.zip Length Date Time Name ------ ---- ---- ---- 2731 01-09-98 09:46 a 2733 01-09-98 09:46 bb 2736 01-09-98 09:46 v.l ------ ------- 8200 3

The following GAMS program illustrates how GAMS2ZIP can be used to save level values and marginals from one model to use as a starting point in another. (For further information on how GAMS and associated solvers communicate basis information, see the appendix listed below.)

$title dattest1.gms -- Illustrates the use of GAMS2ZIP for saving a basis * Use GAMSLIB call to retrieve the transport model into the * local directory: $call "gamslib trnsport" * Run the model (it solves a linear program): $include trnsport.gms * Write out the level values and marginals for this model (this * defines the basis for a model in GAMS) to basis.zip: $libinclude gams2zip basis x.l z.l x.m cost.m supply.m demand.m

Here is how the saved basis information can be read into a subsequent program:

$title dattest2.gms -- Illustrates the use of ZIP2GAMS for retrieving a basis. * Use GAMSLIB call to retrieve the transport model into the * local directory: $call "gamslib trnsport" * Run the model (it solves a linear program): $include trnsport.gms * Extract the level values and marginals from the basis: $libinclude zip2gams basis x.l z.l x.m cost.m supply.m demand.m * When solving the model, GAMS uses the level values and * marginals which have been read to provide a starting point: SOLVE TRANSPORT USING LP MINIMIZING Z ;

The following batch file and GAMS program are
virtually identical to Example 5, aside from the fact that in this
example several arrays are saved in the results file for a sequence
of scenarios. We use the DOS `echo` statement to write the
file `scenario.gms`. This file is then read by
`example9.gms` which then appends output to
`results.gms` using the `gams2txt`.

This technique permits the solution of a sequence of scenarios without having the failure of one scenario interrupt the remaining cases. At the end, all the results are stored in a single GAMS-readable file, results.gms.

**EXAMPLE9.BAT**:

: A DOS batch file for solving a sequence of cases. if exist results.zip del results.zip echo run("sc1") = yes; >scenario.gms call gams example9 echo run("sc2") = yes; >scenario.gms call gams example9 echo run("sc3") = yes; >scenario.gms call gams example9 call gams report9

**EXAMPLE9.GMS**:

* This program for solves a specified scenarios and appends * the results to a GAMS-readable results file: set i /i1*i3/, sc set of all scenarios to solve / sc1*sc3/, run(sc) identifies the current scenario; * Let's pretend that these are variables computed by the model: variable x(i) data parameter (one set of solution values for current case) y(i) data parameter (a second set solution values) z(i) data parameter (a third set solution values ); * Read identifier for the current scenario: $include scenario * First invocation of GAMS2ZIP must be outside a loop: $libinclude gams2zip * Use the scenario label as the prefix: $setglobal prefix "sc.tl'.'" loop(sc$run(sc), * Get the solution (this could be a solve statement): x.l(i) = uniform(0,1); y.l(i) = uniform(0,1); z.l(i) = uniform(0,1); * Save results in the archive (merge all of them!) $libinclude -m gams2zip results x.l y.l z.l );

**REPORT9.GMS**:

* This program generates a report from the previous run: set i /i1*i3/, sc set of all scenarios to solve / sc1*sc3/; * Retrieve the results -- note that in this context we * treat them as two-dimensional parameters: parameter x(sc,i) data parameter (one set of solution values for current case) y(sc,i) data parameter (a second set solution values) z(sc,i) data parameter (a third set solution values ); $libinclude zip2gams results x -n x.l y -n y.l z -n z.l display x, y, z;

The output produced by `REPORT9.GMS` assembles results from
all three sets of scenarios:

**REPORT9.LST** (partial listing):

---- 239 PARAMETER X data parameter (one set of solution values for current case) I1 I2 I3 SC1 0.172 0.843 0.550 SC2 0.172 0.843 0.550 SC3 0.172 0.843 0.550 ---- 239 PARAMETER Y data parameter (a second set solution values) I1 I2 I3 SC1 0.301 0.292 0.224 SC2 0.301 0.292 0.224 SC3 0.301 0.292 0.224 ---- 239 PARAMETER Z data parameter (a third set solution values ) I1 I2 I3 SC1 0.350 0.856 0.067 SC2 0.350 0.856 0.067 SC3 0.350 0.856 0.067

- Charles Rosa (Subject: Basis information passed solver?)
- Thomas Rutherford (Re: Basis information passed solver?)
- Erwin Kalvelagen (Re: Basis information passed solver?)
- Bruce McCarl (Re: Basis information passed solver?)
- Kurt Brinschwitz (Subject: Multiple solves)
- Arne Drud (Re: multiple solves)

From: Charles RosaSubject: Basis information passed solver? Does anyone know just what information is passed to the solver from GAMS besides initial values (e.g. x.l) of variables? If other information is passed, can this information be "reset" in the event that multiple solves are called. I ask this question because my model has multiple solves and it seems that some of the latter solves are failing (ending with the message: THE PROBLEM IS UNBOUNDED (OR BADLY SCALED)) because of the state of the system left by earlier solves. I'm quite certain that the error message isn't due to the problems actually being unbounded. If you have any ideas about this, please respond. Thanks very much, Charlie Rosa (rosa@iiasa.ac.at)

From: Thomas RutherfordSubject: Re: Basis information passed solver? GAMS passes the solver infomation about the marginals (EQU.M), but (so far as I know) not the exact numerical values of the multipliers. The NLP code uses this information(together with the level values and bounds) to install the initial basis. In many cases, if you are solving a sequence of unrelated cases, you may wish to "recenter" the model before each solve. I usually do this with an include file: loop(scenario, $INCLUDE bench.gms ! set benchmark values solve model using ... ); ! end of loop over scenarios -- Tom

From: Erwin KalvelagenSubject: Re: Basis information passed solver? About not using a basis in a solve: In order to let a solver reconstruct a basis GAMS passes on primal values and dual "indicators". It is good enough to know whether or not a variable is basic (this corresponds to a zero marginal) or non-basic (nonzero marginal; the levels can then be used to find out whether a variable is nonbasic at lowerbound or at upperbound). For NLP's the story is a little bit more complicated due to superbasics. This indicator function of the marginals is one of the reasons why EPS is important: this signals a nonbasic with marginal that is numerically zero (a form of degeneracy). If you add variables and equations to the model between two solves the default for these new rows and columns is basic (marginal=0). When the number of zero marginals becomes too large it may be better for the solver to start from scratch and crash a basis, because the model has changed a lot. This is where the obscure BRATIO option plays its role: if the number of zero marginals becomes too large the model is supposed to have changed too much and the basis is rejected. (I think the manual is wrong here, where it says in appendix C: "The use of this basis is rejected if the number of basic variables is smaller than BRATIO times the size of the basis" ). This simple mechanism works wonderfully, and is portable over different algorithms, ie. a basis created by BDMLP can be used by MINOS etc. Setting BRATIO to 1 will cause the basis to be rejected, and BRATIO=0 will cause the basis to be accepted unconditionally. Another way of forcing the basis to be rejected is to set all (or most of the) marginals to zero (don't forget the marginals of the equations). Erwin

From: Bruce McCarlSubject: Re: Basis information passed solver? I have written a program that does an advanced basis save. the information it generates is 1. levels and marginals for all variables. 2. marginals for all constraints you also must be careful about degeneracy and alternative optimals sending the values of eps for the level of variables in the basis at zero and the marginal of eps for alternative optimals My program does this automatically I have had problems similar to yours and have found the following strategies to be appropriate 1. when solving a sequence of runs trying to retain the basis then a. do not eliminate rows and or columns between runs rather adjust rhs and objective coeficients so the rows and columns are effectively eliminated. For example rather than setting up a model with a $ condition on the existance of an equation add in a large constant to the rhs that is conditional. I do this because GAMS suggests the number of basic eleents based on the row count and if you have removed rows and or columns it can mess you up 2. I have tried the suggestion by Rutherford but it doesnt work if you have a big model (50000+ variables 20000+ constraints) the inclusion of a lot of code in a loop blows up the memory requirements and causes you to use codex=1 or greater and this caused GAMS to go over the 48 megs in my work station 3. You may also find success by switching solvers. Alex thinks that OSL or cplex could overcome the problem. I have not tested this since my problems are generally nonlinear. I cann attest that minos does have problems Bruce McCarl

From: Kurt BrinschwitzSubject: multiple solves Dear colleagues, I am currently using a decomposition method to solve a problem. The basic algorithm is: 1 (solve set of subproblems) -> Get solutions vectors zeta. 2 Use optimal subproblem solutions (zeta) to define extra cut constraint 3) add cut constraint to Master problem, solve master to get optimal vectors z,y 4) using optimal z,y, redefine new set of subproblems. 5) goto 1 (if ending condition is not met) My question is this. Given that the master problems differ from iteration to iteration by only an extra constraint; and that the subproblems are of the same structure but differ in coefficient values. Does GAMS solve each problem from scratch, or does it use a warm start (ie starting point being that of the previous solution)? I know that if you solve the same model a number of times sequentially, modifying coefficients each time, then GAMS optimizes using the previous solution as starting point (an efficient method). I am switching between two different models: in iteration k, when solving the master problem, does GAMS known to start of with the Master problem solution from iteration k-1, and optimize from there? or does it just assume no information, and solve the whole problem from scratch every time. Thanks Kurt Brinschwitz

From: Arne DrudSubject: Re: multiple solves The GAMS manual mentions that GAMS will use the previous solution as much as possible. This may sound rather inaccurate; the background is the following: When a SOLVE statement is executed, i.e. when GAMS generates a model, it will use the current value and basis status of all variables and equations that appear in the model, and pass this information on to the solver. If the basis appear to be good measured by the number of good variables relative to the number of constraints, then GAMS will ask the solver to use the basis, and otherwise it should be ignored, see BRATIO in one of the appendices. If you design you model properly GAMS will automatically restart in the best way. However, there are some pitfalls. If you have a dynamic model with a variable X(T,I) and T is changed dynamically from one SOLVE to the next then GAMS cannot reused previous solution values; the value of X('1','ABC') may be known, but the model uses X('2','ABC'). If you remove the T index from the model GAMS may restart more intelligently and solution times may be reduced dramatically. Another example is in decomposition: You alter between model A and model B and both models depend on a variable, say X. When you solve B it will start from the optimal X value from A and when you solve A again it will start from the optimal X value from B. By proper choice of the names of variables you may transfer variables from one model to another or you may keep the models independent so that a second solve of a model not affected by an intermediate solve of another model. Regards Arne Stolbjerg Drud

Economics Department, University of Colorado, Boulder CO 80309-0256

Phone: (303) 492-5169, Fax: (303) 492-8969

email:rutherford@colorado.edu