Python API: Database in GamsWorkspace from gams.transfer.Container/gdxmerge in Python

GAMS
1

Hello,

I like the ease of creating gdx databases using the Container provided by gams.transfer using Python. I wonder whether it is possible to transfer the container into an existing GamsWorkspace without storing the file on disk?

As a second question, would it be possible to merge several containers or databases into a single one within Python? Something similar to gdxmerge but again without storing the files to disk

Any suggestions are welcome!
Many thanks
Jan

2

Hi Jan,

Glad to hear that GT is working well for you. You do not write to a GamsWorkspace directly, but instead you can write to a GamsDatabase object.

import gams.transfer as gt
from gams import GamsWorkspace
from pathlib import Path

ws = GamsWorkspace(working_directory=Path.cwd())
db = ws.add_database()

m = gt.Container()
i = gt.Set(m, "i", records=list("abcd"))
a = gt.Parameter(m, "a", i, records=[("a", 1), ("b", 2), ("d", 4)])

m.write(db)

db.export("outfile.gdx")

We have to be a bit careful when merging. But Containers can read directly from other Containers or gdx files or GamsDatabase objects… so you can merge symbols this way and push to a new endpoint… but you will need to be careful about avoiding symbol name clashes. You cannot have two identical symbol names in a Container.

It is more of a manual process if you want to merge the records of several symbols together into one new symbol. The pandas.merge() method is very powerful and will probably get you far down that road. We will probably add in some new features to make merging more straightforward (and GAMS-like) in a future version of gams.transfer.

3

Dear Adam,

many thanks, that works perfect! And it works in both ways db → container and container → db :slight_smile:

My error was that I tried to create the db from the container but did not consider creating the db first and then writing the container to the existing db.

Consider merging, I see the problem. gdxmerge adds the filenames as an additional dimension to the symbols to avoid name conflicts. Within python I have to pass the name for this additional dimension. An initial approach based on the Container looks like (only for parameters):

import pandas as pd
import gams.transfer as gt

gdx1 = gt.Container()
gdx1.addSet("i", records=["a", "b", "c"])
gdx1.addParameter("cost", domain=["i"], records=[("a", 1), ("b", 2), ("d", 4)])
gdx2 = gt.Container()
gdx2.addSet("i", records=["a", "b", "c"])
gdx2.addParameter("cost", domain=["i"], records=[("a", 4), ("b", 5), ("d", 6)])
gdx_files = {
    "scen1": gdx1,
    "scen2": gdx2,
}

def merge_gdx(
    gdx_files:dict[str, gt.Container], 
    output_file: str| None = None,
    key_name: str = "scenario"    
) -> gt.Container:
    """Merge multiple gdx containers into a single one. The keys of the dictionary 
    are used to add an additional dimension to each symbol of the merge container.
    
    Args:
        gdx_files: A dictionary with the name of the gdx file as key and the container as value.
        output_file: The name of the output file. If None, the merged container is returned.
            and not written to disk.
        key_name: The name of the additional dimension.
            
    Returns:
        The merged container.
    """
    merged = gt.Container()
    for key, container in gdx_files.items():

        # loop over all symbols in the container
        for _, symbol in container:
            print(symbol.name)
            # add an additional dimension to the symbol and add it to the
            # merge container
            if isinstance(symbol, gt.Parameter):
                cols = [key_name] + symbol.domain + ["value"]
                records = symbol.records.assign(**{key_name: key})[cols]
                if symbol.name in merged:
                    records = pd.concat([merged[symbol.name].records, records])
                merged.addParameter(
                    symbol.name,
                    domain=[key_name] + symbol.domain,
                    records=records,
                )
    if output_file:
        merged.write(output_file)
    return merged
t = merge_gdx(gdx_files)
t["cost"].records

Many thanks for your help!
Best
Jan

1 Like