Running NEOS from Python API

GAMS
1

Hi!

I’m able to run a GAMS/NEOS process through the NEOS button “RUN NEOS”,
Screenshot_1.png
I would like to run this process through python api, but for obious reasons I cannot actually press the “RUN NEOS” button from python. I’m running a very basic code in python, something like this:

    workdir = os.path.dirname(os.getcwd())
    ws = GamsWorkspace(workdir)
    job = ws.add_job_from_file('filename')
    job.run()

I’ve tried to change the solver to kestrel:

option solver = kestrel
SOLVE prop MINIMIZING ZFO USING MINLP;

But I get this error:

Connecting to: https://neos-server.org:3333
Traceback (most recent call last):
  File "C:\GAMS\35\gmske_nx.py", line 1049, in <module>
    kestrel.connectServer()
  File "C:\GAMS\35\gmske_nx.py", line 539, in connectServer
    reply = self.neos.ping()
  File "C:\GAMS\35\GMSPython\lib\xmlrpc\client.py", line 1109, in __call__
    return self.__send(self.__name, args)
  File "C:\GAMS\35\GMSPython\lib\xmlrpc\client.py", line 1450, in __request
    response = self.__transport.request(
  File "C:\GAMS\35\GMSPython\lib\xmlrpc\client.py", line 1153, in request
    return self.single_request(host, handler, request_body, verbose)
  File "C:\GAMS\35\GMSPython\lib\xmlrpc\client.py", line 1165, in single_request
    http_conn = self.send_request(host, handler, request_body, verbose)
  File "C:\GAMS\35\GMSPython\lib\xmlrpc\client.py", line 1278, in send_request
    self.send_content(connection, request_body)
  File "C:\GAMS\35\GMSPython\lib\xmlrpc\client.py", line 1308, in send_content
    connection.endheaders(request_body)
  File "C:\GAMS\35\GMSPython\lib\http\client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "C:\GAMS\35\GMSPython\lib\http\client.py", line 1010, in _send_output
    self.send(msg)
  File "C:\GAMS\35\GMSPython\lib\http\client.py", line 950, in send
    self.connect()
  File "C:\GAMS\35\GMSPython\lib\http\client.py", line 1424, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "C:\GAMS\35\GMSPython\lib\ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "C:\GAMS\35\GMSPython\lib\ssl.py", line 1040, in _create
    self.do_handshake()
  File "C:\GAMS\35\GMSPython\lib\ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1125)
ERR: Solver rc 1



How can I run the process with the normal green play button, but running in NEOS?

2

Hi,

It seems that the Kestrel solver uses an invalid certificate. Actually it should be aware of cacert.pem located in [GAMS dir]\GMSPython\Lib\site-packages\certifi.
Does the Kestrel solver work from command line, if you do not have a Python script in between? There is also a Python example that demonstrates job submission to NEOS: [GAMS dir]\apifiles\Python\transport_neos.py

This example has been adjusted after GAMS 35 regarding finding the right/valid certificate. For your convenience, here is the latest version of this file:

import os
import sys
import time
import xmlrpc.client
from gams import *

import ssl
import certifi

# NEOS XML Template (to be filled)
xml = r'''<document>
<category>:category:</category>
<solver>:solver:</solver>
<inputType>GAMS</inputType>
<email>:email:</email>
<priority>short</priority>
<model><![CDATA[:model:]]></model>
<wantgdx><![CDATA[yes]]></wantgdx>
<wantlog><![CDATA[yes]]></wantlog>
<wantlst><![CDATA[yes]]></wantlst>
</document>'''
    
if __name__ == "__main__":
    if len(sys.argv) > 1:
        ws = GamsWorkspace(system_directory = sys.argv[1])
    else:
        ws = GamsWorkspace()

    model = 'trnsport' # str | Name of the main .gms file
    ws.gamslib(model)

    ssl_context = ssl.create_default_context()
    # Explicitly point to 'certifi' *.pem file in case the OpenSSL default CA certificate path points to expired certificates
    if sys.platform == "win32":
        ssl_context.load_verify_locations(certifi.where())
    neos = xmlrpc.client.ServerProxy('https://neos-server.org:3333', context=ssl_context)
    alive = neos.ping()
    if alive != "NeosServer is alive\n":
        raise Exception('*** Could not make connection to NEOS Server')
    # can neither choose SoPlex nor CBC as LP solver on NEOS, so pretend its a MIP and use CBC
    xml = xml.replace(':category:', 'MILP')
    xml = xml.replace(':solver:', 'CBC')
    if 'NEOS_EMAIL' in os.environ:
        xml = xml.replace(':email:', os.environ['NEOS_EMAIL'])
    else:
        raise Exception("Environment variable 'NEOS_EMAIL' not found")
    with open(os.path.join(ws.working_directory, model+'.gms'), 'r') as f:
        xml = xml.replace(':model:', f.read())
    
    jobNumber, password = neos.submitJob(xml)
    print("Job number:", jobNumber)
    print("Job password:", password)
    
    if jobNumber == 0:
        raise Exception('*** NEOS Server error:' + password)
    
    offset = 0
    echo = True
    status = ''
    while status != 'Done':
        time.sleep(1)
        result, offset = neos.getIntermediateResults(jobNumber, password, offset)
        result = result.data.decode()
        if echo:
            if result.find('Composing results.') != -1: #this removes the lst output
                result = result.split('Composing results.', 1)[0]
                echo = False
            print(result, end='')
        status = neos.getJobStatus(jobNumber, password)
    
    result = neos.getFinalResults(jobNumber, password)
    with open(str(jobNumber)+'-'+os.path.splitext(os.path.basename(model+'.gms'))[0]+'.lst', 'w') as rf:
        rf.write(result.data.decode())
    result = neos.getOutputFile(jobNumber, password, 'solver-output.zip')
    with open(str(jobNumber)+'-'+'solver-output.zip', 'wb') as rf:
        rf.write(result.data)

Best,
Clemens

3

Hi Clemens,

Thanks for your response. The Kestrel solver is not working from GAMS, so it should not be a python script problem, as you describe, it is a certificate problem.

The file ‘cacert.pem’ is in the directory you point, but I dont know how to solve the invalid certificate issue.

Best,
Pitters

4

I assume that the error is the same when running Kestrel from command line. So it seems that setting the SSL_CERT_FILE in [GAMS dir]\gmske_nt.cmd is not enough.

If you allow me an experiment, could you change the following line of code in [GAMS dir]\gmske_nx.py (line 537):

      self.neos = xmlrpc.client.Server("%s://%s:%s" % (self.serverProtocol,self.serverHost,self.serverPort))

to:

      import ssl
      import certifi
      ssl_context = ssl.create_default_context()
      if sys.platform == "win32":
        ssl_context.load_verify_locations(certifi.where())
      self.neos = xmlrpc.client.Server("%s://%s:%s" % (self.serverProtocol,self.serverHost,self.serverPort), context=ssl_context)

Let me know if this helps or if you have problems modifying the file.

Best,
Clemens

5

Can we add data from excel file, model from texts and then use python to send it to NEOS ?

model = 'trnsport' # str | Name of the main .gms file
    ws.gamslib(model)

The example in GAMS uses a .gms file which contains the data and model both.
I am trying to build my model from the text and get data from excel as it’s huge.

Thank You!

6

NEOS does not allow you to run external programs like gdxxrw, so do the Excel to GDX conversion locally and submit gms and gdx to NEOS.

-Michael

7

Hi @Clemens,

This change solved the problem.

Thanks a lot!


8

Is it possible to export the optimization output in an excel file by running NEOS from Python API?