Call GAMS within Parallel Loop in R

GAMS
1

Hello,

I previously wrote a parfor loop within Matlab that called a GAMS model with the system() function (e.g., system(‘gams storageGAMS’) where storageGAMS is the filename of the model). Each loop run in parallel provided different underlying data to the same Model equations. The computer was able to successfully open 4 different instances of GAMS (for the 4 cores on my computer) and run each model independently. Each worker then creates and reads a results GDX file to bring the results backs to Matlab.

For various reasons, I need to move from Matlab to R. I moved all the code over to R and wrote the loop with the foreach() function. Everything works correctly when I run it in serial computation mode (using %do% instead of %dopar%). When I try to run it in parallel, it quits the loop very quickly and says it can’t find the unloaded GDX file with the results in it. However, it quits the loop fast enough that it couldn’t possibly have even made it to the call to GAMS.

I created the parallel cluster using: cl<-makeCluster(4); registerDoParallel(cl)

I am loading the required packages into the parallel environment (gdxrrw and reshape2). Using .verbose=TRUE I made sure all the required functions and variables were being exported to each worker. Instead of the system() function in R, I tried using shell(), system2() and even the gams() function from the gdxrrw package to call the GAMS model but none of that worked.

Any ideas on how to make this work? It seems like it might have something to do with different underlying mechanics of the system() methods in Matlab and R but I am not savvy enough to understand the computer science.

Thanks,

Mike

–
To unsubscribe from this group and stop receiving emails from it, send an email to gamsworld+unsubscribe@googlegroups.com.
To post to this group, send email to gamsworld@googlegroups.com.
Visit this group at http://groups.google.com/group/gamsworld.
For more options, visit https://groups.google.com/d/optout.

2

Michael,

That’s interesting. >From your description it sounds like GAMS is not even getting off the ground. I’ve not tried to do this kind of thing in R before, but I’d be happy to have a look. Can you send an example, preferably something like a minimal reproducible example for the issue you’re describing?

Perhaps you can do this offline and post the results once we’ve got things sorted out.

-Steve

On Sun, Aug 30, 2015 at 9:46 AM, Michael Fisher wrote:

Hello,

I previously wrote a parfor loop within Matlab that called a GAMS model with the system() function (e.g., system(‘gams storageGAMS’) where storageGAMS is the filename of the model). Each loop run in parallel provided different underlying data to the same Model equations. The computer was able to successfully open 4 different instances of GAMS (for the 4 cores on my computer) and run each model independently. Each worker then creates and reads a results GDX file to bring the results backs to Matlab.

For various reasons, I need to move from Matlab to R. I moved all the code over to R and wrote the loop with the foreach() function. Everything works correctly when I run it in serial computation mode (using %do% instead of %dopar%). When I try to run it in parallel, it quits the loop very quickly and says it can’t find the unloaded GDX file with the results in it. However, it quits the loop fast enough that it couldn’t possibly have even made it to the call to GAMS.

I created the parallel cluster using: cl<-makeCluster(4); registerDoParallel(cl)

I am loading the required packages into the parallel environment (gdxrrw and reshape2). Using .verbose=TRUE I made sure all the required functions and variables were being exported to each worker. Instead of the system() function in R, I tried using shell(), system2() and even the gams() function from the gdxrrw package to call the GAMS model but none of that worked.

Any ideas on how to make this work? It seems like it might have something to do with different underlying mechanics of the system() methods in Matlab and R but I am not savvy enough to understand the computer science.

Thanks,

Mike

–
To unsubscribe from this group and stop receiving emails from it, send an email to gamsworld+unsubscribe@googlegroups.com.
To post to this group, send email to gamsworld@googlegroups.com.
Visit this group at http://groups.google.com/group/gamsworld.
For more options, visit https://groups.google.com/d/optout.



\

Steven Dirkse, Ph.D.
GAMS Development Corp., Washington DC
Voice: (202)342-0180 Fax: (202)342-0181
sdirkse@gams.com
http://www.gams.com

3

Hi Michael, Steve,

Did you chaps find a solution to this problem? I am faced with the identical problem. A system command seems to be ignored inside a foreach loop with dopar.

BR,
Emil

On Monday, 31 August 2015 02:01:51 UTC+2, Michael Fisher wrote:

Hello,

I previously wrote a parfor loop within Matlab that called a GAMS model with the system() function (e.g., system(‘gams storageGAMS’) where storageGAMS is the filename of the model). Each loop run in parallel provided different underlying data to the same Model equations. The computer was able to successfully open 4 different instances of GAMS (for the 4 cores on my computer) and run each model independently. Each worker then creates and reads a results GDX file to bring the results backs to Matlab.

For various reasons, I need to move from Matlab to R. I moved all the code over to R and wrote the loop with the foreach() function. Everything works correctly when I run it in serial computation mode (using %do% instead of %dopar%). When I try to run it in parallel, it quits the loop very quickly and says it can’t find the unloaded GDX file with the results in it. However, it quits the loop fast enough that it couldn’t possibly have even made it to the call to GAMS.

I created the parallel cluster using: cl<-makeCluster(4); registerDoParallel(cl)

I am loading the required packages into the parallel environment (gdxrrw and reshape2). Using .verbose=TRUE I made sure all the required functions and variables were being exported to each worker. Instead of the system() function in R, I tried using shell(), system2() and even the gams() function from the gdxrrw package to call the GAMS model but none of that worked.

Any ideas on how to make this work? It seems like it might have something to do with different underlying mechanics of the system() methods in Matlab and R but I am not savvy enough to understand the computer science.

Thanks,

Mike

–
To unsubscribe from this group and stop receiving emails from it, send an email to gamsworld+unsubscribe@googlegroups.com.
To post to this group, send email to gamsworld@googlegroups.com.
Visit this group at https://groups.google.com/group/gamsworld.
For more options, visit https://groups.google.com/d/optout.

4

Hi again,

Steve, I have put a minimal example on stackoverflow: http://stackoverflow.com/questions/34652510/calling-system2-within-a-parallel-loop-in-r

BR,
Emil

On Thursday, 7 January 2016 10:31:57 UTC+1, Emil Larsen wrote:


Hi Michael, Steve,

Did you chaps find a solution to this problem? I am faced with the identical problem. A system command seems to be ignored inside a foreach loop with dopar.

BR,
Emil

On Monday, 31 August 2015 02:01:51 UTC+2, Michael Fisher wrote:

Hello,

I previously wrote a parfor loop within Matlab that called a GAMS model with the system() function (e.g., system(‘gams storageGAMS’) where storageGAMS is the filename of the model). Each loop run in parallel provided different underlying data to the same Model equations. The computer was able to successfully open 4 different instances of GAMS (for the 4 cores on my computer) and run each model independently. Each worker then creates and reads a results GDX file to bring the results backs to Matlab.

For various reasons, I need to move from Matlab to R. I moved all the code over to R and wrote the loop with the foreach() function. Everything works correctly when I run it in serial computation mode (using %do% instead of %dopar%). When I try to run it in parallel, it quits the loop very quickly and says it can’t find the unloaded GDX file with the results in it. However, it quits the loop fast enough that it couldn’t possibly have even made it to the call to GAMS.

I created the parallel cluster using: cl<-makeCluster(4); registerDoParallel(cl)

I am loading the required packages into the parallel environment (gdxrrw and reshape2). Using .verbose=TRUE I made sure all the required functions and variables were being exported to each worker. Instead of the system() function in R, I tried using shell(), system2() and even the gams() function from the gdxrrw package to call the GAMS model but none of that worked.

Any ideas on how to make this work? It seems like it might have something to do with different underlying mechanics of the system() methods in Matlab and R but I am not savvy enough to understand the computer science.

Thanks,

Mike

–
To unsubscribe from this group and stop receiving emails from it, send an email to gamsworld+unsubscribe@googlegroups.com.
To post to this group, send email to gamsworld@googlegroups.com.
Visit this group at https://groups.google.com/group/gamsworld.
For more options, visit https://groups.google.com/d/optout.