Loading large but sparse SET from GDX file

GAMS
1

Dear GAMS users,

I am trying to improve/speed up the execution time of the following part of my code:

SET
arc /147677/
node /120626/;

SET
arclink(arc, node);
$gdxin data.gdx
$load arclink
$gdxin
display arclink;

Among all the 47677*20626 elements in the SET “arclink” stored in data.gdx, only 47677 of them are of value “true”. Put differently, in the 47677 by 20626 matrix, there are only 47677 non-zero. (i.e. for each arc, there is only one corresponding initial node). But loading the SET arclink from data.gdx is taking more than 1 hour and I am still waiting. I am using a fast PC.

I am wondering how do I speed up this process in this case where the matrix is large and sparse?

Thanks,
Huibin

–
To unsubscribe from this group and stop receiving emails from it, send an email to gamsworld+unsubscribe@googlegroups.com.
To post to this group, send email to gamsworld@googlegroups.com.
Visit this group at http://groups.google.com/group/gamsworld.
For more options, visit https://groups.google.com/d/optout.

2

Huibin,

That is strange. That should go super fast. This is not that much data. The most expensive statement here is the display, but even that should not take long. There must be something fundamentally weird. Can you make the data available. You might also want to contact support@gams.com.

Michael Bussieck - GAMSWorld Administrator

On Monday, April 14, 2014 10:25:07 PM UTC-4, Huibin Chang wrote:

Dear GAMS users,

I am trying to improve/speed up the execution time of the following part of my code:

SET
arc /147677/
node /120626/;

SET
arclink(arc, node);
$gdxin data.gdx
$load arclink
$gdxin
display arclink;

Among all the 47677*20626 elements in the SET “arclink” stored in data.gdx, only 47677 of them are of value “true”. Put differently, in the 47677 by 20626 matrix, there are only 47677 non-zero. (i.e. for each arc, there is only one corresponding initial node). But loading the SET arclink from data.gdx is taking more than 1 hour and I am still waiting. I am using a fast PC.

I am wondering how do I speed up this process in this case where the matrix is large and sparse?

Thanks,
Huibin

–
To unsubscribe from this group and stop receiving emails from it, send an email to gamsworld+unsubscribe@googlegroups.com.
To post to this group, send email to gamsworld@googlegroups.com.
Visit this group at http://groups.google.com/group/gamsworld.
For more options, visit https://groups.google.com/d/optout.

3

Hi,



My experience is that the GDX becomes slow when the Unique Element List is very large. When you have GAMS code with a lot of large sets and when you use an old GAMS version, the GDX becomes incredibly large (even if you write one very small parameter to a GDX file, i.e. because the UEL is also stored in the GDX). My guess is that this is happening here.



Solutions: use the latest GAMS version to run your GAMS code that creates your GDX. In this latest version the UEL list is minimized. Second possibility: use a tool to convert your GDX file into a new GDX where you only use a UEL with elements that are in the GDX data.





Greetings

Wietse



From: gamsworld@googlegroups.com [mailto:gamsworld@googlegroups.com] On Behalf Of Michael Bussieck
Sent: Tuesday, April 15, 2014 09:56
To: gamsworld@googlegroups.com
Subject: Re: Loading large but sparse SET from GDX file



Huibin,

That is strange. That should go super fast. This is not that much data. The most expensive statement here is the display, but even that should not take long. There must be something fundamentally weird. Can you make the data available. You might also want to contact support@gams.com.

Michael Bussieck - GAMSWorld Administrator

On Monday, April 14, 2014 10:25:07 PM UTC-4, Huibin Chang wrote:

Dear GAMS users,



I am trying to improve/speed up the execution time of the following part of my code:



SET

arc /1*47677/

node /1*20626/;



SET

arclink(arc, node);

$gdxin data.gdx

$load arclink

$gdxin

display arclink;



Among all the 47677*20626 elements in the SET “arclink” stored in data.gdx, only 47677 of them are of value “true”. Put differently, in the 47677 by 20626 matrix, there are only 47677 non-zero. (i.e. for each arc, there is only one corresponding initial node). But loading the SET arclink from data.gdx is taking more than 1 hour and I am still waiting. I am using a fast PC.



I am wondering how do I speed up this process in this case where the matrix is large and sparse?



Thanks,

Huibin

–
To unsubscribe from this group and stop receiving emails from it, send an email to gamsworld+unsubscribe@googlegroups.com.
To post to this group, send email to gamsworld@googlegroups.com.
Visit this group at http://groups.google.com/group/gamsworld.
For more options, visit https://groups.google.com/d/optout.

–
To unsubscribe from this group and stop receiving emails from it, send an email to gamsworld+unsubscribe@googlegroups.com.
To post to this group, send email to gamsworld@googlegroups.com.
Visit this group at http://groups.google.com/group/gamsworld.
For more options, visit https://groups.google.com/d/optout.

4

Michael,

Thank you. I tried removing the display statement, it is super fast. With display, it is still very slow.

I attached the input files.

Huibin


On Tue, Apr 15, 2014 at 2:56 AM, Michael Bussieck wrote:

Huibin,

That is strange. That should go super fast. This is not that much data. The most expensive statement here is the display, but even that should not take long. There must be something fundamentally weird. Can you make the data available. You might also want to contact support@gams.com.

Michael Bussieck - GAMSWorld Administrator


On Monday, April 14, 2014 10:25:07 PM UTC-4, Huibin Chang wrote:

Dear GAMS users,

I am trying to improve/speed up the execution time of the following part of my code:

SET
arc /147677/
node /120626/;

SET
arclink(arc, node);
$gdxin data.gdx
$load arclink
$gdxin
display arclink;

Among all the 47677*20626 elements in the SET “arclink” stored in data.gdx, only 47677 of them are of value “true”. Put differently, in the 47677 by 20626 matrix, there are only 47677 non-zero. (i.e. for each arc, there is only one corresponding initial node). But loading the SET arclink from data.gdx is taking more than 1 hour and I am still waiting. I am using a fast PC.

I am wondering how do I speed up this process in this case where the matrix is large and sparse?

Thanks,
Huibin

–
tranbpinput.gdx (1.08 MB)
Untitled_1.gms (2.44 KB)

5

Hi Weitse,

It seems that the display statement is the reason why this is taking too long. I am using the most recent GAMS distribution. Thank you for your comment on UEL. I will pay attention to that.

Thanks,
Huibin


On Tue, Apr 15, 2014 at 3:15 AM, Dol, Wietse wrote:

Hi,



My experience is that the GDX becomes slow when the Unique Element List is very large. When you have GAMS code with a lot of large sets and when you use an old GAMS version, the GDX becomes incredibly large (even if you write one very small parameter to a GDX file, i.e. because the UEL is also stored in the GDX). My guess is that this is happening here.



Solutions: use the latest GAMS version to run your GAMS code that creates your GDX. In this latest version the UEL list is minimized. Second possibility: use a tool to convert your GDX file into a new GDX where you only use a UEL with elements that are in the GDX data.





Greetings

Wietse



From: gamsworld@googlegroups.com [mailto:gamsworld@googlegroups.com] On Behalf Of Michael Bussieck
Sent: Tuesday, April 15, 2014 09:56
To: gamsworld@googlegroups.com
Subject: Re: Loading large but sparse SET from GDX file



Huibin,

That is strange. That should go super fast. This is not that much data. The most expensive statement here is the display, but even that should not take long. There must be something fundamentally weird. Can you make the data available. You might also want to contact support@gams.com.

Michael Bussieck - GAMSWorld Administrator

On Monday, April 14, 2014 10:25:07 PM UTC-4, Huibin Chang wrote:

Dear GAMS users,



I am trying to improve/speed up the execution time of the following part of my code:



SET

arc /1*47677/

node /1*20626/;



SET

arclink(arc, node);

$gdxin data.gdx

$load arclink

$gdxin

display arclink;



Among all the 47677*20626 elements in the SET “arclink” stored in data.gdx, only 47677 of them are of value “true”. Put differently, in the 47677 by 20626 matrix, there are only 47677 non-zero. (i.e. for each arc, there is only one corresponding initial node). But loading the SET arclink from data.gdx is taking more than 1 hour and I am still waiting. I am using a fast PC.



I am wondering how do I speed up this process in this case where the matrix is large and sparse?



Thanks,

Huibin

–
To unsubscribe from this group and stop receiving emails from it, send an email to gamsworld+unsubscribe@googlegroups.com.
To post to this group, send email to gamsworld@googlegroups.com.
Visit this group at http://groups.google.com/group/gamsworld.
For more options, visit https://groups.google.com/d/optout.

–
To unsubscribe from this group and stop receiving emails from it, send an email to gamsworld+unsubscribe@googlegroups.com.
To post to this group, send email to gamsworld@googlegroups.com.
Visit this group at http://groups.google.com/group/gamsworld.
For more options, visit https://groups.google.com/d/optout.

\

To unsubscribe from this group and stop receiving emails from it, send an email to gamsworld+unsubscribe@googlegroups.com.
To post to this group, send email to gamsworld@googlegroups.com.
Visit this group at http://groups.google.com/group/gamsworld.
For more options, visit https://groups.google.com/d/optout.

6

Huibin,

The display statement is slow in the matrix format. We need to write many many blanks to format properly. Displaying in a list statement is very fast. One should keep this in mind when displaying big data sets:

option arclinkia:0:0:1;
display arclinkia;

So that resolves the issue, I guess.

Best,
Michael Bussieck - GAMSWorld Coordinator

On Tuesday, April 15, 2014 10:04:50 AM UTC-4, Huibin Chang wrote:

Michael,

Thank you. I tried removing the display statement, it is super fast. With display, it is still very slow.

I attached the input files.

Huibin


On Tue, Apr 15, 2014 at 2:56 AM, Michael Bussieck wrote:

Huibin,

That is strange. That should go super fast. This is not that much data. The most expensive statement here is the display, but even that should not take long. There must be something fundamentally weird. Can you make the data available. You might also want to contact support@gams.com.

Michael Bussieck - GAMSWorld Administrator


On Monday, April 14, 2014 10:25:07 PM UTC-4, Huibin Chang wrote:

Dear GAMS users,

I am trying to improve/speed up the execution time of the following part of my code:

SET
arc /147677/
node /120626/;

SET
arclink(arc, node);
$gdxin data.gdx
$load arclink
$gdxin
display arclink;

Among all the 47677*20626 elements in the SET “arclink” stored in data.gdx, only 47677 of them are of value “true”. Put differently, in the 47677 by 20626 matrix, there are only 47677 non-zero. (i.e. for each arc, there is only one corresponding initial node). But loading the SET arclink from data.gdx is taking more than 1 hour and I am still waiting. I am using a fast PC.

I am wondering how do I speed up this process in this case where the matrix is large and sparse?

Thanks,
Huibin

–
To unsubscribe from this group and stop receiving emails from it, send an email to gamsworld+unsubscribe@googlegroups.com.
To post to this group, send email to gamsworld@googlegroups.com.
Visit this group at http://groups.google.com/group/gamsworld.
For more options, visit https://groups.google.com/d/optout.

\

To unsubscribe from this group and stop receiving emails from it, send an email to gamsworld+unsubscribe@googlegroups.com.
To post to this group, send email to gamsworld@googlegroups.com.
Visit this group at http://groups.google.com/group/gamsworld.
For more options, visit https://groups.google.com/d/optout.

7

Michael,

Yes, that resolves the issue. Thanks again!

Sincerely,
Huibin


On Tue, Apr 15, 2014 at 10:28 AM, Michael Bussieck wrote:

Huibin,

The display statement is slow in the matrix format. We need to write many many blanks to format properly. Displaying in a list statement is very fast. One should keep this in mind when displaying big data sets:

option arclinkia:0:0:1;
display arclinkia;

So that resolves the issue, I guess.

Best,
Michael Bussieck - GAMSWorld Coordinator


On Tuesday, April 15, 2014 10:04:50 AM UTC-4, Huibin Chang wrote:

Michael,

Thank you. I tried removing the display statement, it is super fast. With display, it is still very slow.

I attached the input files.

Huibin


On Tue, Apr 15, 2014 at 2:56 AM, Michael Bussieck wrote:

Huibin,

That is strange. That should go super fast. This is not that much data. The most expensive statement here is the display, but even that should not take long. There must be something fundamentally weird. Can you make the data available. You might also want to contact support@gams.com.

Michael Bussieck - GAMSWorld Administrator


On Monday, April 14, 2014 10:25:07 PM UTC-4, Huibin Chang wrote:

Dear GAMS users,

I am trying to improve/speed up the execution time of the following part of my code:

SET
arc /147677/
node /120626/;

SET
arclink(arc, node);
$gdxin data.gdx
$load arclink
$gdxin
display arclink;

Among all the 47677*20626 elements in the SET “arclink” stored in data.gdx, only 47677 of them are of value “true”. Put differently, in the 47677 by 20626 matrix, there are only 47677 non-zero. (i.e. for each arc, there is only one corresponding initial node). But loading the SET arclink from data.gdx is taking more than 1 hour and I am still waiting. I am using a fast PC.

I am wondering how do I speed up this process in this case where the matrix is large and sparse?

Thanks,
Huibin

–
To unsubscribe from this group and stop receiving emails from it, send an email to gamsworld+unsubscribe@googlegroups.com.
To post to this group, send email to gamsworld@googlegroups.com.
Visit this group at http://groups.google.com/group/gamsworld.
For more options, visit https://groups.google.com/d/optout.

\

To unsubscribe from this group and stop receiving emails from it, send an email to gamsworld+unsubscribe@googlegroups.com.
To post to this group, send email to gamsworld@googlegroups.com.
Visit this group at http://groups.google.com/group/gamsworld.
For more options, visit https://groups.google.com/d/optout.

\

To unsubscribe from this group and stop receiving emails from it, send an email to gamsworld+unsubscribe@googlegroups.com.
To post to this group, send email to gamsworld@googlegroups.com.
Visit this group at http://groups.google.com/group/gamsworld.
For more options, visit https://groups.google.com/d/optout.