Model takes a lot of time to generate

Archiver May 4, 2011, 4:02pm 1

Hello Everybody,

I have an interesting problem which I have been battling against for some time.

I am running a MIP model on a multiprocessor PC with 64 -bit windows 7 and Gams 22.3 and ILOG CPLEX 12.1 MIP solver.

I have a problem that has happened many times when I go from a small example to a largescale example.

So far I have survived by reducing the data set by reducing the number of periods so I can complete the run of the model, but when I increase the time periods from about 12 months (which takes arround 10 minutes) to arround 120 that is a realcase setting this is what happens.

I find that the processor gets about 100% busy, then I find the memory gradually increasing and gets over 90% ocupied or so. Simultaneously the processor usage drops to arround 2% of processor capacity and then it will takes hours and days to complete the job.

All this happens when GAMS is still generating the model. Maybe the processing slows down because of memory ocupation, and as a consequence takes an ieternity to complete.

I have 8Gbytes of memory and 8 processores and still have problems. I need to get over this problem so I can fully test my model.

Can anybody share some light of what might be happening and how I may resolve the situation. This is the print of the GAMS screen.

Thank You,

LeÃ£o Fernandes.

–
To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

Archiver May 5, 2011, 5:32am 2

Dear LeÃ£o
Please refer to this link:
http://www.gams.com/mccarl/speed.pdf
it may be useful. Some of the tips there has worked for me.
Regards

Ù…ØÙ…Ø¯ ØµØ§Ø¯Ù‚ ØªÙˆÙ„Ù‘Ù„ÛŒ
Mohammad Sadegh Tavallali
National University of Singapore

2011/5/4 LeÃ£o Fernandes

Hello Everybody,

I have an interesting problem which I have been battling against for some time.

I am running a MIP model on a multiprocessor PC with 64 -bit windows 7 and Gams 22.3 and ILOG CPLEX 12.1 MIP solver.

I have a problem that has happened many times when I go from a small example to a largescale example.

So far I have survived by reducing the data set by reducing the number of periods so I can complete the run of the model, but when I increase the time periods from about 12 months (which takes arround 10 minutes) to arround 120 that is a realcase setting this is what happens.

I find that the processor gets about 100% busy, then I find the memory gradually increasing and gets over 90% ocupied or so. Simultaneously the processor usage drops to arround 2% of processor capacity and then it will takes hours and days to complete the job.

All this happens when GAMS is still generating the model. Maybe the processing slows down because of memory ocupation, and as a consequence takes an ieternity to complete.

I have 8Gbytes of memory and 8 processores and still have problems. I need to get over this problem so I can fully test my model.

Can anybody share some light of what might be happening and how I may resolve the situation. This is the print of the GAMS screen.

Thank You,

LeÃ£o Fernandes.

–
To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

\

To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

Archiver May 5, 2011, 12:24pm 3

Hello Tavallali,

Thank you for your reply. I have used this document and the recommendations on this model so I can say it is very useful. However I have not used all of the recommendations, especially the one â€œConsidering Unnecessary Casesâ€ which I think can make a big difference. The reason why I havenâ€™t used it so far is because I am afraid as the conditionals may interfere with relevant cases so one has to be extremely careful in filtering these unnecessary cases consistently in the various constraints.

I will take a new look at the document and try to implement conditionals on various constraints. But is there another way of improving performance for such cases while at the same time guaranteeing that relevant case will not be overlooked or filtered due to these conditionals?

Melhores Cumprimentos,

LeÃ£o Fernandes.

From: gamsworld@googlegroups.com [mailto:gamsworld@googlegroups.com] On Behalf Of Tavallali
Sent: quinta-feira, 5 de Maio de 2011 04:33
To: gamsworld@googlegroups.com
Subject: Re: Model takes a lot of time to generate

Dear LeÃ£o
Please refer to this link:
http://www.gams.com/mccarl/speed.pdf
it may be useful. Some of the tips there has worked for me.
Regards

Ù…ØÙ…Ø¯ ØµØ§Ø¯Ù‚ ØªÙˆÙ„Ù‘Ù„ÛŒ
Mohammad Sadegh Tavallali
National University of Singapore

2011/5/4 LeÃ£o Fernandes

Hello Everybody,

I have an interesting problem which I have been battling against for some time.

I am running a MIP model on a multiprocessor PC with 64 -bit windows 7 and Gams 22.3 and ILOG CPLEX 12.1 MIP solver.

I have a problem that has happened many times when I go from a small example to a largescale example.

So far I have survived by reducing the data set by reducing the number of periods so I can complete the run of the model, but when I increase the time periods from about 12 months (which takes arround 10 minutes) to arround 120 that is a realcase setting this is what happens.

I find that the processor gets about 100% busy, then I find the memory gradually increasing and gets over 90% ocupied or so. Simultaneously the processor usage drops to arround 2% of processor capacity and then it will takes hours and days to complete the job.

All this happens when GAMS is still generating the model. Maybe the processing slows down because of memory ocupation, and as a consequence takes an ieternity to complete.

I have 8Gbytes of memory and 8 processores and still have problems. I need to get over this problem so I can fully test my model.

Can anybody share some light of what might be happening and how I may resolve the situation. This is the print of the GAMS screen.

Thank You,

LeÃ£o Fernandes.

–
To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

\

To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

–
To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

Archiver May 5, 2011, 12:55pm 4

Thank you for your reply.

You may like to consider pre-calculating the equations that are repeatedly used in the model, or defining a subset, instead of using combined conditions each time.

Also you may like to consider this :
Apart from the mathematics behind the solution algorithm, the model should be implemented in the most efficient way to reduce the execution time. As an example I would like to point to the storage syntax. Consider the following sample assignment:

dlambda_o_dp(l,ts,i) $(ord(i)=iPM) = f(l,ts,i); (a)

loop(i$(ord(i)=iPM),dlambda_o_dp(l,ts,i)= f(l,ts,i); ); (b)

I have tested the above two assignments and checked the execution time per iteration for both cases using OPTION PROFILE =3. Compare the outcome:

---- 1720 Assignment dlambda_o_dp 1.328 1.516 SECS 5 Mb 11250 (a)

---- 1721 Assignment dlambda_o_dp 0.016 0.188 SECS 5 Mb 11250 (b)

It is evident that the execution time of the second assignment is less than the first assignment ;

Apart from these small tricks, I strongly suggest that you try removing the unnecessary cases. Spending one day and thinking deeply and carefully on that can save you a lot of time!

I hope it will help.

Mohammad Sadegh Tavallali
National University of Singapore

2011/5/5 LeÃ£o Fernandes

Hello Tavallali,

Thank you for your reply. I have used this document and the recommendations on this model so I can say it is very useful. However I have not used all of the recommendations, especially the one â€œConsidering Unnecessary Casesâ€ which I think can make a big difference. The reason why I havenâ€™t used it so far is because I am afraid as the conditionals may interfere with relevant cases so one has to be extremely careful in filtering these unnecessary cases consistently in the various constraints.

I will take a new look at the document and try to implement conditionals on various constraints. But is there another way of improving performance for such cases while at the same time guaranteeing that relevant case will not be overlooked or filtered due to these conditionals?

Melhores Cumprimentos,

LeÃ£o Fernandes.

From: gamsworld@googlegroups.com [mailto:gamsworld@googlegroups.com] On Behalf Of Tavallali
Sent: quinta-feira, 5 de Maio de 2011 04:33
To: gamsworld@googlegroups.com
Subject: Re: Model takes a lot of time to generate

Dear LeÃ£o
Please refer to this link:
http://www.gams.com/mccarl/speed.pdf
it may be useful. Some of the tips there has worked for me.
Regards

Ù…ØÙ…Ø¯ ØµØ§Ø¯Ù‚ ØªÙˆÙ„Ù‘Ù„ÛŒ
Mohammad Sadegh Tavallali
National University of Singapore

2011/5/4 LeÃ£o Fernandes

Hello Everybody,

I have an interesting problem which I have been battling against for some time.

I am running a MIP model on a multiprocessor PC with 64 -bit windows 7 and Gams 22.3 and ILOG CPLEX 12.1 MIP solver.

I have a problem that has happened many times when I go from a small example to a largescale example.

So far I have survived by reducing the data set by reducing the number of periods so I can complete the run of the model, but when I increase the time periods from about 12 months (which takes arround 10 minutes) to arround 120 that is a realcase setting this is what happens.

I find that the processor gets about 100% busy, then I find the memory gradually increasing and gets over 90% ocupied or so. Simultaneously the processor usage drops to arround 2% of processor capacity and then it will takes hours and days to complete the job.

All this happens when GAMS is still generating the model. Maybe the processing slows down because of memory ocupation, and as a consequence takes an ieternity to complete.

I have 8Gbytes of memory and 8 processores and still have problems. I need to get over this problem so I can fully test my model.

Can anybody share some light of what might be happening and how I may resolve the situation. This is the print of the GAMS screen.

Thank You,

LeÃ£o Fernandes.

–
To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

\

To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

–
To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

\

To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

Archiver May 5, 2011, 1:08pm 5

Hello Arne Drud,

Thank you for your reply.

The line number is an excellent help which I did not know or overlooked. The constraint on this line will perhaps generate the most combinatory situations. Unfortunately it is not the only one as I see that the generation of each line takes long.

This constraint in particular is a mass balance equation which says that:

the initial stock at depot + product receival at depot = product delivery to other depots + product delivery to clients + final stock at depot.

The depot mass balance equation is enforced for all possible depots (18), products (7), entities (5), periods (120) and scenarios (1).

The receival quantity variables have two more dimensions transport mode (4) and plants (10) in addition to the above.

The delivery variables also have two additional dimensions transport mode (4) and clients (18) which is something like sum((k,m), QS(j,k,m,p,e,t,s))

So far I am only using 1 scenario, so the model will explode further, so I have to get it under control.

This means perhaps that I have to look at this statement and develop some cuts.

In 32 bit version of GAMS I used to get the out of memory error but had the same symptoms mentioned earlier.

In 64 bit version, after 24 hours, it has not completed generation of the model, so I interrupted it.

I agree that the memory is full and the machine goes slow so I cannot use it to write into a word document, and I see it is busy not writing but reading from disk, which I donot know what.

Can I assume from your reply, that I should not have been using so much memory and that it is normal for GAMS to start writing to disk from memory and then consequently the processor will drop down to 1% usage and 99% idle as the computer is more busy writing to disk than processing. That too during one full day.

I know that I must have a problem with my model whose memory usage could be reduced. However this performance of GAMS does not look to be normal even for a largescale problem, which frankly I do not know if it is.

However I would not expect the generation of the model to take so long. I would expect the execution to take long.

I would like to receive comments as to where and what I could start looking at. Sorry that I may be asking too much, but I am still quite new to GAMS and perhaps it is easier for an expert to make more sense from the information I am sending here.

Melhores Cumprimentos,

LeÃ£o Fernandes.

From: gamsworld@googlegroups.com [mailto:gamsworld@googlegroups.com] On Behalf Of Arne Stolbjerg Drud
Sent: quarta-feira, 4 de Maio de 2011 15:59
To: gamsworld@googlegroups.com
Subject: SV: Model takes a lot of time to generate

Hi LeÃ£o Fernandes

If you look at the line in the bottom left corner of the GAMS screen it tells you which line is being generated (1318) and how much memory you are using (9828 Mbytes). The amount of memory is larger than your physical machine so it starts to swap, and memory on disk is MUCH slower than Ram.

You must reduce the amount of memory you use. The McCarl-guide in the Help memnu has some comments of this â€“ just search for â€œmemoryâ€.

Regards

Arne

\

Arne Stolbjerg Drud

ARKI Consulting & Development A/S

Bagsvaerdvej 246A, DK-2880 Bagsvaerd, Denmark

Phone: (+45) 44 49 03 23, Fax: (+45) 44 49 03 33, email: adrud@arki.dk

Fra: gamsworld@googlegroups.com [mailto:gamsworld@googlegroups.com] PÃ¥ vegne af LeÃ£o Fernandes
Sendt: 4. maj 2011 16:02
Til: gamsworld@googlegroups.com
Emne: Model takes a lot of time to generate

Hello Everybody,

I have an interesting problem which I have been battling against for some time.

I am running a MIP model on a multiprocessor PC with 64 -bit windows 7 and Gams 22.3 and ILOG CPLEX 12.1 MIP solver.

I have a problem that has happened many times when I go from a small example to a largescale example.

So far I have survived by reducing the data set by reducing the number of periods so I can complete the run of the model, but when I increase the time periods from about 12 months (which takes arround 10 minutes) to arround 120 that is a realcase setting this is what happens.

I find that the processor gets about 100% busy, then I find the memory gradually increasing and gets over 90% ocupied or so. Simultaneously the processor usage drops to arround 2% of processor capacity and then it will takes hours and days to complete the job.

All this happens when GAMS is still generating the model. Maybe the processing slows down because of memory ocupation, and as a consequence takes an ieternity to complete.

I have 8Gbytes of memory and 8 processores and still have problems. I need to get over this problem so I can fully test my model.

Can anybody share some light of what might be happening and how I may resolve the situation. This is the print of the GAMS screen.

Thank You,

LeÃ£o Fernandes.

–
To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

–
To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

–
To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

Archiver May 5, 2011, 2:31pm 6

Hello Mohammad,

These are nice tricks but above all thank you for your advice. I thought I could improve the performance without making changes to the model.

I will follow your advice and get back once I have implemented and tested some improvements.

Any other recommendations and examples which you think may help, please feel free to send them across.

Melhores Cumprimentos,

LeÃ£o Fernandes.

CLC - Companhia LogÃstica de CombustÃveis, S.A.
Ãrea de Tecnologias e Sistemas de InformaÃ§Ã£o
Tel. +351 263 470 700
Fax: +351 263 470 790
email: leao.fernandes@clc.pt
visite-nos em www.clc.pt

From: gamsworld@googlegroups.com [mailto:gamsworld@googlegroups.com] On Behalf Of Tavallali
Sent: quinta-feira, 5 de Maio de 2011 11:56
To: gamsworld@googlegroups.com
Subject: Re: Model takes a lot of time to generate

Thank you for your reply.

You may like to consider pre-calculating the equations that are repeatedly used in the model, or defining a subset, instead of using combined conditions each time.

Also you may like to consider this :
Apart from the mathematics behind the solution algorithm, the model should be implemented in the most efficient way to reduce the execution time. As an example I would like to point to the storage syntax. Consider the following sample assignment:

dlambda_o_dp(l,ts,i) $(ord(i)=iPM) = f(l,ts,i); (a)

loop(i$(ord(i)=iPM),dlambda_o_dp(l,ts,i)= f(l,ts,i); ); (b)

I have tested the above two assignments and checked the execution time per iteration for both cases using OPTION PROFILE =3. Compare the outcome:

---- 1720 Assignment dlambda_o_dp 1.328 1.516 SECS 5 Mb 11250 (a)

---- 1721 Assignment dlambda_o_dp 0.016 0.188 SECS 5 Mb 11250 (b)

It is evident that the execution time of the second assignment is less than the first assignment ;

Apart from these small tricks, I strongly suggest that you try removing the unnecessary cases. Spending one day and thinking deeply and carefully on that can save you a lot of time!

I hope it will help.

–
Mohammad Sadegh Tavallali
National University of Singapore

2011/5/5 LeÃ£o Fernandes

Hello Tavallali,

Thank you for your reply. I have used this document and the recommendations on this model so I can say it is very useful. However I have not used all of the recommendations, especially the one â€œConsidering Unnecessary Casesâ€ which I think can make a big difference. The reason why I havenâ€™t used it so far is because I am afraid as the conditionals may interfere with relevant cases so one has to be extremely careful in filtering these unnecessary cases consistently in the various constraints.

I will take a new look at the document and try to implement conditionals on various constraints. But is there another way of improving performance for such cases while at the same time guaranteeing that relevant case will not be overlooked or filtered due to these conditionals?

Melhores Cumprimentos,

LeÃ£o Fernandes.

From: gamsworld@googlegroups.com [mailto:gamsworld@googlegroups.com] On Behalf Of Tavallali
Sent: quinta-feira, 5 de Maio de 2011 04:33
To: gamsworld@googlegroups.com
Subject: Re: Model takes a lot of time to generate

Dear LeÃ£o
Please refer to this link:
http://www.gams.com/mccarl/speed.pdf
it may be useful. Some of the tips there has worked for me.
Regards

Ù…ØÙ…Ø¯ ØµØ§Ø¯Ù‚ ØªÙˆÙ„Ù‘Ù„ÛŒ
Mohammad Sadegh Tavallali
National University of Singapore

2011/5/4 LeÃ£o Fernandes

Hello Everybody,

I have an interesting problem which I have been battling against for some time.

I am running a MIP model on a multiprocessor PC with 64 -bit windows 7 and Gams 22.3 and ILOG CPLEX 12.1 MIP solver.

I have a problem that has happened many times when I go from a small example to a largescale example.

So far I have survived by reducing the data set by reducing the number of periods so I can complete the run of the model, but when I increase the time periods from about 12 months (which takes arround 10 minutes) to arround 120 that is a realcase setting this is what happens.

I find that the processor gets about 100% busy, then I find the memory gradually increasing and gets over 90% ocupied or so. Simultaneously the processor usage drops to arround 2% of processor capacity and then it will takes hours and days to complete the job.

All this happens when GAMS is still generating the model. Maybe the processing slows down because of memory ocupation, and as a consequence takes an ieternity to complete.

I have 8Gbytes of memory and 8 processores and still have problems. I need to get over this problem so I can fully test my model.

Can anybody share some light of what might be happening and how I may resolve the situation. This is the print of the GAMS screen.

Thank You,

LeÃ£o Fernandes.

–
To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

\

To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

–
To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

\

To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

–
To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

Archiver May 5, 2011, 2:47pm 7

Hi Arne Drud,

Thank you for your assessment and comments. I now know that I have to invest in the model so I can keep the million dollars for me instead of giving it to the memory.

All the questions you are making are relevant and the reply is: Yes to all. And I understand the point you are making. Entities are major petroleum companies in a petroleum supply chain and the problem is a petroleum supply chain design problem.

I will take your advice and work small scale again to fine trim the model before getting back to the large scale data.

Thanks again.

Melhores Cumprimentos,

LeÃ£o Fernandes.

From: gamsworld@googlegroups.com [mailto:gamsworld@googlegroups.com] On Behalf Of Arne Stolbjerg Drud
Sent: quinta-feira, 5 de Maio de 2011 13:01
To: gamsworld@googlegroups.com
Subject: SV: Model takes a lot of time to generate

Hi LeÃ£o Fernandes

Try to use consider your own numbers. As you describe it you will have 18751201 = 75600 mass balance equations. On the receiving side you have 410 = 40 variables and on the delivery side 418 = 72 variables in each mass balance equation. Since the variables are all different you will have a total of 3,024,000 variables on the receiving side and 5,443,200 variables on the delivery side or a total of 8,467,200 variables. Before adding scenarios that will give another multiplicative factor. And I assume that this is just part of your model. This is not large scale â€“ this is EXTREME scale.

As others have commented, you will have to constrain your model by adding conditions. Typically you should look at questions like these:

     Does all depots handle all products?

     Is there a relationship between products and entities (I do not know what you mean)?

     Will all plants deliver to all depots?

     Will all depots deliver to all clients?

     Is there a relationship between transport mode and products?

```
     Etc etc.
```

You ask if you should not be using so much memory. My answer is that if you can get rid of unnecessary variables and equations then you will not use so much memory. But if you insist on have all combinations of everything in your model, then you (and GAMS) will need a million dollarsâ€™ worth of memory to generate the model. If it can be solved afterwards is then then another question.

Spend the days it takes to get your small-scale model in shape before trying a large case again.

Regards

Arne Stolbjerg Drud

\

Arne Stolbjerg Drud

ARKI Consulting & Development A/S

Bagsvaerdvej 246A, DK-2880 Bagsvaerd, Denmark

Phone: (+45) 44 49 03 23, Fax: (+45) 44 49 03 33, email: adrud@arki.dk

Fra: gamsworld@googlegroups.com [mailto:gamsworld@googlegroups.com] PÃ¥ vegne af LeÃ£o Fernandes
Sendt: 5. maj 2011 13:09
Til: gamsworld@googlegroups.com
Emne: RE: Model takes a lot of time to generate

Hello Arne Drud,

Thank you for your reply.

The line number is an excellent help which I did not know or overlooked. The constraint on this line will perhaps generate the most combinatory situations. Unfortunately it is not the only one as I see that the generation of each line takes long.

This constraint in particular is a mass balance equation which says that:

the initial stock at depot + product receival at depot = product delivery to other depots + product delivery to clients + final stock at depot.

The depot mass balance equation is enforced for all possible depots (18), products (7), entities (5), periods (120) and scenarios (1).

The receival quantity variables have two more dimensions transport mode (4) and plants (10) in addition to the above.

The delivery variables also have two additional dimensions transport mode (4) and clients (18) which is something like sum((k,m), QS(j,k,m,p,e,t,s))

So far I am only using 1 scenario, so the model will explode further, so I have to get it under control.

This means perhaps that I have to look at this statement and develop some cuts.

In 32 bit version of GAMS I used to get the out of memory error but had the same symptoms mentioned earlier.

In 64 bit version, after 24 hours, it has not completed generation of the model, so I interrupted it.

I agree that the memory is full and the machine goes slow so I cannot use it to write into a word document, and I see it is busy not writing but reading from disk, which I donot know what.

Can I assume from your reply, that I should not have been using so much memory and that it is normal for GAMS to start writing to disk from memory and then consequently the processor will drop down to 1% usage and 99% idle as the computer is more busy writing to disk than processing. That too during one full day.

I know that I must have a problem with my model whose memory usage could be reduced. However this performance of GAMS does not look to be normal even for a largescale problem, which frankly I do not know if it is.

However I would not expect the generation of the model to take so long. I would expect the execution to take long.

I would like to receive comments as to where and what I could start looking at. Sorry that I may be asking too much, but I am still quite new to GAMS and perhaps it is easier for an expert to make more sense from the information I am sending here.

Melhores Cumprimentos,

LeÃ£o Fernandes.

From: gamsworld@googlegroups.com [mailto:gamsworld@googlegroups.com] On Behalf Of Arne Stolbjerg Drud
Sent: quarta-feira, 4 de Maio de 2011 15:59
To: gamsworld@googlegroups.com
Subject: SV: Model takes a lot of time to generate

Hi LeÃ£o Fernandes

If you look at the line in the bottom left corner of the GAMS screen it tells you which line is being generated (1318) and how much memory you are using (9828 Mbytes). The amount of memory is larger than your physical machine so it starts to swap, and memory on disk is MUCH slower than Ram.

You must reduce the amount of memory you use. The McCarl-guide in the Help memnu has some comments of this â€“ just search for â€œmemoryâ€.

Regards

Arne

\

Arne Stolbjerg Drud

ARKI Consulting & Development A/S

Bagsvaerdvej 246A, DK-2880 Bagsvaerd, Denmark

Phone: (+45) 44 49 03 23, Fax: (+45) 44 49 03 33, email: adrud@arki.dk

Fra: gamsworld@googlegroups.com [mailto:gamsworld@googlegroups.com] PÃ¥ vegne af LeÃ£o Fernandes
Sendt: 4. maj 2011 16:02
Til: gamsworld@googlegroups.com
Emne: Model takes a lot of time to generate

Hello Everybody,

I have an interesting problem which I have been battling against for some time.

I am running a MIP model on a multiprocessor PC with 64 -bit windows 7 and Gams 22.3 and ILOG CPLEX 12.1 MIP solver.

I have a problem that has happened many times when I go from a small example to a largescale example.

So far I have survived by reducing the data set by reducing the number of periods so I can complete the run of the model, but when I increase the time periods from about 12 months (which takes arround 10 minutes) to arround 120 that is a realcase setting this is what happens.

I find that the processor gets about 100% busy, then I find the memory gradually increasing and gets over 90% ocupied or so. Simultaneously the processor usage drops to arround 2% of processor capacity and then it will takes hours and days to complete the job.

All this happens when GAMS is still generating the model. Maybe the processing slows down because of memory ocupation, and as a consequence takes an ieternity to complete.

I have 8Gbytes of memory and 8 processores and still have problems. I need to get over this problem so I can fully test my model.

Can anybody share some light of what might be happening and how I may resolve the situation. This is the print of the GAMS screen.

Thank You,

LeÃ£o Fernandes.

–
To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

–
To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

–
To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

–
To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

–
To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

Archiver May 8, 2011, 9:30am 8

Dear
Although your inquiry was related to model generation phase, I would like to remind you about the importance of well scaling a model. It significantly affects the solution time. Please refer to this link for more information :
http://www.bmx.gams.com/mccarl/mccarlhtml/index.html?simultaneous_equation_and_variable_scaling.htm
Regards

Ù…ØÙ…Ø¯ ØµØ§Ø¯Ù‚ ØªÙˆÙ„Ù‘Ù„ÛŒ
Mohammad Sadegh Tavallali
National University of Singapore

2011/5/5 LeÃ£o Fernandes

Hello Mohammad,

These are nice tricks but above all thank you for your advice. I thought I could improve the performance without making changes to the model.

I will follow your advice and get back once I have implemented and tested some improvements.

Any other recommendations and examples which you think may help, please feel free to send them across.

Melhores Cumprimentos,

LeÃ£o Fernandes.

CLC - Companhia LogÃstica de CombustÃveis, S.A.
Ãrea de Tecnologias e Sistemas de InformaÃ§Ã£o
Tel. +351 263 470 700
Fax: +351 263 470 790
email: leao.fernandes@clc.pt
visite-nos em www.clc.pt

From: gamsworld@googlegroups.com [mailto:gamsworld@googlegroups.com] On Behalf Of Tavallali
Sent: quinta-feira, 5 de Maio de 2011 11:56

To: gamsworld@googlegroups.com
Subject: Re: Model takes a lot of time to generate

Thank you for your reply.

You may like to consider pre-calculating the equations that are repeatedly used in the model, or defining a subset, instead of using combined conditions each time.

Also you may like to consider this :
Apart from the mathematics behind the solution algorithm, the model should be implemented in the most efficient way to reduce the execution time. As an example I would like to point to the storage syntax. Consider the following sample assignment:

dlambda_o_dp(l,ts,i) $(ord(i)=iPM) = f(l,ts,i); (a)

loop(i$(ord(i)=iPM),dlambda_o_dp(l,ts,i)= f(l,ts,i); ); (b)

I have tested the above two assignments and checked the execution time per iteration for both cases using OPTION PROFILE =3. Compare the outcome:

---- 1720 Assignment dlambda_o_dp 1.328 1.516 SECS 5 Mb 11250 (a)

---- 1721 Assignment dlambda_o_dp 0.016 0.188 SECS 5 Mb 11250 (b)

It is evident that the execution time of the second assignment is less than the first assignment ;

Apart from these small tricks, I strongly suggest that you try removing the unnecessary cases. Spending one day and thinking deeply and carefully on that can save you a lot of time!

I hope it will help.

–
Mohammad Sadegh Tavallali
National University of Singapore

2011/5/5 LeÃ£o Fernandes

Hello Tavallali,

Thank you for your reply. I have used this document and the recommendations on this model so I can say it is very useful. However I have not used all of the recommendations, especially the one â€œConsidering Unnecessary Casesâ€ which I think can make a big difference. The reason why I havenâ€™t used it so far is because I am afraid as the conditionals may interfere with relevant cases so one has to be extremely careful in filtering these unnecessary cases consistently in the various constraints.

I will take a new look at the document and try to implement conditionals on various constraints. But is there another way of improving performance for such cases while at the same time guaranteeing that relevant case will not be overlooked or filtered due to these conditionals?

Melhores Cumprimentos,

LeÃ£o Fernandes.

From: gamsworld@googlegroups.com [mailto:gamsworld@googlegroups.com] On Behalf Of Tavallali
Sent: quinta-feira, 5 de Maio de 2011 04:33
To: gamsworld@googlegroups.com
Subject: Re: Model takes a lot of time to generate

Dear LeÃ£o
Please refer to this link:
http://www.gams.com/mccarl/speed.pdf
it may be useful. Some of the tips there has worked for me.
Regards

Ù…ØÙ…Ø¯ ØµØ§Ø¯Ù‚ ØªÙˆÙ„Ù‘Ù„ÛŒ
Mohammad Sadegh Tavallali
National University of Singapore

2011/5/4 LeÃ£o Fernandes

Hello Everybody,

I have an interesting problem which I have been battling against for some time.

I am running a MIP model on a multiprocessor PC with 64 -bit windows 7 and Gams 22.3 and ILOG CPLEX 12.1 MIP solver.

I have a problem that has happened many times when I go from a small example to a largescale example.

So far I have survived by reducing the data set by reducing the number of periods so I can complete the run of the model, but when I increase the time periods from about 12 months (which takes arround 10 minutes) to arround 120 that is a realcase setting this is what happens.

I find that the processor gets about 100% busy, then I find the memory gradually increasing and gets over 90% ocupied or so. Simultaneously the processor usage drops to arround 2% of processor capacity and then it will takes hours and days to complete the job.

All this happens when GAMS is still generating the model. Maybe the processing slows down because of memory ocupation, and as a consequence takes an ieternity to complete.

I have 8Gbytes of memory and 8 processores and still have problems. I need to get over this problem so I can fully test my model.

Can anybody share some light of what might be happening and how I may resolve the situation. This is the print of the GAMS screen.

Thank You,

LeÃ£o Fernandes.

–
To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

\

To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

–
To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

\

To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

–
To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

–
To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

Archiver May 9, 2011, 11:24am 9

Dear Mohammad.

Thank you for this additional tip. I had read about it but not the impact on performance.

I will look into it more carefully.

Melhores Cumprimentos,

LeÃ£o Fernandes.

CLC - Companhia LogÃstica de CombustÃveis, S.A.
Ãrea de Tecnologias e Sistemas de InformaÃ§Ã£o
Tel. +351 263 470 700
Fax: +351 263 470 790
email: leao.fernandes@clc.pt
visite-nos em www.clc.pt

From: gamsworld@googlegroups.com [mailto:gamsworld@googlegroups.com] On Behalf Of Tavallali
Sent: domingo, 8 de Maio de 2011 08:31
To: gamsworld@googlegroups.com
Subject: Re: Model takes a lot of time to generate

Dear
Although your inquiry was related to model generation phase, I would like to remind you about the importance of well scaling a model. It significantly affects the solution time. Please refer to this link for more information :
http://www.bmx.gams.com/mccarl/mccarlhtml/index.html?simultaneous_equation_and_variable_scaling.htm
Regards

Ù…ØÙ…Ø¯ ØµØ§Ø¯Ù‚ ØªÙˆÙ„Ù‘Ù„ÛŒ
Mohammad Sadegh Tavallali
National University of Singapore

2011/5/5 LeÃ£o Fernandes

Hello Mohammad,

These are nice tricks but above all thank you for your advice. I thought I could improve the performance without making changes to the model.

I will follow your advice and get back once I have implemented and tested some improvements.

Any other recommendations and examples which you think may help, please feel free to send them across.

Melhores Cumprimentos,

LeÃ£o Fernandes.

CLC - Companhia LogÃstica de CombustÃveis, S.A.
Ãrea de Tecnologias e Sistemas de InformaÃ§Ã£o
Tel. +351 263 470 700
Fax: +351 263 470 790
email: leao.fernandes@clc.pt
visite-nos em www.clc.pt

From: gamsworld@googlegroups.com [mailto:gamsworld@googlegroups.com] On Behalf Of Tavallali
Sent: quinta-feira, 5 de Maio de 2011 11:56

To: gamsworld@googlegroups.com
Subject: Re: Model takes a lot of time to generate

Thank you for your reply.

You may like to consider pre-calculating the equations that are repeatedly used in the model, or defining a subset, instead of using combined conditions each time.

Also you may like to consider this :
Apart from the mathematics behind the solution algorithm, the model should be implemented in the most efficient way to reduce the execution time. As an example I would like to point to the storage syntax. Consider the following sample assignment:

dlambda_o_dp(l,ts,i) $(ord(i)=iPM) = f(l,ts,i); (a)

loop(i$(ord(i)=iPM),dlambda_o_dp(l,ts,i)= f(l,ts,i); ); (b)

I have tested the above two assignments and checked the execution time per iteration for both cases using OPTION PROFILE =3. Compare the outcome:

---- 1720 Assignment dlambda_o_dp 1.328 1.516 SECS 5 Mb 11250 (a)

---- 1721 Assignment dlambda_o_dp 0.016 0.188 SECS 5 Mb 11250 (b)

It is evident that the execution time of the second assignment is less than the first assignment ;

Apart from these small tricks, I strongly suggest that you try removing the unnecessary cases. Spending one day and thinking deeply and carefully on that can save you a lot of time!

I hope it will help.

–
Mohammad Sadegh Tavallali
National University of Singapore

2011/5/5 LeÃ£o Fernandes

Hello Tavallali,

Thank you for your reply. I have used this document and the recommendations on this model so I can say it is very useful. However I have not used all of the recommendations, especially the one â€œConsidering Unnecessary Casesâ€ which I think can make a big difference. The reason why I havenâ€™t used it so far is because I am afraid as the conditionals may interfere with relevant cases so one has to be extremely careful in filtering these unnecessary cases consistently in the various constraints.

I will take a new look at the document and try to implement conditionals on various constraints. But is there another way of improving performance for such cases while at the same time guaranteeing that relevant case will not be overlooked or filtered due to these conditionals?

Melhores Cumprimentos,

LeÃ£o Fernandes.

From: gamsworld@googlegroups.com [mailto:gamsworld@googlegroups.com] On Behalf Of Tavallali
Sent: quinta-feira, 5 de Maio de 2011 04:33
To: gamsworld@googlegroups.com
Subject: Re: Model takes a lot of time to generate

Dear LeÃ£o
Please refer to this link:
http://www.gams.com/mccarl/speed.pdf
it may be useful. Some of the tips there has worked for me.
Regards

Ù…ØÙ…Ø¯ ØµØ§Ø¯Ù‚ ØªÙˆÙ„Ù‘Ù„ÛŒ
Mohammad Sadegh Tavallali
National University of Singapore

2011/5/4 LeÃ£o Fernandes

Hello Everybody,

I have an interesting problem which I have been battling against for some time.

I am running a MIP model on a multiprocessor PC with 64 -bit windows 7 and Gams 22.3 and ILOG CPLEX 12.1 MIP solver.

I have a problem that has happened many times when I go from a small example to a largescale example.

So far I have survived by reducing the data set by reducing the number of periods so I can complete the run of the model, but when I increase the time periods from about 12 months (which takes arround 10 minutes) to arround 120 that is a realcase setting this is what happens.

I find that the processor gets about 100% busy, then I find the memory gradually increasing and gets over 90% ocupied or so. Simultaneously the processor usage drops to arround 2% of processor capacity and then it will takes hours and days to complete the job.

All this happens when GAMS is still generating the model. Maybe the processing slows down because of memory ocupation, and as a consequence takes an ieternity to complete.

I have 8Gbytes of memory and 8 processores and still have problems. I need to get over this problem so I can fully test my model.

Can anybody share some light of what might be happening and how I may resolve the situation. This is the print of the GAMS screen.

Thank You,

LeÃ£o Fernandes.

–
To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

\

To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

–
To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

\

To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

–
To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

–
To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.

–
To post to this group, send email to gamsworld@googlegroups.com.
To unsubscribe from this group, send email to gamsworld+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/gamsworld?hl=en.