Memory is required by GAMS for model compilation and generation and by the solver during the optimization. The command line option procTreeMemMonitor can be used to monitor the memory used by the GAMS process tree.
More details on the option are available in the documentation https://www.gams.com/latest/docs/UG_GamsCall.html#GAMSAOproctreememmonitor