For example, if I have an algorithm in matlab, how can I use it to solve a GAMS model?

In general, it is not possible to hook a solver coded in Matlab to GAMS in the same way you’d connect a solver implemented in C or C++.

Depending on the model type, you could dump the model instance in one of several generic formats (e.g. MPS, LP-file) using GAMS/CONVERT or some other scheme (many exist) and feed that you your Matlab solver. In fact, I recall seeing a post very recently by one of my colleagues (Bussieck, maybe) that used embedded Python to automate the steps of dumping the model, calling an external solver, and importing the solution back into GAMS. That isn’t quite the same as a true GAMS link, but it has some utility.

-Steve