To generate a custom format, you can add a custom serializer to the ABLDoc framework. You can use the ABLDoc model to get the file information and serialize it to the format you want. When output is generated, your custom serializer class is called and not the default HTML serializer.
To add a custom serializer from Progress Developer Studio for OpenEdge, go to DLC_HOME}\oeide\eclipse\plugins\com.progress.openedge.pdt.abldoc.ui, and edit abldoc.properties to add new properties as follows:
serializer: fully qualified name of the class
serializerpath: the absolute path of the folder where this class is located. If you generate the ABLDoc documentation, the custom serializer that is added, is called.