Seems the query service hardcodes the alertmanager URL to always be http://alertmanager:9093/api and not use what is supplied in the prometheus.yml file. Is this intended? Seems I need to use this other env var to get it to use the actual alertmanager URL?

Yes, you can use that env var. We will document it if it's not already done.

cc: @Ashu