Hi,
we have a setup where we use Gatling in rather short-running (less than 1 minute) cron-jobs to test the availability and responsiveness of our application. The test metrics are written to Prometheus using a Prometheus Graphite Exporter (https://github.com/prometheus/graphite_exporter), which is scraped every minute.
The problem we see is that the count metrics are mostly zero in Prometheus. My theory on why this happens is that the GraphiteDataWriter flushes its metrics every second and then clears them. Then in the next second, it will send a zero to the Graphite Exporter. The Exporter, when scraped, returns the latest value it has, which is zero unless the scrape happens within a second after the last meaningful value was written.
We tried to work around this by setting the writePeriod (which, incidentally, is called writeInterval in the docs) to 60s (equal to the scrape interval). This, however, leads to no data being written at all because the test runs for less than a minute, so the process finishes before the Graphite Writer writes anything.
Looking at the code of GraphiteDataWriter, I could not find a way to either avoid clearing the requestsByPath metrics or to make it flush when the simulation stops. Is there any way this can be done?
Thanks
Lars