Trouble sending results to graphite

I’m a new gatling user and am attempting to send results to graphite. I’m using Gatling 2.0.0-RC2 running on an ubuntu box (virtual with only Gatling installed).

My Gatling.conf relevant section looks like this:

data {
writers = “console, file, graphite”
reader = file
console {
light = false
}
}
graphite {
host = “x.x.x.x”
port = 2003
light = false
protocol = “tcp”
rootPathPrefix = “gatling”
bucketWidth = 100
bufferSize = 8192
}

On my graphite box, I have:

Carbon.conf

[cache]
LOCAL_DATA_DIR = /opt/graphite/storage/whisper/
USER =
MAX_CACHE_SIZE = inf
MAX_UPDATES_PER_SECOND = 1000
MAX_CREATES_PER_MINUTE = inf
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2003
PICKLE_RECEIVER_INTERFACE = 127.0.0.1
PICKLE_RECEIVER_PORT = 2004
CACHE_QUERY_INTERFACE = 127.0.0.1
CACHE_QUERY_PORT = 7002
LOG_UPDATES = True

storage-schemas.conf

[stats]
pattern = ^stats…*
priority = 100
retentions = 10s:7d,1m:31d,10m:5y

[Gatling stats]
priority = 110
pattern = ^gatling…*
retentions = 1s:6d,10s:60d

[catchall]
pattern = ^.*
priority = 0
retentions = 60s:5y

storage-aggregation.conf

[sum]
pattern = .count$
xFilesFactor = 0
aggregationMethod = sum

[min]
pattern = .min$
xFilesFactor = 0.1
aggregationMethod = min

[max]
pattern = .max$
xFilesFactor = 0.1
aggregationMethod = max

[default_average]
pattern = .*
xFilesFactor = 0.3
aggregationMethod = average

On the gatling machine, I can issue the command (using the same user under which i run gatling.sh):

echo "gatling.myapp.datapoint2 15 date -d -${value}min +%s" | nc x.x.x.x 2003

This command executes and I can see the results in graphite

However, when i attempt to run gatling.sh and choose the computerdatabase.BasicSimulation simulation, I receive this error:

Simulation computerdatabase.BasicSimulation started…
[ERROR] [08/19/2014 15:31:39.616] [GatlingSystem-akka.actor.default-dispatcher-8] [akka://GatlingSy
stem/user/$y/$a] Connection refused
akka.actor.ActorInitializationException: exception during creation
at akka.actor.ActorInitializationException$.apply(Actor.scala:164)
at akka.actor.ActorCell.create(ActorCell.scala:596)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scal
a:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at java.net.Socket.(Socket.java:425)
at java.net.Socket.(Socket.java:208)
at io.gatling.metrics.sender.TcpSender.(TcpSender.scala:27)
at io.gatling.metrics.sender.MetricsSender$.newMetricsSender(MetricsSender.scala:23)
at io.gatling.metrics.GraphiteSender.preStart(GraphiteDataWriter.scala:99)
at akka.actor.Actor$class.aroundPreStart(Actor.scala:470)
at io.gatling.core.akka.BaseActor.aroundPreStart(BaseActor.scala:23)
at akka.actor.ActorCell.create(ActorCell.scala:580)
… 9 more

[INFO] [08/19/2014 15:31:39.635] [GatlingSystem-akka.actor.default-dispatcher-2] [akka://GatlingSys
tem/user/$y/$a] Message [io.gatling.metrics.SendMetrics] from Actor[akka://GatlingSystem/user/$y#13
20734734] to Actor[akka://GatlingSystem/user/$y/$a#1979925254] was not delivered. [1] dead letters
encountered. This logging can be turned off or adjusted with configuration settings ‘akka.log-dead-
letters’ and ‘akka.log-dead-letters-during-shutdown’.

The simulation runs fine, except for sending the metrics to Graphite. I’m assuming i have something not configured properly, any ideas?

Thanks,
Craig

Hi Craig,

If there’s no logging to show what ip/port is being refused you can use strace to attach to the process to output what is happening at the system call level ie. What exact connection refused.

Likelihood is your gatling conf not being picked up.
Have you double checked

http://gatling.io/docs/2.0.0-RC2/general/configuration.html?highlight=command#gatling-conf

?

I’m a new gatling user and am attempting to send results to graphite. I’m using Gatling 2.0.0-RC2 running on an ubuntu box (virtual with only Gatling installed).

My Gatling.conf relevant section looks like this:

data {
writers = “console, file, graphite”
reader = file
console {
light = false
}
}
graphite {
host = “x.x.x.x”
port = 2003
light = false
protocol = “tcp”
rootPathPrefix = “gatling”
bucketWidth = 100
bufferSize = 8192
}

On my graphite box, I have:

Carbon.conf

[cache]
LOCAL_DATA_DIR = /opt/graphite/storage/whisper/
USER =
MAX_CACHE_SIZE = inf
MAX_UPDATES_PER_SECOND = 1000
MAX_CREATES_PER_MINUTE = inf
LINE_RECEIVER_INTERFACE = 0.0.0.0
LINE_RECEIVER_PORT = 2003
PICKLE_RECEIVER_INTERFACE = 127.0.0.1
PICKLE_RECEIVER_PORT = 2004
CACHE_QUERY_INTERFACE = 127.0.0.1
CACHE_QUERY_PORT = 7002
LOG_UPDATES = True

storage-schemas.conf

[stats]
pattern = ^stats…*
priority = 100
retentions = 10s:7d,1m:31d,10m:5y

[Gatling stats]
priority = 110
pattern = ^gatling…*
retentions = 1s:6d,10s:60d

[catchall]
pattern = ^.*
priority = 0
retentions = 60s:5y

storage-aggregation.conf

[sum]
pattern = .count$
xFilesFactor = 0
aggregationMethod = sum

[min]
pattern = .min$
xFilesFactor = 0.1
aggregationMethod = min

[max]
pattern = .max$
xFilesFactor = 0.1
aggregationMethod = max

[default_average]
pattern = .*
xFilesFactor = 0.3
aggregationMethod = average

On the gatling machine, I can issue the command (using the same user under which i run gatling.sh):

echo "gatling.myapp.datapoint2 15 date -d -${value}min +%s" | nc x.x.x.x 2003

This command executes and I can see the results in graphite

However, when i attempt to run gatling.sh and choose the computerdatabase.BasicSimulation simulation, I receive this error:

Simulation computerdatabase.BasicSimulation started…
[ERROR] [08/19/2014 15:31:39.616] [GatlingSystem-akka.actor.default-dispatcher-8] [akka://GatlingSy
stem/user/$y/$a] Connection refused
akka.actor.ActorInitializationException: exception during creation
at akka.actor.ActorInitializationException$.apply(Actor.scala:164)
at akka.actor.ActorCell.create(ActorCell.scala:596)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:456)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scal
a:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at java.net.Socket.(Socket.java:425)
at java.net.Socket.(Socket.java:208)
at io.gatling.metrics.sender.TcpSender.(TcpSender.scala:27)
at io.gatling.metrics.sender.MetricsSender$.newMetricsSender(MetricsSender.scala:23)
at io.gatling.metrics.GraphiteSender.preStart(GraphiteDataWriter.scala:99)
at akka.actor.Actor$class.aroundPreStart(Actor.scala:470)
at io.gatling.core.akka.BaseActor.aroundPreStart(BaseActor.scala:23)
at akka.actor.ActorCell.create(ActorCell.scala:580)
… 9 more

[INFO] [08/19/2014 15:31:39.635] [GatlingSystem-akka.actor.default-dispatcher-2] [akka://GatlingSys
tem/user/$y/$a] Message [io.gatling.metrics.SendMetrics] from Actor[akka://GatlingSystem/user/$y#13
20734734] to Actor[akka://GatlingSystem/user/$y/$a#1979925254] was not delivered. [1] dead letters
encountered. This logging can be turned off or adjusted with configuration settings ‘akka.log-dead-
letters’ and ‘akka.log-dead-letters-during-shutdown’.

The simulation runs fine, except for sending the metrics to Graphite. I’m assuming i have something not configured properly, any ideas?

Thanks,
Craig

Hi Craig,

As you have to choose between UDP or TCP with Gatling’s Graphite writer, please check in carbon.conf that you’ve only enabled TCP and not UDP on port 2003.

Cheers,

Pierre

Thanks for the help. Turns out it was a configuration error. On the documentation page:

http://gatling.io/docs/2.0.0-RC2/graphite/index.html

It shows:

Configuration of Gatling

In $GATLING_HOME/conf/gatling.conf, be sure to :

  • have the GraphiteDataWriter in the list of writers
  • have the correct host and port for Graphite
data {
  writers = "console, file, graphite"
  reader = file
}
graphite {
        host = "192.168.56.101"
        port = 2003
        #light = false              # only send the all* stats
        #protocol = "tcp"           # Choose between 'tcp' or 'udp'
        #rootPathPrefix = "gatling"
        #bucketWidth = 100
        #bufferSize = 8192
      }

The graphite node needs to be contained within data, not separate. Therefore, I fixed my config file to look like this, and it now works:

data {
writers = “console, file, graphite”
reader = file
console {
light = false
}

graphite {

host = “x.x.x.x”

port = 2003

light = false

protocol = “tcp”

rootPathPrefix = “gatling”

bucketWidth = 100

bufferSize = 8192

}

}

–Craig

Created pull request to change documentation:

https://github.com/gatling/gatling/pull/2108

Thanks! I had the same problem.