Akka "No space left" errors while running Simulation (v 2.0.2)

I think my simulation might have triggered a race condition.

Everything worked great for about the first 250 sec. After that point, an error came up with text like this :

[ERROR] [11/25/2014 01:22:52.447] [GatlingSystem-akka.actor.default-dispatcher-6] [akka://GatlingSystem/user/$O] No space left on device
java.io.IOException: No space left on device
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:51)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
at io.gatling.core.result.writer.FileDataWriter.flush(FileDataWriter.scala:118)
at io.gatling.core.result.writer.FileDataWriter.push(FileDataWriter.scala:126)
at io.gatling.core.result.writer.FileDataWriter.onRequestMessage(FileDataWriter.scala:141)
at io.gatling.core.result.writer.DataWriter$$anonfun$initialized$1.applyOrElse(DataWriter.scala:103)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at io.gatling.core.akka.BaseActor.aroundReceive(BaseActor.scala:23)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

ERROR GatlingSystem-akka.actor.default-dispatcher-4 - Actor FileDataWriter crashed on message Some(RequestMessage(GetFooCount,1006934578893141165-89615,List(),GetFooCount,1416878572427,1416878572428,1416878572437,1416878572437,OK,None,List()))
java.io.IOException: No space left on device
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:51)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
at io.gatling.core.result.writer.FileDataWriter.flush(FileDataWriter.scala:118)
at io.gatling.core.result.writer.FileDataWriter.push(FileDataWriter.scala:126)
at io.gatling.core.result.writer.FileDataWriter.onRequestMessage(FileDataWriter.scala:141)
at io.gatling.core.result.writer.DataWriter$$anonfun$initialized$1.applyOrElse(DataWriter.scala:103)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at io.gatling.core.akka.BaseActor.aroundReceive(BaseActor.scala:23)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
tee: output.log: No space left on device
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

After these appeared, there were several lines like this:

ERROR GatlingSystem-akka.actor.default-dispatcher-4 - Can’t handle UserMessage(GetFooCount,1006934578893141165-89615,End,1416878572426,1416878572438) when in uninitialized state, discarding
ERROR GatlingSystem-akka.actor.default-dispatcher-4 - Can’t handle RequestMessage(RegisterOwner,1006934578893141165-3224,List(),RegisterOwner,1416878572437,1416878572439,1416878572446,1416878572446,OK,None,List()) when in uninitialized state, discarding
ERROR GatlingSystem-akka.actor.default-dispatcher-4 - Can’t handle UserMessage(RegisterOwner,1006934578893141165-3224,End,1416878572437,1416878572446) when in uninitialized state, discarding
ERROR GatlingSystem-akka.actor.default-dispatcher-4 - Can’t handle RequestMessage(GetObjectFoo,1006934578893141165-215015,List(),GetObjectCheers,1416878572437,1416878572439,1416878572446,1416878572446,OK,None,List()) when in uninitialized state, discarding

Then a bunch of errors like this:

[INFO] [11/25/2014 02:47:54.821] [GatlingSystem-akka.actor.default-dispatcher-2] [akka://GatlingSystem/user/$L] Message [io.gatling.core.result.writer.UserMessage] from Actor[akka://GatlingSystem/deadLetters] to Actor[akka://GatlingSystem/user/$L#1723891586] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [11/25/2014 02:47:54.822] [GatlingSystem-akka.actor.default-dispatcher-2] [akka://GatlingSystem/user/$L] Message [io.gatling.core.result.writer.RequestMessage] from Actor[akka://GatlingSystem/deadLetters] to Actor[akka://GatlingSystem/user/$L#1723891586] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
[INFO] [11/25/2014 02:47:54.822] [GatlingSystem-akka.actor.default-dispatcher-2] [akka://GatlingSystem/user/$L] Message [io.gatling.core.result.writer.UserMessage] from Actor[akka://GatlingSystem/deadLetters] to Actor[akka://GatlingSystem/user/$L#1723891586] was not delivered. [3] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.


Finally this:

No. Disk is full, can’t write simulation.log file.

Wow, ok.

This script was running on a Jenkins server and there were other jobs running periodically. I’m not sure if they were encountering similar problems.

In any case, the problem seemed to have disappeared the following day. I was able to run a Gatling script just fine