Out of Memory Crash Attempting to Load Feeder for Query of 5.5 Million Records

Gatling (Open-Source)
1

See this error on our Jenkins job running Gatling 3.0.3. Have we really reached the Gatling memory limit, and if so, is it recommended to up that limit, or should we look into another approach to load the data such as sqlQueryFeeder.batch?

Query:

val sqlQueryFeeder = jdbcFeeder(
  dbUrl, userName, dbPass,
  "SELECT TOP 5500000

Job stacktrace:

`

Starting decryption
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid86972.hprof ...
Heap dump file created [1441802272 bytes in 15.384 secs]
15:23:46.511 [ERROR] i.g.a.Gatling$ - Run crashed
java.lang.OutOfMemoryError: Java heap space
        at scala.collection.immutable.Map$EmptyMap$.updated(Map.scala:106)
        at scala.collection.immutable.Map$EmptyMap$.$plus(Map.scala:107)
        at scala.collection.immutable.Map$EmptyMap$.$plus(Map.scala:99)
        at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:32)
        at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:28)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
        at scala.collection.TraversableLike$$Lambda$48/1795960102.apply(Unknown Source)
        at scala.collection.immutable.Range.foreach(Range.scala:158)
        at scala.collection.TraversableLike.map(TraversableLike.scala:237)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
        at scala.collection.AbstractTraversable.map(Traversable.scala:108)
        at io.gatling.jdbc.feeder.JdbcFeederSource$.computeRecord$1(JdbcFeederSource.scala:39)
        at io.gatling.jdbc.feeder.JdbcFeederSource$.loadRec$1(JdbcFeederSource.scala:44)
        at io.gatling.jdbc.feeder.JdbcFeederSource$.$anonfun$apply$1(JdbcFeederSource.scala:46)
        at io.gatling.jdbc.feeder.JdbcFeederSource$$$Lambda$258/544966217.apply(Unknown Source)
        at io.gatling.commons.util.Io$.withCloseable(Io.scala:125)
        at io.gatling.jdbc.feeder.JdbcFeederSource$.apply(JdbcFeederSource.scala:30)
        at io.gatling.jdbc.Predef$.jdbcFeeder(Predef.scala:26)
        at tests.operations.mass_action_get_tax_rates_load.<init>(mass_action_get_tax_rates_load.scala:15)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at io.gatling.app.Runner.run0(Runner.scala:75)
        at io.gatling.app.Runner.run(Runner.scala:61)
        at io.gatling.app.Gatling$.start(Gatling.scala:74)
        at io.gatling.app.Gatling$.fromArgs(Gatling.scala:47)
        at io.gatling.app.Gatling$.main(Gatling.scala:39)
        at io.gatling.app.Gatling.main(Gatling.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Wrapped by: java.lang.reflect.InvocationTargetException: null
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at io.gatling.app.Runner.run0(Runner.scala:75)
        at io.gatling.app.Runner.run(Runner.scala:61)
        at io.gatling.app.Gatling$.start(Gatling.scala:74)
        at io.gatling.app.Gatling$.fromArgs(Gatling.scala:47)
        at io.gatling.app.Gatling$.main(Gatling.scala:39)
        at io.gatling.app.Gatling.main(Gatling.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at io.gatling.mojo.MainWithArgsInFile.runMain(MainWithArgsInFile.java:50)
        at io.gatling.mojo.MainWithArgsInFile.main(MainWithArgsInFile.java:33)
Uncaught error from thread [GatlingSystem-scheduler-1]: Java heap space, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[GatlingSystem]
java.lang.OutOfMemoryError: Java heap space
        at akka.actor.LightArrayRevolverScheduler$$anon$3.nextTick(LightArrayRevolverScheduler.scala:269)
        at akka.actor.LightArrayRevolverScheduler$$anon$3.run(LightArrayRevolverScheduler.scala:235)
        at java.lang.Thread.run(Thread.java:748)
15:23:46.518 [ERROR] a.a.ActorSystemImpl - exception on LARS’ timer thread
java.lang.OutOfMemoryError: Java heap space
        at akka.actor.LightArrayRevolverScheduler$$anon$3.nextTick(LightArrayRevolverScheduler.scala:269)
        at akka.actor.LightArrayRevolverScheduler$$anon$3.run(LightArrayRevolverScheduler.scala:235)
        at java.lang.Thread.run(Thread.java:748)

`

2

Hi,

batch option is not supported on jdbc feeder. The method was available in the DSL for this feeder type, but that’s something I’ve just fixed in upcoming Gatling 3.2.0, see https://github.com/gatling/gatling/issues/3765.

Have you taken some time to check the maven plugin documentation so figure out how to increase heap size? https://gatling.io/docs/current/extensions/maven_plugin

3

Of course we didn’t(sorry). Upped the limit to 4gb, and now able to run test no problem.

4

:+1