Grab all feeder records in 3.0.1

vu.nguyen · November 28, 2018, 9:56pm

Hi Stephane et team,

While trying out our simulator with Gatling 3.0.1, we found another unexpected change in behavior of our feeders using readRecords:

This is the definition of one of our feeders:

`

val simDataTokens: Vector[Map[String, String]] = {
try {
csv(file).readRecords.as[Vector[Map[String, String]]]
.zipWithIndex
.withFilter(r => {
r._1.contains(“name”) &&
r._1.contains(“token”)
})
.map(r => {
val id = r._2.toString
val tokens = r._1

// DEBUG
println(">>> id: " + id + ", tokens: " + tokens)

tokens + (“id” → id)
})
} catch {
case _: Exception => Vector.empty
}
}

// DEBUG
println(">>> simDataTokens records: " + simDataTokens.size)

`

With Gatling 2.3.1 (using csv(file).**records.**toVector instead), simDataTokens returned three expected records and size = 3 with following debugging info:

`

GATLING_HOME is set to /home/gatling/2.3.1

id: 0, tokens: Map(name → a, token → 1)
id: 1, tokens: Map(name → b, token → 2)
id: 2, tokens: Map(name → c, token → 3)
simDataTokens records: 3

10:49:44.179 [INFO ] i.g.h.a.HttpEngine - Start warm up
10:49:44.287 [INFO ] i.g.h.a.HttpEngine - Warm up done
Simulation com.illumio.agentSimulation started…

`

With Gatling 3.0.1, simDataTokens only returned the first record, and its size becomes 0. This is unexpected:

`

GATLING_HOME is set to /home/gatling/3.0.1

id: 0, tokens: Map(name → a, token → 1)
simDataTokens records: 0
Simulation com.illumio.agentSimulation started…

`

Thank you for your help along the way.

Vu

slandelle · November 28, 2018, 10:11pm

Nah, your code is broken:

.withFilter(r => {
r._1.contains(“name”) &&
r._1.contains(“token”)
})

How do you expect r._1 to both contain “name” and “token”?

vu.nguyen · November 28, 2018, 10:27pm

Because with zipWithIndex, r._1 is a Map[String, String] (which are the records from csv(file)).

These codes have worked for us for years using Gatling 2.3.1 and Gatling 2.2.0. They only failed when we ran them with Gatling 3.0.1.

(BTW, the debug outputs were generated earlier today).

Vu

slandelle · November 28, 2018, 10:37pm

My2cents: there’s an exception and you don’t see it because you silently trap it and return Vector.empty, which looks very fishy to me.
Have you checked the migration guide and how feeder file location resolution has changed (“data” folder is dropped)?

vu.nguyen · November 28, 2018, 11:39pm

Much appreciation for Your2cents, Stephane.

I repeated my experiment with following trimmed down feeder (with line number for reference):

`

200 val simDataTokens: Vector[Map[String, Any]] = {
201 csv(simTokens).readRecords.as[Vector[Map[String, String]]]
202 .zipWithIndex
203 .map(r => {
204 val id = r._2
205 val tokens = r._1
206
207 // DEBUG
208 println(">>> id: " + id + “, tokens: " + tokens)
209
210 tokens + (“id” → id)
211 })
212 }
213
214 println(”>>> simDataTokens records: " + simDataTokens.size)

`

And there was indeed an exception:

`

GATLING_HOME is set to /home/gatling/3.0.1

id: 0, tokens: Map(name → a, token → 1)
15:04:51.395 [ERROR] i.g.a.Gatling$ - Run crashed
java.lang.UnsupportedOperationException: null
at io.gatling.core.feeder.ArrayBasedMap.$plus(ArrayBasedMap.scala:28)
at io.gatling.core.feeder.ArrayBasedMap.$plus(ArrayBasedMap.scala:26)
at com.illumio.commonHeader$.$anonfun$simDataTokens$1(commonHeader.scala:210)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233)
at scala.collection.Iterator.foreach(Iterator.scala:937)
at scala.collection.Iterator.foreach$(Iterator.scala:937)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
at scala.collection.IterableLike.foreach(IterableLike.scala:70)
at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike.map(TraversableLike.scala:233)
at scala.collection.TraversableLike.map$(TraversableLike.scala:226)
at scala.collection.AbstractTraversable.map(Traversable.scala:104)
at com.illumio.commonHeader$.(commonHeader.scala:203)
at com.illumio.commonHeader$.(commonHeader.scala)
… 14 common frames omitted

`

It appears the Map addition operation (tokens + (“id” → id)) is not supported? Any suggestion if that’s the case?

github.com

gatling/gatling/blob/b7617bb05e895558da9c71c70df13c6db34d8951/gatling-core/src/main/scala/io/gatling/core/feeder/ArrayBasedMap.scala#L28


      
          
          import scala.collection.immutable.{ AbstractMap, Map }
          
          private[feeder] object ArrayBasedMap {
            def apply[K, V](keys: Array[K], values: Array[V]): ArrayBasedMap[K, V] =
              new ArrayBasedMap(keys, values, math.min(keys.length, values.length))
          }
          
          private[feeder] class ArrayBasedMap[K, +V](keys: Array[K], values: Array[V], override val size: Int) extends AbstractMap[K, V] with Map[K, V] with Serializable {
          
            override def +[V1 >: V](kv: (K, V1)) = throw new UnsupportedOperationException
          
            override def updated[V1 >: V](key: K, value: V1): Map[K, V1] = throw new UnsupportedOperationException
          
            override def get(key: K): Option[V] = {
              var i = 0
              while (i < size) {
                if (keys(i) == key) {
                  return Some(values(i))
                }
                i += 1

Thank you again, Stephane.

Vu

slandelle · November 29, 2018, 12:14am

Our ArrayBasedMap doesn’t currently implement +, - and updated operations, simply because we don’t need them internally in Gatling and use cases such as yours are pretty exotic.
Feel free to send a PR to implement them.

vu.nguyen · November 29, 2018, 4:12am

https://github.com/gatling/gatling/issues/3625

Thank you.

Topic		Replies	Views
Gatling 3.0.1.1 - how to make a CSV feeder read all data before starting the simulation? Gatling (Open-Source)	3	172	November 30, 2018
Inject as many users as contained in feeder Gatling (Open-Source)	3	151	March 10, 2015
custom feeder for maping records from csv to userId Gatling (Open-Source)	3	153	August 19, 2013
Iterating over all the records in a feeder & "syncing" up scenarios Gatling (Open-Source)	0	117	January 4, 2018
How to use Gatling for bulk operations Gatling (Open-Source)	4	140	December 5, 2019

Grab all feeder records in 3.0.1

Related topics