Custom csv feeder by from and to

Gatling (Open-Source)
1

Gatling version: 3.13.1 (must be up to date)
Gatling flavor: scala

Hi, back to the feature request I made here
Due to project big file (for real, which caused me a pain) csv, I did try to write some extra code, tho I’m a Scala Noobie, I managed to have it run as expectation as below:

package com.CustomCsvFeeder

import scala.io.Source
import io.gatling.core.feeder._
import io.gatling.core.Predef._

object CustomCsvFeeder {
  def apply(fileName: String, from: Int, to: Int): IndexedSeq[Map[String, String]] = {
    val source = Source.fromFile(fileName)
    try {
      val lines = source.getLines().toList
      if (lines.isEmpty) {
        throw new IllegalArgumentException(s"CSV file $fileName is empty")
      }

      val headers = lines.head.split(",").map(_.trim)
      val rows = lines.tail.slice(from - 1, to)

      val data: IndexedSeq[Map[String, String]] = rows.map { line =>
        val values = line.split(",").map(_.trim)
        headers.zip(values).toMap
      }.toIndexedSeq

      data
    } finally {
      source.close()
    }
  }
}

This code is really ugly, but it at least can satisfy my use case to have implement as:

val csvFill = csv("my/dir/to/file.csv)
                    .circular 
                    .from(rowNumber = 1)
                    .to(rowNumber = 1000)

Now I have a small problem, the user I got here, cannot be looped in forever group as:

  val feeder = CustomCsvFeeder
                .apply("myDataSource.csv").circular
                .circular
  val scn = scenario("Extract ID")
    .feed(feeder)
    .forever{
      exec(
        http("Get Callback Page")
          .get("/api/unknown/#{num}")
    )}
  setUp(
    scn.inject(constantUsersPerSec(2) during (5))
  ).protocols(httpProtocol).maxDuration(5 minutes)

This will have a result where the users are generated as expected, with data defined in the source csv, but when it hits 10 Users (which is 2 Users / sec in 5 seconds), the user stopped sending request.
Pardon if I tried to hack Gatling. Hopefully I can get some hint for this.

Edit: Tried with Original Gatling built-in of csv feeder, got the same behavior:

package demoScript

import io.gatling.core.Predef._
import io.gatling.http.Predef._
import com.CustomCsvFeeder._
import scala.concurrent.duration._

class demoScript extends Simulation {
  val feeder= csv("search.csv").circular()
  val scn = scenario("Extract ID")
    .feed(feeder)
    .forever{
      exec(
        http("Get Callback Page")
          .get("https://reqres.in/api/unknown/#{num}")
    )}
  setUp(
    scn.inject(constantUsersPerSec(1) during (10))
  ).maxDuration(1 minutes)
}
2

Hi @slandelle :smiling_face_with_tear: sorry for the ping, could you share me some thoughts if this implementation affect to the internal of Gatling ?

3

@trinp

Not sure what do you want to achieve, but I bet you want your feed in the forever loop.
Keep in mind that, by default, gatling will cache as a browser will.

Cheers!

4

ah, I will try to explain again.
As the ticket refer, I suggest that Gatling’s csv should have a built-in where I can declare a from and to to define set of data in a big csv file.

As my code above showed, I managed to have that function work as expectation (csv feeder now can get data from row to row)

Finally, I was trying to verify with a forever loop where I expect that user can repeatedly using the same data all over again, but then the forever loop does not keep sending request, it was hanging the scenario.

So my concern are:
1/ does my built in function potentially break anything in Gatling? As a matter of philosophy of Gatling team, I just need slight review as I’m a scala noob
2/ I want to know why the forever loop does not sending request after I reach duration point? you can see ConstantUserPerSec(2) during (5) , when reach 5 seconds, everything just hanging without anymore request

5

I was looking and think a little about your case.
First of all if you don’t know Scala… don’t use it.
Java is preferred for “beginners” :smirk:

As you can read in https://github.com/gatling/gatling/issues/4589 you can read all file in memory and filter it as you want :slight_smile:

Beware where you use feed() function.
To show interesting behavior and show one of possible solutions for you case (filter feeder), look below:

1 Like
6

Referring to the suggestion on LN, which says that my code does not solve the case in full manner when you have to choose N-records. Below are the magic of streams:

  • first 1000 records:
List<Map<String, Object>> foo = bar.stream().skip(0*1000).limit(1000).toList();
  • next 1000 records:
List<Map<String, Object>> foo = bar.stream().skip(1*1000).limit(1000).toList();
  • next 1000 records:
List<Map<String, Object>> foo = bar.stream().skip(2*1000).limit(1000).toList();
1 Like
7

This is an interesting way to pass the data, in Java manner.
I think I will try to tweak in Java following your suggestion, maybe this one is more cleaner than my attempts.
It’s just fun trying to tweak things, but still have to follow their philosophy.

8

marked @sbrevet as solutions,
Based on the response of the website, there is cache-control header which will not receive request from same user.
Overall, my little tweak works, so does @GeMi .
Appreciate for the discussion.