# Custom csv feeder by from and to

**URL:** <https://community.gatling.io/t/custom-csv-feeder-by-from-and-to/9431>\
**Category:** Gatling (Open-Source)\
**Created:** [December 2, 2024, 2:16am UTC](https://community.gatling.io/t/custom-csv-feeder-by-from-and-to/9431 "2024-12-02T02:16:27Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![trinp](https://dub1.discourse-cdn.com/flex013/user_avatar/community.gatling.io/trinp/32/1593_2.png) [@trinp](https://community.gatling.io/u/trinp)\
**Post date:** [December 2, 2024, 2:16am UTC](https://community.gatling.io/t/custom-csv-feeder-by-from-and-to/9431/1 "2024-12-02T02:16:27Z")

</div>

**Gatling version** : 3.13.1 (must be up to date)  
**Gatling flavor** : scala

Hi, back to the feature request I made [here](https://github.com/gatling/gatling/issues/4589)  
Due to project big file (for real, which caused me a pain) csv, I did try to write some extra code, tho I’m a Scala Noobie, I managed to have it run as expectation as below:

```auto
package com.CustomCsvFeeder

import scala.io.Source
import io.gatling.core.feeder._
import io.gatling.core.Predef._

object CustomCsvFeeder {
  def apply(fileName: String, from: Int, to: Int): IndexedSeq[Map[String, String]] = {
    val source = Source.fromFile(fileName)
    try {
      val lines = source.getLines().toList
      if (lines.isEmpty) {
        throw new IllegalArgumentException(s"CSV file $fileName is empty")
      }

      val headers = lines.head.split(",").map(_.trim)
      val rows = lines.tail.slice(from - 1, to)

      val data: IndexedSeq[Map[String, String]] = rows.map { line =>
        val values = line.split(",").map(_.trim)
        headers.zip(values).toMap
      }.toIndexedSeq

      data
    } finally {
      source.close()
    }
  }
}

```

This code is really ugly, but it at least can satisfy my use case to have implement as:

```auto
val csvFill = csv("my/dir/to/file.csv)
                    .circular 
                    .from(rowNumber = 1)
                    .to(rowNumber = 1000)

```

Now I have a small problem, the user I got here, cannot be looped in `forever` group as:

```auto
  val feeder = CustomCsvFeeder
                .apply("myDataSource.csv").circular
                .circular
  val scn = scenario("Extract ID")
    .feed(feeder)
    .forever{
      exec(
        http("Get Callback Page")
          .get("/api/unknown/#{num}")
    )}
  setUp(
    scn.inject(constantUsersPerSec(2) during (5))
  ).protocols(httpProtocol).maxDuration(5 minutes)

```

This will have a result where the users are generated as expected, with data defined in the source csv, but when it hits 10 Users (which is 2 Users / sec in 5 seconds), the user stopped sending request.  
Pardon if I tried to hack Gatling. Hopefully I can get some hint for this.

**Edit** : Tried with Original Gatling built-in of csv feeder, got the same behavior:

```auto
package demoScript

import io.gatling.core.Predef._
import io.gatling.http.Predef._
import com.CustomCsvFeeder._
import scala.concurrent.duration._

class demoScript extends Simulation {
  val feeder= csv("search.csv").circular()
  val scn = scenario("Extract ID")
    .feed(feeder)
    .forever{
      exec(
        http("Get Callback Page")
          .get("https://reqres.in/api/unknown/#{num}")
    )}
  setUp(
    scn.inject(constantUsersPerSec(1) during (10))
  ).maxDuration(1 minutes)
}

```

---

<div class="post-metadata">

**Author:** ![trinp](https://dub1.discourse-cdn.com/flex013/user_avatar/community.gatling.io/trinp/32/1593_2.png) [@trinp](https://community.gatling.io/u/trinp)\
**Post date:** [December 2, 2024, 2:32pm UTC](https://community.gatling.io/t/custom-csv-feeder-by-from-and-to/9431/2 "2024-12-02T14:32:04Z")

</div>

Hi @slandelle 🥲 sorry for the ping, could you share me some thoughts if this implementation affect to the internal of Gatling ?

---

<div class="post-metadata">

**Author:** ![sbrevet](https://dub1.discourse-cdn.com/flex013/user_avatar/community.gatling.io/sbrevet/32/830_2.png) [@sbrevet](https://community.gatling.io/u/sbrevet)\
**Post date:** [December 2, 2024, 3:45pm UTC](https://community.gatling.io/t/custom-csv-feeder-by-from-and-to/9431/3 "2024-12-02T15:45:27Z")

</div>

@trinp

Not sure what do you want to achieve, but I bet you want your `feed` in the `forever` loop.  
Keep in mind that, by default, gatling will cache as a browser will.

Cheers!

---

<div class="post-metadata">

**Author:** ![trinp](https://dub1.discourse-cdn.com/flex013/user_avatar/community.gatling.io/trinp/32/1593_2.png) [@trinp](https://community.gatling.io/u/trinp)\
**Post date:** [December 2, 2024, 4:01pm UTC](https://community.gatling.io/t/custom-csv-feeder-by-from-and-to/9431/4 "2024-12-02T16:01:09Z")

</div>

ah, I will try to explain again.  
As the ticket refer, I suggest that Gatling’s csv should have a built-in where I can declare a **from** and **to** to define set of data in a big csv file.

As my code above showed, I managed to have that function work as expectation (csv feeder now can get data from row to row)

Finally, I was trying to verify with a forever loop where I expect that user can repeatedly using the same data all over again, but then the forever loop does not keep sending request, it was hanging the scenario.

So my concern are:  
1/ does my built in function potentially break anything in Gatling? As a matter of philosophy of Gatling team, I just need slight review as I’m a scala noob  
2/ I want to know why the forever loop does not sending request after I reach duration point? you can see **ConstantUserPerSec(2) during (5)** , when reach 5 seconds, everything just hanging without anymore request

---

<div class="post-metadata">

**Author:** ![GeMi](https://dub1.discourse-cdn.com/flex013/user_avatar/community.gatling.io/gemi/32/939_2.png) [@GeMi](https://community.gatling.io/u/GeMi)\
**Post date:** [December 3, 2024, 8:58pm UTC](https://community.gatling.io/t/custom-csv-feeder-by-from-and-to/9431/5 "2024-12-03T20:58:19Z")

</div>

I was looking and think a little about your case.  
First of all if you don’t know Scala… don’t use it.  
Java is preferred for “beginners” 😏

As you can read in [https://github.com/gatling/gatling/issues/4589](https://github.com/gatling/gatling/issues/4589) you can read all file in memory and filter it as you want 🙂

Beware where you use **feed()** function.  
To show interesting behavior and show one of possible solutions for you case (filter feeder), look below:

> <https://github.com/gemiusz/gatling-examples-maven-java/blob/master/src/test/java/pl/gemiusz/Case0027FilterFeederAndForeverLoopSameRecordByVU.java>

---

<div class="post-metadata">

**Author:** ![GeMi](https://dub1.discourse-cdn.com/flex013/user_avatar/community.gatling.io/gemi/32/939_2.png) [@GeMi](https://community.gatling.io/u/GeMi)\
**Post date:** [December 4, 2024, 5:48pm UTC](https://community.gatling.io/t/custom-csv-feeder-by-from-and-to/9431/6 "2024-12-04T17:48:57Z")

</div>

Referring to the suggestion on LN, which says that my code does not solve the case in full manner when you have to choose N-records. Below are the magic of **streams** :

- first 1000 records:

```java
List<Map<String, Object>> foo = bar.stream().skip(0*1000).limit(1000).toList();

```

- next 1000 records:

```java
List<Map<String, Object>> foo = bar.stream().skip(1*1000).limit(1000).toList();

```

- next 1000 records:

```java
List<Map<String, Object>> foo = bar.stream().skip(2*1000).limit(1000).toList();

```

---

<div class="post-metadata">

**Author:** ![trinp](https://dub1.discourse-cdn.com/flex013/user_avatar/community.gatling.io/trinp/32/1593_2.png) [@trinp](https://community.gatling.io/u/trinp)\
**Post date:** [December 4, 2024, 9:07pm UTC](https://community.gatling.io/t/custom-csv-feeder-by-from-and-to/9431/7 "2024-12-04T21:07:22Z")

</div>

This is an interesting way to pass the data, in Java manner.  
I think I will try to tweak in Java following your suggestion, maybe this one is more cleaner than my attempts.  
It’s just fun trying to tweak things, but still have to follow their philosophy.

---

<div class="post-metadata">

**Author:** ![trinp](https://dub1.discourse-cdn.com/flex013/user_avatar/community.gatling.io/trinp/32/1593_2.png) [@trinp](https://community.gatling.io/u/trinp)\
**Post date:** [December 5, 2024, 12:01am UTC](https://community.gatling.io/t/custom-csv-feeder-by-from-and-to/9431/8 "2024-12-05T00:01:49Z")

</div>

marked @sbrevet as solutions,  
Based on the response of the website, there is `cache-control` header which will not receive request from same user.  
Overall, my little tweak works, so does @GeMi .  
Appreciate for the discussion.

---

<div class="post-metadata">

**Author:** ![system](https://europe1.discourse-cdn.com/flex013/uploads/gatling2/original/2X/b/b148e0742de35c6280726cabf3ecd3de2d320f63.png) [@system](https://community.gatling.io/u/system)\
**Post date:** [January 4, 2025, 12:02am UTC](https://community.gatling.io/t/custom-csv-feeder-by-from-and-to/9431/9 "2025-01-04T00:02:35Z")

</div>

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.