Batch loads all rows in a file?

General discussions
1

Hi all,
I think I’m not understanding this. I’m new to gatling.
Why do this line on my FeederFactory.feeder loads all data from file to the memory when I set it to 10?
I checked the length of sourceFile and confirmed that it did load all the rows in one call.

val sourceFile = separatedValues(“file.csv”, ‘|’).batch(10).readRecords.toList

thanks.

2

batch and readRecords are 2 unrelated things that are not compatible with each other.

Please read the documentation.

A feeder is basically an Iterator.
batch defines how the read from the underlying source.
readRecords reads all the Iterator and loads all it’s content in memory.

No idea what you’re trying to achieve.

3

Hi @slandelle ,
What I have is a sample below. This is called by scenarios scala code.


object FeederFactory {
  val some_data: Iterator[Record[Any]] = feeder("thefilename")
  
def feeder(key: String): Iterator[Record[Any]] = {
    //this line causes OOM when it loads a 20MB of csv to memory and I need to find way to fix it. 
    //What I did was changed it to batch(10).readRecords.toList but upon testing, 
    //it loads all the records in the csv. 
    //so I'm kinda confused on what ways I can prevent the OOM from happening.
    val sourceFile = separatedValues(key, '|').readRecords.toList 

    val fileSmall = sourceFile.filter(_.exists(y=>{y.toString().toLowerCase().contains("string_filter")}))

    Iterator.continually({
      fileSmall(ThreadLocalRandom.current.nextInt(fileSmall.length)).dropRight(1)
    })
  }
}
4

Hi,

Could you please explain your use case, with words?

5

Yes, the def feeder above is causing “out of memory heap space” as it loads 20MB of CSV using readRecords.toList.
I have to change that line to avoid that mem issue. So my understanding is I need to load data into memory by chunks. Say batch(10) then append this 10 in another var. When theres no more records returned by batch, I can finally do some processing like filter and return a row with Iterator.continually.

Hope it makes sense :frowning:

6

Sorry but this is not a use case.
This is describing with words the code you’ve already provided.

What are you trying to achieve with this piece of code?
Why are you not using a built-in csv feeder?