Batch vs queue: too few documentation and sample about the usage

General discussions
1

Hi, I have researched but I don’t get much clear idea bout the ff:
The reason why I’m researching is that I get OOM when only using readRecords which loads a 20M file.
So I read that I should used batch or queue to avoid this.
But I don’t get how these 2 lines differ from each other.
Should I put one inside a loop and each call returns the next batch or queue? Please enlighten me.

val sourceFile = separatedValues(“filename.csv”, ‘|’).queue.readRecords.toList

val sourceFile = separatedValues(“filename.csv”, ‘|’).batch.readRecords.toList

note that after the above line I have the below:
Iterator.continually({
…perform random selection
})

2

readRecords loads all the data in memory.

queue and batch only do a difference when using feed.