Monika
Sun, 12 Mar 2023 21:07:24 +0000
Gatling (Open-Source)

Gatling Scala script causing heap memory issue with huge data processing.

Please help me with solution

I tried with gatling version 3.7.6 to 3.9.2 still issue.

I have to process 300000 data set in stretch ( condition I cannot batch or split the load)
Project ( gradlew kotlin - gatling -scala)

in all the steps I am getting after increasing the heap size to 4GB

“java.lang.OutOfMemoryError: Java heap space”

there are no memory leaks in the code, but still gatling could not process such huge data.

1. I am creating Json variable holding 300000 data set

2. I am providing that Json variable to request to process & provide a response

3. I am save a session variable to hold 300000 data set fetched from response

4. I am looping 300000 times ( single data processing API schema) modifying 300000 data individually

5. Saving each processed data in to list holding 300000 data.

6. Deleting 300000 data set passed to a bulk request.

Please find my code block

val  testCount = 300000
.
.
.
    .exec { session =>
      val testtypeids = Array("000000001", "000001", "000001101", "00000000001")
      val statusID = Array("000008", "0000058", "00000000-00019", "00000000-0009")
      val PerfIDs = session("tperfId").as[Vector[String]]
      val testData = (1 to testCount)
        .map { r =>
          Json.toJson(Map(
            "name" -> Json.toJson(s"Perftest${Random.alphanumeric.take(6).mkString}"),
            "displayName" -> Json.toJson(s"Perftest${Random.alphanumeric.take(6).mkString}"),
            "PerfIDs" -> Json.toJson(s"${PerfIDs(r % PerfIDs.length)}"),
            "testtypeids" -> Json.toJson(s"${Random.shuffle(testtypeids.toList).head}"),
            "statusId" -> Json.toJson(s"${Random.shuffle(statusID.toList).head}"),
            "excludedFromAutoHyperlinking" -> Json.toJson(true)
          ))
        }
      val datatestJson = Json.toJson(testData)
      session.set("datatestJson", datatestJson)
    }

    .exec(http("Add json data holding 300000 data set")
      .post("test/bulk")
      .body(StringBody(session => s"""${session("datatestJson").as[String]}""")).asJson
      .requestTimeout(2.hours)
      .check(jsonPath("$[*].id").findAll.saveAs("testIDs"))
      .check(status.is(201)))
    .pause(2.seconds)

    .foreach("#{testIDs}", "TestID") {
      exec(http("update 300000 data individually")
        .post("updating/testdata")
        .body(StringBody(session =>
          s"""{
             |  "id": "",
             |  "additionalTypeId": "000000001",
             |  "legs": [
             |    {
             |      "additionalTypeId2": "00000000112311",
             |      "testID": "${session("TestID").as[String]}"
             |    }
             |    ]
             |  }""".stripMargin)).asJson
        .check(jsonPath("$.id").saveAs("newtestID"))
        .check(status.is(201)))
    }
    .pause(2.seconds)

    .exec(http("delete 300000 data set")
      .delete("/tests/bulk")
      .header("Content-Type", "application/json")
      .body(StringBody(session => s"""[ "${session("newtestID").as[Vector[String]].mkString("\",\"")}" ]"""))
      .requestTimeout(2.hours)
      .check(status.is(204)))
    .pause(2.seconds)

Please help me with solutions without compromising on 300000 data set request & value saving from response in stretch.

8 posts - 5 participants