Method to parse and build the HTTP request from the cURL command using CSV feeder

siddesh · August 27, 2024, 7:43am

Hello team i am facing issue in gatling opensource and tried implement the curl command scripts in gatling.

My requirement is :

read the curl input from CSV
Split the Data into method, url, headers, data
Then as per the csv input my script should execute.

Note : when i am hard codded my curl command its working fine but when i am reading the data from csv session is not sending the request so could you please help us.

With CSV in put simulation code :

import io.gatling.core.Predef._
import io.gatling.http.Predef._
import io.gatling.http.request.builder.HttpRequestBuilder

class Curl1Simulation extends Simulation {

  // Define the HTTP protocol configuration
  val httpProtocol = http
    .baseUrl("https://example.com/api/resource")
    .inferHtmlResources()

  // Function to parse the cURL command
  def parseCurlCommand(curl: String): (String, String, Map[String, String], String) = {
    val methodPattern = "-X (\\w+)".r
    val urlPattern = "((https?|ftp)://[^\\s]+)".r
    val headerPattern = "-H '([^']+)'".r
    val dataPattern = "-d '([^']+)'".r

    val method = methodPattern.findFirstMatchIn(curl).map(_.group(1).toUpperCase).getOrElse("GET")
    val url = urlPattern.findFirstIn(curl).getOrElse("")
    val headers = headerPattern.findAllMatchIn(curl).map { m =>
      val header = m.group(1)
      val Array(key, value) = header.split(": ", 2)
      key -> value.stripPrefix("\"").stripSuffix("\"")
    }.toMap
    val data = dataPattern.findFirstMatchIn(curl).map(_.group(1)).getOrElse("")

    (method, url, headers, data)
  }

  // Function to create the HTTP request outside of the session
  def createHttpRequest(curlCommand: String): HttpRequestBuilder = {
    val (method, url, headers, body) = parseCurlCommand(curlCommand)
    println(s"Method: $method, URL: $url, Headers: $headers, Body: $body")

    method match {
      case "GET" => http("GET Request").get(url).headers(headers)
      case "POST" => http("POST Request").post(url).headers(headers).body(StringBody(body)).asJson
      case "PUT" => http("PUT Request").put(url).headers(headers).body(StringBody(body)).asJson
      case "DELETE" => http("DELETE Request").delete(url).headers(headers)
      case _ => throw new IllegalArgumentException(s"Unsupported method: $method")
    }
  }

  // Load the CSV file containing the cURL commands
  val csvFeeder = csv("data/ServiceCurl.csv").circular

  // Define the scenario
  val scn = scenario("Curl Scenario")
    .feed(csvFeeder) // Feeder to load curlCommand from CSV
    .exec ({ session =>
      val curlCommand = session("curlCommand").as[String]
      val httpRequest = createHttpRequest(curlCommand)
      session.set("httpRequest", httpRequest)
    })


  // Set up the simulation
  setUp(
    scn.inject(atOnceUsers(1)) // Adjust the number of users and injection profile as needed
  ).protocols(httpProtocol)
}

siddesh · August 27, 2024, 7:48am

import config.GatlingConfig.market
import io.gatling.core.Predef._
import io.gatling.http.Predef._

import scala.concurrent.duration._

class ServiceCurlSimulation extends Simulation {

  // Get the cURL command from system properties
  val curlCommand: String = System.getProperty("CURL_COMMAND", "curl -X GET https://example.com/api/resource -H "Authorization: Bearer token"'")

  // Function to parse the cURL command
  def parseCurlCommand(curl: String): (String, String, Map[String, String], String) = {
    val methodPattern = "-X (\\w+)".r
    val urlPattern = "((https?|ftp)://[^\\s]+)".r
    val headerPattern = "-H '([^']+)'".r
    val dataPattern = "-d '([^']+)'".r

    // Extract HTTP method
    val method = methodPattern.findFirstMatchIn(curl).map(_.group(1).toUpperCase).getOrElse("GET")

    // Extract URL
    val url = urlPattern.findFirstIn(curl).getOrElse("")

    // Extract headers
    val headers = headerPattern.findAllMatchIn(curl).map { m =>
      val header = m.group(1)
      val Array(key, value) = header.split(": ", 2)
      key -> value.stripPrefix("\"").stripSuffix("\"")
    }.toMap

    // Extract body data
    val data = dataPattern.findFirstMatchIn(curl).map(_.group(1)).getOrElse("")

    (method, url, headers, data)
  }

  // Parse the cURL command
  val (method, url, headers, body) = parseCurlCommand(curlCommand)

  // Configure the HTTP protocol
  val httpProtocol = http
    .baseUrl(url.split("/").take(3).mkString("/")) // Extract base URL
    .inferHtmlResources()

  // Define the HTTP request based on the method
  val httpRequest = method match {
    case "GET"    => http("Request").get(url).headers(headers)
    case "POST"   => http("Request").post(url).headers(headers).body(StringBody(body)).asJson
    case "PUT"    => http("Request").put(url).headers(headers).body(StringBody(body)).asJson
    case "DELETE" => http("Request").delete(url).headers(headers)
    case _        => throw new IllegalArgumentException(s"Unsupported method: $method")
  }

  // Define the scenario
  val scn = scenario("Curl Scenario")
    .exec(httpRequest.check(status.is(200)))

  // Set up the simulation
  setUp(
    scn.inject(atOnceUsers(1))
  ).protocols(httpProtocol)
}

Hard coded curl command scripts and its working as expected but we have to read the data from csv and then execute the scripts that changes is required.

trinp · August 28, 2024, 12:36am

If you have a csv file that contains data like this:

column_name
curl1
curl2
...
curln

You can first feed your scenario or request a csv feeder then you can call that curl from user session data by using #{column_name}

siddesh · August 28, 2024, 7:52am

That is not the right solution

siddesh · August 28, 2024, 7:54am

Then how do i pass here.

def createHttpRequest(curlCommand: String): HttpRequestBuilder = {
val (method, url, headers, body) = parseCurlCommand(curlCommand)
println(s"Method: $method, URL: $url, Headers: $headers, Body: $body")

method match {
  case "GET" => http("GET Request").get(url).headers(headers)
  case "POST" => http("POST Request").post(url).headers(headers).body(StringBody(body)).asJson
  case "PUT" => http("PUT Request").put(url).headers(headers).body(StringBody(body)).asJson
  case "DELETE" => http("DELETE Request").delete(url).headers(headers)
  case _ => throw new IllegalArgumentException(s"Unsupported method: $method")
}

}

trinp · August 29, 2024, 1:40am

I don’t actually understand your logic to support on this, perhaps you can give me the script where you hardcoded the curl ?
Also, for methods that are not impacted directly and verified worked on hardcoded value, please leave the name function only so that people can have better view and understanding what you are trying to do.

siddesh · August 29, 2024, 5:39am

Trinp for your reference hardcoded curl code on below.

import config.GatlingConfig.market
import io.gatling.core.Predef._
import io.gatling.http.Predef._

import scala.concurrent.duration._

class ServiceCurlSimulation extends Simulation {

  // Get the cURL command from system properties
  val curlCommand: String = System.getProperty("CURL_COMMAND", "curl -X GET https://example.com/api/resource -H "Authorization: Bearer token"'")

  // Function to parse the cURL command
  def parseCurlCommand(curl: String): (String, String, Map[String, String], String) = {
    val methodPattern = "-X (\\w+)".r
    val urlPattern = "((https?|ftp)://[^\\s]+)".r
    val headerPattern = "-H '([^']+)'".r
    val dataPattern = "-d '([^']+)'".r

    // Extract HTTP method
    val method = methodPattern.findFirstMatchIn(curl).map(_.group(1).toUpperCase).getOrElse("GET")

    // Extract URL
    val url = urlPattern.findFirstIn(curl).getOrElse("")

    // Extract headers
    val headers = headerPattern.findAllMatchIn(curl).map { m =>
      val header = m.group(1)
      val Array(key, value) = header.split(": ", 2)
      key -> value.stripPrefix("\"").stripSuffix("\"")
    }.toMap

    // Extract body data
    val data = dataPattern.findFirstMatchIn(curl).map(_.group(1)).getOrElse("")

    (method, url, headers, data)
  }

  // Parse the cURL command
  val (method, url, headers, body) = parseCurlCommand(curlCommand)

  // Configure the HTTP protocol
  val httpProtocol = http
    .baseUrl(url.split("/").take(3).mkString("/")) // Extract base URL
    .inferHtmlResources()

  // Define the HTTP request based on the method
  val httpRequest = method match {
    case "GET"    => http("Request").get(url).headers(headers)
    case "POST"   => http("Request").post(url).headers(headers).body(StringBody(body)).asJson
    case "PUT"    => http("Request").put(url).headers(headers).body(StringBody(body)).asJson
    case "DELETE" => http("Request").delete(url).headers(headers)
    case _        => throw new IllegalArgumentException(s"Unsupported method: $method")
  }

  // Define the scenario
  val scn = scenario("Curl Scenario")
    .exec(httpRequest.check(status.is(200)))

  // Set up the simulation
  setUp(
    scn.inject(atOnceUsers(1))
  ).protocols(httpProtocol)
}

trinp · August 29, 2024, 6:06am

based on your code, I think that you did not actually used my recommendation, at least in the csv feeder
Pseudo-code you may want to see:

class ServiceCurlSimulation extends Simulation {

  //I removed all method code for easier view
  val scn = scenario("Curl Scenario")
    .feed(curlCSV) //your CSV feed should be here
    .exec(session => {
          val string = session("ColumnName").as[String]  //you need to take curl value here then do your modification on it.
         //your logic method etc. etc.
    })
    .exec(httpRequest.check(status.is(200)))

  // Set up the simulation
  setUp(
    scn.inject(atOnceUsers(1))
  ).protocols(httpProtocol)
}

For session attribute, please see here

Explaination: When you feed your csv, Virtual User session attribute will initiate a val which it is as:

ColumnName (user session attribute name)
curl1 (this will be in user 1 session)
curl2 (user 2 session)
...
curln (user n...)

siddesh · August 29, 2024, 7:25am

This i have already tried and how it will pass the session value to in the below curlCommand.

// Parse the cURL command
val (method, url, headers, body) = parseCurlCommand(curlCommand)

trinp · August 29, 2024, 7:40am

    .exec(session => {
          //your class methods...
          val (method, url, headers, body) = parseCurlCommand(session("curlCommand"))
    })

siddesh · August 29, 2024, 7:45am

If i use your method how it will parse the url and headers deatils to this method.

// Define the HTTP request based on the method
val httpRequest = method match {
case “GET” => http(“Request”).get(url).headers(headers)
case “POST” => http(“Request”).post(url).headers(headers).body(StringBody(body)).asJson
case “PUT” => http(“Request”).put(url).headers(headers).body(StringBody(body)).asJson
case “DELETE” => http(“Request”).delete(url).headers(headers)
case _ => throw new IllegalArgumentException(s"Unsupported method: $method")
}

trinp · August 29, 2024, 7:47am

that is what you should think for, as I saw above you had handled everything by yourself, using session of Gatling will put all method there to process your data.

siddesh · August 29, 2024, 8:06am

which ever possibilities you said right i have already tried but its not send any request and seeing below error.

Generating reports…
Exception in thread “main” java.lang.UnsupportedOperationException: There were no requests sent during the simulation,

Please refer the below code for your ref. and please run the scripts in your local machine.

import config.DataConfig.serviceNamesData
import io.gatling.core.Predef._
import io.gatling.http.Predef._

import scala.concurrent.duration._

class Curl2Simulation extends Simulation {

  // Function to parse the cURL command
  def parseCurlCommand(curl: String): (String, String, Map[String, String], String) = {
    val methodPattern = "-X (\\w+)".r
    val urlPattern = "((https?|ftp)://[^\\s]+)".r
    val headerPattern = "-H '([^']+)'".r
    val dataPattern = "-d '([^']+)'".r

    // Extract HTTP method
    val method = methodPattern.findFirstMatchIn(curl).map(_.group(1).toUpperCase).getOrElse("GET")

    // Extract URL
    val url = urlPattern.findFirstIn(curl).getOrElse("")

    // Extract headers
    val headers = headerPattern.findAllMatchIn(curl).map { m =>
      val header = m.group(1)
      val Array(key, value) = header.split(": ", 2)
      key -> value.stripPrefix("\"").stripSuffix("\"")
    }.toMap

    // Extract body data
    val data = dataPattern.findFirstMatchIn(curl).map(_.group(1)).getOrElse("")

    (method, url, headers, data)
  }

  // Configure the HTTP protocol
  val httpProtocol = http
    .inferHtmlResources()

  // Define the scenario
  val scn = scenario("Curl Scenario")
    .feed(serviceNamesData) // Feeder to load curlCommand from CSV
    .exec(session => {
      // Get the cURL command from the session
      val curlCommand = session("curlCommand").as[String]

      // Parse the cURL command
      val (method, url, headers, body) = parseCurlCommand(curlCommand)

      // Store the parsed values in the session for further use
      session
        .set("method", method)
        .set("url", url)
        .set("headers", headers)
        .set("body", body)
    })
    .exec(session => {
      // Retrieve parsed values from the session
      val method = session("method").as[String]
      val url = session("url").as[String]
      val headers = session("headers").as[Map[String, String]]
      val body = session("body").as[String]

      // Build the HTTP request based on the parsed method
      val httpRequest = method match {
        case "GET"    => http("GET Request").get(url).headers(headers)
        case "POST"   => http("POST Request").post(url).headers(headers).body(StringBody(body)).asJson
        case "PUT"    => http("PUT Request").put(url).headers(headers).body(StringBody(body)).asJson
        case "DELETE" => http("DELETE Request").delete(url).headers(headers)
        case _        => throw new IllegalArgumentException(s"Unsupported method: $method")
      }

      // Execute the HTTP request and check the response status
      httpRequest.check(status.is(200))
      session
    })

  // Set up the simulation
  setUp(
    scn.inject(atOnceUsers(1)) // Adjust the number of users and injection profile as needed
  ).protocols(httpProtocol)
}

siddesh · August 29, 2024, 8:09am

what changes needs in above scripts let me know, it will great helpful

siddesh · August 29, 2024, 8:19am

CSV In put details.

curlCommand
curl -X GET https://example.com/api/resource -H “Authorization: Bearer token”
curl -X POST https://example.com/api/resource -H “Content-Type: application/json” -d ‘{“key”:“value”}’

slandelle · August 29, 2024, 9:20am

Honestly, what you’re asking for is consulting.

siddesh · August 29, 2024, 10:11am

Yes, technical solution i need.

trinp · August 30, 2024, 6:29am

why don’t you try to separate your curl in csv then use session parsing in ?

Column1, method, content
curl -X, POST, {...}
curl -X, PUT, {...},
curl -X, PATCH, {...}

This could be as far as I can to help you.

vu.nguyen · September 21, 2024, 12:35am

For building different HTTP requests, use doSwitch

HTTP requests must be passed to the exec() method to be attached to the scenario and executed.

doSwitch("#{method}")(
  "GET" -> exec(http("GET Request").get("#{url}").headers().check()),
  "POST" -> exec(http("POST Request").post("#{url}").headers().body().check()),
  ...
)

Topic		Replies	Views
Having httpMethod in the csv file Gatling (Open-Source)	2	117	June 19, 2019
Is there any way in Gatling to feed csv in Http protocol Gatling (Open-Source) gatling-academy	2	343	July 27, 2023
Help with Post Requests Gatling (Open-Source)	1	111	May 7, 2015
Feeder + java method Gatling (Open-Source)	0	197	February 10, 2017
Issue with using csv feed. Getting an error value feed is not a member of io.gatling.http.request.builder.Http Gatling (Open-Source)	4	166	February 20, 2018

Method to parse and build the HTTP request from the cURL command using CSV feeder

Related Topics