Issue with multipart form data upload in Gatling 2.0.0-M3

Vivek_R · January 6, 2014, 11:11pm

Hi,

I’m trying to stress test our File upload API using gatling and running into some issues. It was working with Gatling 1 properly with upload parameter, but as it’s been deprecated in Gatling 2, I’m having issues in making it work. The file’s not being uploaded properly and I’m getting EOF errors on the server. It used to work fine with Gatling 1, but I need inject API functionality which is available only in Gatling 2.

Here’s my simulation and the CSV contains productId, checkSum, fileSize and fileName

import io.gatling.core.Predef._
import io.gatling.http.Predef._
import io.gatling.jdbc.Predef._
import io.gatling.http.Headers.Names._
import io.gatling.http.Headers.Values._
import scala.concurrent.duration._
import io.gatling.core.Predef.bootstrap._
import io.gatling.core.Predef.assertions._

class PerformanceTest extends Simulation {
val performanceTest = csv(“uploadUrls.csv”).random
val scn = scenario(“Performance Test”)
.feed(performanceTest)
.exec(
http(“Performance”)
.post(“http://ul.dev.test.com:8080/uploader/api/v1/upload/${productId}/1/userPDF/upload”)
.header(“CheckSum”, “${checkSum}”)
.header(“FileSize”, “${fileSize}”)
.header(“UAUSERID”, “17”)
.header(“userId”, “17”)
.header(“EmailAddress”, “admin@test.com”)
.bodyPart(upload(“data”, “${fileName}”, “application/pdf”))
.check(status.is(200))
)
setUp(scn.inject(atOnce(10 users), nothingFor(10 seconds), constantRate(20 usersPerSec) during (2 minutes)))
}

Excilys · January 7, 2014, 7:07am

Hi,

Which version do you use? What is upload() (not a Gatling built-in)?

Vivek_R · January 7, 2014, 7:16am

Hi,

I currently use 2.0.0-M3. I tried using .bodyPart(RawFileBodyPart(“data”, “${fileName}”, “application/pdf”)) and that didn’t work either. upload was used without .bodyPart with 1.5.3 and I guess upload is deprecated in Gatling 2. Sorry for the confusion

Excilys · January 7, 2014, 8:28am

Look like a problem in either async-http-client or Netty.
Could you try latest Gatling snapshot, please?
https://oss.sonatype.org/content/repositories/snapshots/io/gatling/highcharts/gatling-charts-highcharts/2.0.0-SNAPSHOT/

Here’s your simulation migrated for this version:

import io.gatling.core.Predef._
import io.gatling.http.Predef._
import io.gatling.jdbc.Predef._
import scala.concurrent.duration._

class PerformanceTest extends Simulation {
val performanceTest = csv(“uploadUrls.csv”).random
val scn = scenario(“Performance Test”)
.feed(performanceTest)
.exec(
http(“Performance”)
.post(“http://ul.dev.test.com:8080/uploader/api/v1/upload/${productId}/1/userPDF/upload”)
.header(“CheckSum”, “${checkSum}”)
.header(“FileSize”, “${fileSize}”)
.header(“UAUSERID”, “17”)
.header(“userId”, “17”)
.header(“EmailAddress”, “ad…@test.com”)
.bodyPart(RawFileBodyPart(“data”, “${fileName}”).withContentType(“application/pdf”))
.check(status.is(200)))
setUp(scn.inject(atOnceUsers(10), nothingFor(10 seconds), constantUsersPerSec(20) during (2 minutes)))
}

Vivek_R · January 7, 2014, 7:27pm

I’m getting this error with the new snapshot and your simulation

value withContentType is not a member of io.gatling.http.request.FileBodyPart
possible cause: maybe a semicolon is missing before `value withContentType’?
11:19:21.147 [ERROR] i.g.a.ZincCompiler$ - .withContentType(“application/pdf”))
11:19:21.148 [ERROR] i.g.a.ZincCompiler$ - ^
11:19:21.156 [ERROR] i.g.a.ZincCompiler$ - one error found
Compilation failed
Heap

Should I be trying with some other function?

Vivek_R · January 7, 2014, 7:56pm

Also, I changed it to contentType instead of withContentType for the mime type and that seems to have gone to the next step, but more errors this time about the HTTP engine

11:35:59.409 [ERROR] i.g.h.a.HttpRequestAction - Action HttpRequestAction crashed on message Some(Session(Performance Test,8083881021089530740-8,Map(productId → prd56746-3456, checkSum → 56746, fileSize → 3456, fileName → file_3370.pdf),1389123359319,0,List(),List(OK),List(),List())), forwarding user to the next one
java.lang.UnsupportedOperationException: HTTP engine hasn’t been started
at io.gatling.http.ahc.HttpEngine$.instance(HttpEngine.scala:62) ~[gatling-http-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
at io.gatling.http.action.HttpRequestAction$.send$1(HttpRequestAction.scala:54) ~[gatling-http-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
at io.gatling.http.action.HttpRequestAction$.beginHttpTransaction(HttpRequestAction.scala:65) ~[gatling-http-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]

Excilys · January 7, 2014, 9:23pm

I changed the signature a few hours after I send previous email.
Use contentType() instead of withContentType(), please.

Excilys · January 7, 2014, 9:38pm

Was it the first exception you got or was it the consequence of something bad happening before?
Haven’t been able to reproduce for now, still digging.

Excilys · January 7, 2014, 9:43pm

Did you disable warmUp?

Vivek_R · January 7, 2014, 10:01pm

I haven’t disbaled warmUp and attached the exception in full for your reference

[ERROR] [01/07/2014 11:35:59.369] [GatlingSystem-akka.actor.default-dispatcher-6] [akka://GatlingSystem/user/$c] HTTP engine hasn’t been started
java.lang.UnsupportedOperationException: HTTP engine hasn’t been started
at io.gatling.http.ahc.HttpEngine$.instance(HttpEngine.scala:62)
at io.gatling.http.action.HttpRequestAction$.send$1(HttpRequestAction.scala:54)
at io.gatling.http.action.HttpRequestAction$.beginHttpTransaction(HttpRequestAction.scala:65)
at io.gatling.http.action.HttpRequestAction$$anonfun$executeOrFail$1$$anonfun$2.apply(HttpRequestAction.scala:109)
at io.gatling.http.action.HttpRequestAction$$anonfun$executeOrFail$1$$anonfun$2.apply(HttpRequestAction.scala:105)
at io.gatling.core.validation.Success.map(Validation.scala:29)
at io.gatling.http.action.HttpRequestAction$$anonfun$executeOrFail$1.apply(HttpRequestAction.scala:105)
at io.gatling.http.action.HttpRequestAction$$anonfun$executeOrFail$1.apply(HttpRequestAction.scala:102)
at io.gatling.core.validation.Success.flatMap(Validation.scala:30)
at io.gatling.http.action.HttpRequestAction.executeOrFail(HttpRequestAction.scala:102)
at io.gatling.core.action.Failable$class.execute(Actions.scala:70)
at io.gatling.http.action.HttpRequestAction.execute(HttpRequestAction.scala:95)
at io.gatling.core.action.Action$$anonfun$receive$1.applyOrElse(Actions.scala:30)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:166)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
at akka.actor.ActorCell.invoke(ActorCell.scala:456)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

11:35:59.377 [ERROR] i.g.h.a.HttpRequestAction - Action HttpRequestAction crashed on message Some(Session(Performance Test,8083881021089530740-0,Map(productId → prd56746-3456, checkSum → 56746, fileSize → 3456, fileName → file_4289.pdf),1389123359318,0,List(),List(OK),List(),List())), forwarding user to the next one
java.lang.UnsupportedOperationException: HTTP engine hasn’t been started
at io.gatling.http.ahc.HttpEngine$.instance(HttpEngine.scala:62) ~[gatling-http-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
at io.gatling.http.action.HttpRequestAction$.send$1(HttpRequestAction.scala:54) ~[gatling-http-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
at io.gatling.http.action.HttpRequestAction$.beginHttpTransaction(HttpRequestAction.scala:65) ~[gatling-http-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
at io.gatling.http.action.HttpRequestAction$$anonfun$executeOrFail$1$$anonfun$2.apply(HttpRequestAction.scala:109) ~[gatling-http-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
at io.gatling.http.action.HttpRequestAction$$anonfun$executeOrFail$1$$anonfun$2.apply(HttpRequestAction.scala:105) ~[gatling-http-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
at io.gatling.core.validation.Success.map(Validation.scala:29) ~[gatling-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
at io.gatling.http.action.HttpRequestAction$$anonfun$executeOrFail$1.apply(HttpRequestAction.scala:105) ~[gatling-http-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
at io.gatling.http.action.HttpRequestAction$$anonfun$executeOrFail$1.apply(HttpRequestAction.scala:102) ~[gatling-http-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
at io.gatling.core.validation.Success.flatMap(Validation.scala:30) ~[gatling-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
at io.gatling.http.action.HttpRequestAction.executeOrFail(HttpRequestAction.scala:102) ~[gatling-http-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
at io.gatling.core.action.Failable$class.execute(Actions.scala:70) ~[gatling-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
at io.gatling.http.action.HttpRequestAction.execute(HttpRequestAction.scala:95) ~[gatling-http-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
at io.gatling.core.action.Action$$anonfun$receive$1.applyOrElse(Actions.scala:30) ~[gatling-core-2.0.0-SNAPSHOT.jar:2.0.0-SNAPSHOT]
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:166) ~[scala-library-2.10.3.jar:na]
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498) ~[akka-actor_2.10-2.2.3.jar:2.2.3]
at akka.actor.ActorCell.invoke(ActorCell.scala:456) ~[akka-actor_2.10-2.2.3.jar:2.2.3]
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237) ~[akka-actor_2.10-2.2.3.jar:2.2.3]
at akka.dispatch.Mailbox.run(Mailbox.scala:219) ~[akka-actor_2.10-2.2.3.jar:2.2.3]
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386) ~[akka-actor_2.10-2.2.3.jar:2.2.3]
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) ~[scala-library-2.10.3.jar:na]
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) ~[scala-library-2.10.3.jar:na]
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) ~[scala-library-2.10.3.jar:na]
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ~[scala-library-2.10.3.jar:na]

Excilys · January 7, 2014, 10:25pm

Found it!

This happens when no HTTP protocol has been set up and the default one is used.
Let me fix this.

Thanks for reporting!

Excilys · January 7, 2014, 10:42pm

Vivek,

Until I get this other issue fixed, could you try this one, please?

import io.gatling.core.Predef._
import io.gatling.http.Predef._
import io.gatling.jdbc.Predef._
import scala.concurrent.duration._

class PerformanceTest extends Simulation {

val httpProtocol = http
.baseURL(“http://ul.dev.test.com:8080”)

val performanceTest = csv(“uploadUrls.csv”).random
val scn = scenario(“Performance Test”)
.feed(performanceTest)
.exec(
http(“Performance”)
.post("/uploader/api/v1/upload/${productId}/1/userPDF/upload")
.header(“CheckSum”, “${checkSum}”)
.header(“FileSize”, “${fileSize}”)
.header(“UAUSERID”, “17”)
.header(“userId”, “17”)
.header(“EmailAddress”, “ad…@test.com”)
.bodyPart(RawFileBodyPart(“data”, “${fileName}”).contentType(“application/pdf”))
.check(status.is(200)))
setUp(scn.inject(atOnceUsers(10), nothingFor(10 seconds), constantUsersPerSec(20) during (2 minutes))).protocols(httpProtocol)
}

Vivek_R · January 7, 2014, 10:49pm

No exceptions this time, but the files are still not getting uploaded
so I'm getting a 400 back instead of 200.
Vivek

Excilys · January 8, 2014, 6:26am

Could you try this, please? The problem could be that the multipart-form-data header was no longer automatically added:

import io.gatling.core.Predef._
import io.gatling.http.Predef._
import io.gatling.jdbc.Predef._
import scala.concurrent.duration._

class PerformanceTest extends Simulation {

val httpProtocol = http
.baseURL(“http://ul.dev.test.com:8080”)

val performanceTest = csv(“uploadUrls.csv”).random
val scn = scenario(“Performance Test”)
.feed(performanceTest)
.exec(
http(“Performance”)
.post("/uploader/api/v1/upload/${productId}/1/userPDF/upload")
.header(“CheckSum”, “${checkSum}”)
.header(“FileSize”, “${fileSize}”)
.header(“UAUSERID”, “17”)
.header(“userId”, “17”)
.header(“EmailAddress”, “ad…@test.com”)
.bodyPart(RawFileBodyPart(“data”, “${fileName}”).contentType(“application/pdf”))
.check(status.is(200))
.asMultipartForm)
setUp(scn.inject(atOnceUsers(10), nothingFor(10 seconds), constantUsersPerSec(20) during (2 minutes))).protocols(httpProtocol)
}

Vivek_R · January 8, 2014, 6:29pm

Still no luck.

Attached some info:

Select simulation id (default is ‘performancetest’). Accepted characters are a-z, A-Z, 0-9, - and _

Select run description (optional)

Simulation performanceTest.PerformanceTest started…

Excilys · January 8, 2014, 6:32pm

Damn.
I will try to reproduce tomorrow. Do you think you could get request dumps, for example with Charles?

Excilys · January 8, 2014, 9:53pm

OK, I think I get it.
I refactored multipart in Gatling 2, so that it can address the more general use case of multipart, and not only form upload.

The thing is with form is that browsers always send filename attribute (in Content-Disposition header) and servers usually expect it, but from a multipart perspective, this information is optional.

So, you have to specify it:

import io.gatling.core.Predef._
import io.gatling.http.Predef._
import io.gatling.jdbc.Predef._
import scala.concurrent.duration._

class PerformanceTest extends Simulation {

val httpProtocol = http
.baseURL(“http://ul.dev.test.com:8080”)

val performanceTest = csv(“uploadUrls.csv”).random
val scn = scenario(“Performance Test”)
.feed(performanceTest)
.exec(
http(“Performance”)
.post("/uploader/api/v1/upload/${productId}/1/userPDF/upload")
.header(“CheckSum”, “${checkSum}”)
.header(“FileSize”, “${fileSize}”)
.header(“UAUSERID”, “17”)
.header(“userId”, “17”)
.header(“EmailAddress”, “ad…@test.com”)
.bodyPart(RawFileBodyPart(“data”, “${fileName}”).contentType(“application/pdf”).fileName("${fileName}"))
.check(status.is(200))
.asMultipartForm)

setUp(scn.inject(atOnceUsers(10), nothingFor(10 seconds), constantUsersPerSec(20) during (2 minutes))).protocols(httpProtocol)
}

Could you check this out, please?
If that does fix your problem (I’m almost sure it does), I’ll probably add a formUpload built-in back.

Vivek_R · January 8, 2014, 10:41pm

Great. This works. Yes, I can definitely see the need for a formUpload built-in.

I’m running into other memory issues, but I’m playing with JAVA_OPTS to increase the heap and stack sizes.

Thank you so much for your help again.

Vivek_R · January 9, 2014, 1:40am

Maybe I spoke too soon. We were able to run 18 to 20 concurrent users, but after that it fails every time with the following error and dies

Excilys · January 9, 2014, 6:19am

Weird.
Could you try downgrading Netty to 3.8.0 (the version used in 1.5.3), please?

Topic		Replies	Views
[Gatling 2] Fileuploads Gatling (Open-Source)	10	242	June 25, 2013
Gatling test - pdf upload Gatling (Open-Source)	1	192	February 13, 2017
Multi-part form problem Gatling (Open-Source)	2	205	March 24, 2016
[Gatling 2.0.0-M3] What are HTTP API changes since Gatling 2.0.0-M2 ? Gatling (Open-Source)	7	144	July 8, 2013
multipart/related implementation in gatling Gatling (Open-Source)	6	227	May 27, 2015

Issue with multipart form data upload in Gatling 2.0.0-M3

Related topics