Hello, My task to is test the performance of a LLM. Along with total response time I need the get the time-to-first-token metric. The call is an HTTPS POST, per example ‘ https://api.openai.com/v1/chat/completions ’ with ‘stream’ option added to the request body ( which serves the response as chat.c…

LLM inference - completions endpoint

tcristi74 October 2, 2023, 10:08pm 3

Thanks for the quick response. However, I need to pass a request body and doesn’t look like SseConnectRequestBuilder supports that ( I am at version 3.7)

Topic		Replies	Views
Gatling from a loadrunner perspective Gatling (Open-Source)	4	200	August 30, 2013
Response time for each request Gatling (Open-Source)	3	182	November 9, 2015
Individual Page Response time Gatling (Open-Source)	1	172	March 30, 2013
Large overhead on first request Gatling (Open-Source)	19	103	May 30, 2012
Output mean response times in realtime monitoring Gatling (Open-Source)	1	144	September 12, 2016

LLM inference - completions endpoint

Related topics