feat: make packet loss opt-in and drop dead TURN defaults

README.md

@@ -46,8 +46,8 @@ them.
 | **autoStart**: *boolean* | Whether to automatically start the measurements on instantiation. | `true` |
 | **downloadApiUrl**: *string* | The URL of the API for performing download GET requests. | `https://speed.cloudflare.com/__down` |
 | **uploadApiUrl**: *string* | The URL of the API for performing upload POST requests. | `https://speed.cloudflare.com/__up` |
-| **turnServerUri**: *string* | The URI of the TURN server used to measure packet loss. | `turn.cloudflare.com:3478` |
-| **turnServerCredsApiUrl**: *string* | A URI that returns TURN server credentials. Expects a JSON response with `username` and `credential` keys. | - | 
+| **turnServerUri**: *string* | The URI of the TURN server used to measure packet loss, e.g. `turn.cloudflare.com:3478`. See [packetLoss](#packetloss). | `null` |
+| **turnServerCredsApiUrl**: *string* | A URI that returns TURN server credentials. Expects a JSON response with `username` and `credential` keys. See [packetLoss](#packetloss). | `null` |
 | **turnServerUser**: *string* | The username for the TURN server credentials. | - |
 | **turnServerPass**: *string* | The password for the TURN server credentials. | - |
 | **authorizationToken**: *string* | 🧪 *Experimental.* An opaque token attributing the test to a registered customer, sent as an `Authorization: Bearer` header on the measurement, TURN credential and results-logging requests. Obtain it from your own backend — the engine never requests one itself. Requires `authorizationEnabled`; on its own it is never sent. Not sent over plain HTTP unless `allowInsecureAuthorizationToken` is set. | `null` |
@@ -105,7 +105,6 @@ The default set of measurements that is performed by the engine is:
   { type: 'download', bytes: 1e5, count: 9 },
   { type: 'download', bytes: 1e6, count: 8 },
   { type: 'upload', bytes: 1e5, count: 8 },
-  { type: 'packetLoss', numPackets: 1e3, responsesWaitTime: 3000 },
   { type: 'upload', bytes: 1e6, count: 6 },
   { type: 'download', bytes: 1e7, count: 6 },
   { type: 'upload', bytes: 1e7, count: 4 },
@@ -139,9 +138,25 @@ Each of these measurement sets are bound to a specific file size. The engine fol
 
 Packet loss is measured by submitting a set of UDP packets to a WebRTC TURN server in a round-trip fashion, and determining how many packets do not arrive. The submission of these packets can be done in a batching method, in which there's a sleep time in between batches.
 
-**Note:** You must provide your own TURN server configuration if you'd like to get packet loss results from this engine.
+Packet loss is opt-in: it is not part of the default measurements, and it needs your own TURN server configuration. To enable it, add a `packetLoss` measurement and either:
 
-Example code to enable this feature through a Cloudflare Worker, using a [Cloudflare Realtime TURN server](https://developers.cloudflare.com/realtime/turn/), is available at [example/turn-worker](example/turn-worker). Refer to the worker's [documentation](example/turn-worker/README.md) for setup and deployment instructions.
+- set `turnServerCredsApiUrl` to an endpoint you host that returns TURN credentials, plus `turnServerUri` (e.g. `turn.cloudflare.com:3478`) unless the endpoint also returns a `server` field, or
+- set `turnServerUri` together with your own `turnServerUser` and `turnServerPass`.
+
+```js
+new SpeedTest({
+  turnServerUri: 'turn.cloudflare.com:3478',
+  turnServerCredsApiUrl: 'https://turn.example.com/turn-credentials',
+  measurements: [
+    // ...
+    { type: 'packetLoss', numPackets: 1e3, responsesWaitTime: 3000 }
+  ]
+});
+```
+
+A `packetLoss` measurement without either option fails with an error, and the engine moves on to the next measurement.
+
+Example code to serve TURN credentials from a Cloudflare Worker, using a [Cloudflare Realtime TURN server](https://developers.cloudflare.com/realtime/turn/), is available at [example/turn-worker](example/turn-worker). Refer to the worker's [documentation](example/turn-worker/README.md) for setup and deployment instructions.
 
 | Field | Required | Description | Default |
 | --- | :--: | --- | :--: |
@@ -174,7 +189,7 @@ An instance object used to access the results of the speedtest measurements. The
 | **getPacketLoss()** | Returns the reduced value of the measured packet loss ratio (between 0 and 1). Requires a `packetLoss` measurement set. |
 | **getPacketLossDetails()** | Returns an object with the details of the packet loss measurement. Includes the following fields: `{ packetLoss, totalMessages, numMessagesSent, lostMessages }`. Requires a `packetLoss` measurement set. |
 | **getTotalDurationMs()** | Returns the total time (in milliseconds) the engine spent running, excluding any time spent paused. Only available after the engine has finished. |
-| **getScores()** | Returns the computed [AIM scores](https://developers.cloudflare.com/fundamentals/speed/aim/) that categorize the quality of the network connection according to use cases such as streaming, gaming or real-time communications. This score is only available after the engine has finished performing all of the measurements. |
+| **getScores()** | Returns the computed [AIM scores](https://developers.cloudflare.com/fundamentals/speed/aim/) that categorize the quality of the network connection according to use cases such as streaming, gaming or real-time communications. This score is only available after the engine has finished performing all of the measurements. Without a `packetLoss` measurement, packet loss is left out of the scores, so they come out lower than for a connection measured with under 5% loss. A failed `packetLoss` measurement counts as 0 points. |
 
 [npm-img]: https://img.shields.io/npm/v/@cloudflare/speedtest
 [npm-url]: https://npmjs.org/package/@cloudflare/speedtest

example/turn-worker/README.md

@@ -34,9 +34,9 @@ Run `npm run deploy` and follow the instructions.
 
 ### Configuring the measurement engine
 
-When instantiating the measurement engine, set the `turnServerCredsApiUrl` option to `https://<your-worker-domain>/turn-credentials`.
+When instantiating the measurement engine, set the `turnServerCredsApiUrl` option to `https://<your-worker-domain>/turn-credentials` and `turnServerUri` to `turn.cloudflare.com:3478`, and add a `packetLoss` measurement (it is not part of the defaults).
 
-For local development, run `npm run start` and set `turnServerCredsApiUrl` to `http://localhost:8787/turn-credentials`
+For local development, run `npm run start` and set `turnServerCredsApiUrl` to `http://localhost:8787/turn-credentials`, keeping `turnServerUri` set to `turn.cloudflare.com:3478`.
 
 ### More information
 

src/Results/ScoresCalculations.ts

@@ -113,12 +113,14 @@ class ScoresCalculations {
    *    via the experience's `pointThresholds`.
    *
    * Experiences are omitted from the result if any of their required
-   * input metrics are not yet available.
+   * input metrics are not yet available. A missing packet loss value counts
+   * as 0 points (e.g. a failed measurement), unless it is listed in `omit`.
    *
    * @param measurements - Flat summary from {@link Results.getSummary}.
+   * @param omit - Metrics that were not measured, left out of every experience's inputs.
    * @returns Map of experience names to their score, classification index, and classification name.
    */
-  getScores(measurements: MeasurementSummary): Scores {
+  getScores(measurements: MeasurementSummary, omit: string[] = []): Scores {
     const scores: Record<string, number> = Object.assign(
       {} as Record<string, number>,
       ...Object.entries(this.#config.aimMeasurementScoring).map(
@@ -140,6 +142,13 @@ class ScoresCalculations {
     return Object.assign(
       {} as Scores,
       ...Object.entries(this.#config.aimExperiencesDefs)
+        .map(
+          ([k, { input, pointThresholds }]) =>
+            [
+              k,
+              { input: input.filter(m => !omit.includes(m)), pointThresholds }
+            ] as const
+        )
         .filter(([, { input }]) => input.every(k => scores.hasOwnProperty(k)))
         .map(([k, { input, pointThresholds }]) => {
           const sumPoints = Math.max(0, sum(input.map(k => scores[k])));

src/Results/index.ts

@@ -203,9 +203,14 @@ class Results {
   /**
    * Computes AIM experience scores (streaming, gaming, rtc) from the current
    * measurement summary. Each score includes a point total, classification
-   * index (0–4), and classification name (bad → great).
+   * index (0–4), and classification name (bad → great). Packet loss is left
+   * out when no `packetLoss` measurement is configured.
    */
-  getScores = (): Scores => this.#scoresCalc.getScores(this.getSummary());
+  getScores = (): Scores =>
+    this.#scoresCalc.getScores(
+      this.getSummary(),
+      this.raw.packetLoss ? [] : ['packetLoss']
+    );
 
   // Internal state
   #config: ResultsConfig;

src/config/defaultConfig.ts

@@ -53,10 +53,13 @@ export interface Config {
   logMeasurementApiUrl: string | null;
   /** URL for logging test results. Set to `null` to disable. Default: `https://speed.cloudflare.com/__results`. */
   logAimApiUrl: string | null;
-  /** TURN server URI for packet loss measurement. Default: `turn.speed.cloudflare.com:50000`. */
-  turnServerUri: string;
-  /** URL for fetching TURN server credentials. */
-  turnServerCredsApiUrl: string;
+  /**
+   * TURN server URI for packet loss measurement, e.g. `turn.cloudflare.com:3478`.
+   * Default: `null`, so a `packetLoss` measurement needs your own TURN setup.
+   */
+  turnServerUri: string | null;
+  /** URL for fetching TURN server credentials. Default: `null`. */
+  turnServerCredsApiUrl: string | null;
   /** Static TURN username override. Set to `null` to fetch credentials dynamically. */
   turnServerUser: string | null;
   /** Static TURN password override. Set to `null` to fetch credentials dynamically. */
@@ -164,8 +167,8 @@ const defaultConfig: Config = {
   uploadApiUrl: `${REL_API_URL}/__up`,
   logMeasurementApiUrl: null,
   logAimApiUrl: `${REL_API_URL}/__results`,
-  turnServerUri: 'turn.speed.cloudflare.com:50000',
-  turnServerCredsApiUrl: `${REL_API_URL}/turn-creds`,
+  turnServerUri: null,
+  turnServerCredsApiUrl: null,
   turnServerUser: null,
   turnServerPass: null,
   rpkiInvalidHost: 'invalid.rpki.cloudflare.com',
@@ -186,13 +189,6 @@ const defaultConfig: Config = {
     { type: 'latency', numPackets: 2 },
     { type: 'upload', bytes: 1e5, count: 8 },
     { type: 'latency', numPackets: 2 },
-    {
-      type: 'packetLoss',
-      numPackets: 1e3,
-      batchSize: 10,
-      batchWaitTime: 10, // ms (in between batches)
-      responsesWaitTime: 3000 // ms (silent time after last sent msg)
-    },
     { type: 'upload', bytes: 1e6, count: 6 },
     { type: 'latency', numPackets: 2 },
     { type: 'download', bytes: 1e7, count: 6 },

src/engines/PacketLossEngine/PacketLossEngine.ts

@@ -69,11 +69,13 @@ export default class PacketLossEngine implements Engine {
     connectionTimeout = 5000 // ms
   }: PacketLossEngineOptions = {}) {
     if (!turnServerUri && !turnServerCredsApi)
-      throw new Error('Missing turnServerCredsApi or turnServerUri argument');
+      throw new Error(
+        'No TURN server configured: set turnServerCredsApiUrl or turnServerUri'
+      );
 
     if ((!turnServerUser || !turnServerPass) && !turnServerCredsApi)
       throw new Error(
-        'Missing either turnServerCredsApi or turnServerUser+turnServerPass arguments'
+        'No TURN credentials configured: set turnServerCredsApiUrl, or turnServerUser and turnServerPass'
       );
 
     this.#numMsgs = numMsgs;
@@ -110,10 +112,18 @@ export default class PacketLossEngine implements Engine {
           turnServerUri: credsApiTurnServerUri
         } = creds;
 
+        const uri = credsApiTurnServerUri || turnServerUri;
+        if (!uri) {
+          this.#onConnectionError(
+            'No TURN server URI: set turnServerUri or return server from the credentials API'
+          );
+          return;
+        }
+
         const c = new SelfWebRtcDataConnection({
           iceServers: [
             {
-              urls: `turn:${credsApiTurnServerUri || turnServerUri}?transport=udp`,
+              urls: `turn:${uri}?transport=udp`,
               username: credsUser,
               credential: credsPass
             }

src/index.ts

@@ -438,27 +438,44 @@ class MeasurementEngine {
             turnServerPass,
             includeCredentials
           } = this.#config;
-          engine = new PacketLossEngine({
-            turnServerUri,
-            turnServerCredsApi,
-            turnServerCredsApiIncludeCredentials: includeCredentials,
-            turnServerUser: turnServerUser ?? undefined,
-            turnServerPass: turnServerPass ?? undefined,
-            authorization: this.authorization,
-            numMsgs,
 
-            // if under load
-            downloadChunkSize: msmConfig.loadDown
-              ? this.#optimalDownloadChunkSize
-              : undefined,
-            uploadChunkSize: msmConfig.loadUp
-              ? this.#optimalUploadChunkSize
-              : undefined,
-            downloadApiUrl,
-            uploadApiUrl,
+          try {
+            engine = new PacketLossEngine({
+              turnServerUri: turnServerUri ?? undefined,
+              turnServerCredsApi: turnServerCredsApi ?? undefined,
+              turnServerCredsApiIncludeCredentials: includeCredentials,
+              turnServerUser: turnServerUser ?? undefined,
+              turnServerPass: turnServerPass ?? undefined,
+              authorization: this.authorization,
+              numMsgs,
+
+              // if under load
+              downloadChunkSize: msmConfig.loadDown
+                ? this.#optimalDownloadChunkSize
+                : undefined,
+              uploadChunkSize: msmConfig.loadUp
+                ? this.#optimalUploadChunkSize
+                : undefined,
+              downloadApiUrl,
+              uploadApiUrl,
 
-            ...ptCfg
-          }) as Engine;
+              ...ptCfg
+            }) as Engine;
+          } catch (e) {
+            const message = e instanceof Error ? e.message : String(e);
+            const measurementId = this.#measurementId;
+            // Deferred so that, with autoStart, the caller can set onError first
+            queueMicrotask(() => {
+              if (measurementId !== this.#measurementId) return; // restarted since
+              msmResults.started = true;
+              msmResults.finished = true;
+              msmResults.error = message;
+              this.onResultsChange({ type });
+              this.#onError(`Error while measuring packet loss: ${message}`);
+              this.#running && this.#next();
+            });
+            return;
+          }
         }
 
         engine!.onMsgReceived = () => {
@@ -476,13 +493,15 @@ class MeasurementEngine {
         };
 
         engine!.onConnectionError = (e: unknown) => {
+          msmResults.finished = true;
           msmResults.error = e;
           this.onResultsChange({ type });
           this.#onError(`Connection error while measuring packet loss: ${e}`);
           this.#next();
         };
 
         engine!.onCredentialsFailure = () => {
+          msmResults.finished = true;
           msmResults.error = 'unable to get turn server credentials';
           this.onResultsChange({ type });
           this.#onError(

tests/unit/Results/Results.test.ts

@@ -1,4 +1,4 @@
-import { describe, it, expect } from 'vitest';
+import { describe, it, expect, vi, afterEach } from 'vitest';
 import Results from '../../../src/Results/index.ts';
 import type {
   RawMeasurementEntry,
@@ -15,20 +15,29 @@ function createResults(configOverrides: Partial<ResultsConfig> = {}) {
   } as ResultsConfig);
 }
 
+const WITH_PACKET_LOSS: ResultsConfig['measurements'] = [
+  { type: 'packetLoss' }
+];
+
 describe('Results', () => {
   describe('constructor and clear', () => {
     it('initializes raw results with measurement types from config', () => {
       const results = createResults();
       expect(results.raw).toHaveProperty('latency');
       expect(results.raw).toHaveProperty('download');
       expect(results.raw).toHaveProperty('upload');
-      expect(results.raw).toHaveProperty('packetLoss');
       expect(results.raw).toHaveProperty('totalDurationMs');
+      expect(results.raw).not.toHaveProperty('packetLoss');
+    });
+
+    it('initializes packet loss results only when configured', () => {
+      const results = createResults({ measurements: WITH_PACKET_LOSS });
+      expect(results.raw).toHaveProperty('packetLoss');
     });
 
     it('marks all measurements as not started and not finished', () => {
       const results = createResults();
-      for (const key of ['latency', 'download', 'upload', 'packetLoss']) {
+      for (const key of ['latency', 'download', 'upload']) {
         const entry = results.raw[key] as RawMeasurementEntry;
         expect(entry.started).toBe(false);
         expect(entry.finished).toBe(false);
@@ -137,7 +146,7 @@ describe('Results', () => {
     });
 
     it('returns packet loss details with error when error exists', () => {
-      const results = createResults();
+      const results = createResults({ measurements: WITH_PACKET_LOSS });
       const packetLoss = results.raw.packetLoss as RawMeasurementEntry;
       packetLoss.started = true;
       packetLoss.error = 'Connection failed';
@@ -207,4 +216,48 @@ describe('Results', () => {
       expect(typeof scores).toBe('object');
     });
   });
+
+  describe('getScores packet loss handling', () => {
+    // 20 (latency) + 30 (download) + 20 (loadedLatencyIncrease) for streaming
+    const SUMMARY = {
+      latency: 5,
+      jitter: 5,
+      download: 100e6,
+      downLoadedLatency: 10,
+      upLoadedLatency: 8
+    };
+
+    afterEach(() => {
+      vi.restoreAllMocks();
+    });
+
+    it('counts a measured packet loss', () => {
+      const results = createResults({ measurements: WITH_PACKET_LOSS });
+      vi.spyOn(results, 'getSummary').mockReturnValue({
+        ...SUMMARY,
+        packetLoss: 0 // 10 points
+      });
+
+      expect(results.getScores().streaming.points).toBe(80);
+    });
+
+    it('scores every experience without packet loss when it is not configured', () => {
+      const results = createResults();
+      vi.spyOn(results, 'getSummary').mockReturnValue(SUMMARY);
+
+      const scores = results.getScores();
+      expect(Object.keys(scores)).toEqual(['streaming', 'gaming', 'rtc']);
+      expect(scores.streaming.points).toBe(70);
+    });
+
+    it('scores a failed packet loss measurement as 0 points', () => {
+      const results = createResults({ measurements: WITH_PACKET_LOSS });
+      const packetLoss = results.raw.packetLoss as RawMeasurementEntry;
+      packetLoss.started = true;
+      packetLoss.error = 'Connection failed';
+      vi.spyOn(results, 'getSummary').mockReturnValue(SUMMARY);
+
+      expect(results.getScores().streaming.points).toBe(70);
+    });
+  });
 });

tests/unit/Results/ScoresCalculations.test.ts

@@ -105,5 +105,24 @@ describe('ScoresCalculations', () => {
       // + download (30) + loadedLatencyIncrease (10-5=5 => 20 points)
       expect(scores.streaming).toBeDefined();
     });
+
+    it('leaves omitted packetLoss out of every experience', () => {
+      const calc = createScoresCalc();
+      const scores = calc.getScores(
+        {
+          latency: 5, // 20 points
+          jitter: 5, // 10 points
+          packetLoss: 0, // 10 points, unless omitted
+          download: 100e6, // 30 points
+          downLoadedLatency: 10, // loadedLatencyIncrease = 5 => 20 points
+          upLoadedLatency: 8
+        },
+        ['packetLoss']
+      );
+
+      expect(scores.streaming.points).toBe(70);
+      expect(scores.gaming.points).toBe(40);
+      expect(scores.rtc.points).toBe(50);
+    });
   });
 });