Scene detection

Find where the shots of a video change, as a JSON list of scenes with their times and scores.

POST /v1/scenes reads a video and writes a JSON file with every scene: where it starts, where it ends, and how sharp the change into it was. In POST /v1/jobs it is an output of type scenes, next to any other outputs of the same input.

The request#

bash

curl "$VIDEXO_API_URL/v1/scenes" \
  -H "Authorization: Bearer $VIDEXO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "input": "https://example.com/talk.mp4",
  "filename": "scenes.json",
  "threshold": 12
}'

JavaScript

const response = await fetch(`${process.env.VIDEXO_API_URL}/v1/scenes`, {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${process.env.VIDEXO_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    "input": "https://example.com/talk.mp4",
    "filename": "scenes.json",
    "threshold": 12
  }),
});
const job = await response.json();

bash

ffmpeg \
  -i https://example.com/talk.mp4 \
  -map 0:v:0 \
  -vf scdet=threshold=12,settb=1/1000000,metadata=mode=print:key=lavfi.scd.score:file=scenes.scores.txt \
  -f null \
  -

scenes.json: every scene with its start and end in seconds, and the score from 0 to 100 of the change that starts it.

The file#

JSON

{
  "threshold": 12,
  "start": 0,
  "end": 6,
  "scenes": [
    { "start": 0, "end": 2, "score": null },
    { "start": 2, "end": 4, "score": 26.552 },
    { "start": 4, "end": 6, "score": 22.034 }
  ]
}

Times are in seconds on the input's clock. score runs from 0 (the frame before looks the same) to 100, and the first scene has none. A video with no change has one scene from start to end.

The threshold#

A frame whose score is at least threshold starts a new scene. The range is 1 to 100, and the default is 10. Lower it for soft cuts and dark footage, raise it if movement in a shot splits the scene. A gradual fade spreads its change over many frames, so it may not reach the threshold.

A part of a video#

trim reads only that part; the times stay on the input's clock. A trim with none of the video in it fails with processing_failed (time_past_end) and costs only the data fetched, even when the file's sound runs on past the picture.

bash

curl "$VIDEXO_API_URL/v1/scenes" \
  -H "Authorization: Bearer $VIDEXO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "input": "https://example.com/talk.mp4",
  "trim": {
    "start": 60,
    "end": 120
  }
}'

JavaScript

const response = await fetch(`${process.env.VIDEXO_API_URL}/v1/scenes`, {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${process.env.VIDEXO_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    "input": "https://example.com/talk.mp4",
    "trim": {
      "start": 60,
      "end": 120
    }
  }),
});
const job = await response.json();

bash

ffmpeg \
  -ss 60 -t 60 -i https://example.com/talk.mp4 \
  -map 0:v:0 \
  -vf setpts=PTS+60/TB,scdet=threshold=10,settb=1/1000000,metadata=mode=print:key=lavfi.scd.score:file=scenes.scores.txt \
  -f null \
  -

The output has no other options. To get a picture of a scene, send its start to POST /v1/thumbnails in times.

As a command#

.json output files are scene lists in commands too (Command rules). ffmpeg_command runs the same detection on your own ffmpeg, which writes the raw scores to <name>.scores.txt instead of the JSON file.