Hoàng
Building biomedical NLP systems

Experimental pipeline

POV Video Segmentation and Keyframe Selection

Started
2026-07
Last revised

A shot-aware preprocessing pipeline for long first-person video: locate visual boundaries, sample representative frames, embed them, and remove near-duplicates without erasing meaningful transitions.

  • TransNetV2
  • FFmpeg
  • SigLIP2
  • Python
  • Cosine similarity
RQ

Research question

Can shot boundaries and semantic frame embeddings produce a compact keyframe sequence that preserves the events of long, motion-heavy POV video?

01

From input to inspectable output

System architecture

Temporal trace · boundary-aware sampling
Shot segmentation and keyframe selection timelineA video is divided at two inferred shot boundaries. Candidate frames are embedded, and three representative frames remain after local similarity filtering.SHOT 01SHOT 02SHOT 03keyframe · 01keyframe · 02keyframe · 03
Fig. 02Large points survive local cosine filtering; shot identity and timestamps remain attached.
  1. 01

    Decode and probe

    FFmpeg exposes stable timestamps, frame rate, and a low-resolution analysis stream.

    in
    POV video
    out
    timestamped frames
  2. 02

    Boundary inference

    TransNetV2 proposes hard and gradual shot transitions on the analysis stream.

    in
    frame window
    out
    shot intervals
  3. 03

    Candidate sampling

    Frames are sampled within each shot, with extra coverage near uncertain boundaries.

    in
    shot interval
    out
    candidate frames
  4. 04

    Semantic embedding

    SigLIP2 maps candidates into a common visual representation.

    in
    candidate image
    out
    normalized vector
  5. 05

    Temporal deduplication

    Cosine similarity removes redundant neighbors while retaining timestamps and shot identity.

    in
    ordered vectors
    out
    keyframe sequence
02

What the design must respect

Operating constraints

  1. 01

    Camera motion and abrupt head turns can resemble hard cuts.

  2. 02

    Long recordings make uniform dense sampling expensive.

  3. 03

    Adjacent frames may be visually redundant while encoding a brief but meaningful action.

  4. 04

    Frame timestamps must remain traceable to the original media.

03

Evaluation ledger

Current results

Placeholder results — no benchmark claim is being made.

Values will be added only with the evaluation set, protocol, ontology version, and relevant runtime conditions.

R01Boundary F1
Not reportedPlaceholder

Requires a manually checked POV boundary slice.

R02Keyframe reduction ratio
Not reportedPlaceholder

Will be reported together with event-retention review, not in isolation.

R03Processing rate
Not reportedPlaceholder

Hardware, decode resolution, and sampling policy will accompany the measurement.

04

Negative evidence

Failed or insufficient approaches

F01

Uniform sampling alone

It spends frames on static intervals and can miss brief transitions between sampling points.

Design response

Use shot intervals to allocate samples and preserve boundary-adjacent evidence.

F02

Global similarity deduplication

Visually similar scenes far apart in time can represent distinct visits or events.

Design response

Deduplicate locally and retain temporal and shot constraints.

05

What changed in the model of the problem

Lessons

  • A compact summary needs both visual diversity and temporal coverage.
  • Boundary confidence is more useful when preserved than when immediately thresholded away.
  • Timestamp provenance should survive every transformation.
06

Queue, not promises

Next experiments

  1. 01Create a small hand-labelled POV set containing motion blur and gradual transitions.
  2. 02Compare local greedy deduplication with diversity-aware subset selection.
  3. 03Test whether boundary confidence should control per-shot sampling density.
07
  1. How TransNetV2 Detects Shot Boundaries

    A model-level reading of local and long-range temporal evidence.