Looking around and reframing

A stitched 360 video is an equirectangular picture of the whole sphere. web360 draws flat views of it, for looking around in a player and for exporting a flat video that follows a path of views.

A view

A View is five numbers, the ones 360 editors keyframe:

Part	Meaning
pan	Degrees to the right of straight ahead
tilt	Degrees above the horizon, from -90 to 90
roll	Degrees the top of the picture leans to the right, from -180 to 180
fov	The field of view across the picture's longer side, from 30 to 160 degrees
distance	0 for a flat lens, up to 1 for a stereographic one

VIEW_PRESETS holds five lenses: linear, dewarp, ultra, mega and tinyPlanet. Spread one over a view to apply it: { ...view, ...VIEW_PRESETS.ultra }. clampView holds a view to its limits and wraps its pan. DEFAULT_VIEW looks straight ahead through the linear lens.

A path of keyframes

viewAt(keyframes, time) is the view a path of { time, view } keyframes shows at a time in seconds:

• it passes through each keyframe;
• it moves smoothly between them without overshooting;
• it eases out of the first keyframe and into the last;
• it holds them beyond the ends;
• the pan turns the short way round.

Look around in a player

createViewer(canvas) draws views of a <video> (or a decoded VideoFrame) on a canvas with WebGPU. It keeps no clock and handles no input: call draw on every video frame and every drag. Import it from @bvdmitri/web360/view, which leaves the stitcher out of the page.

// Look around in a playing 360 video: a canvas over the <video>, redrawn on
// every video frame and on every drag.
import {
  clampView,
  createViewer,
  DEFAULT_VIEW,
  type View,
} from '@bvdmitri/web360/view';

const DEGREES_PER_PIXEL = 0.2;

export async function lookAround(
  video: HTMLVideoElement,
  canvas: HTMLCanvasElement,
): Promise<() => void> {
  const viewer = await createViewer(canvas);
  let view: View = DEFAULT_VIEW;
  let isStopped = false;
  const draw = (): void => {
    if (video.readyState >= 2) void viewer.draw(video, view);
  };
  const onFrame = (): void => {
    if (isStopped) return;
    draw();
    video.requestVideoFrameCallback(onFrame);
  };
  video.requestVideoFrameCallback(onFrame);
  // Drag the picture: moving it left turns the view right.
  canvas.addEventListener('pointermove', (event) => {
    if (event.buttons !== 1) return;
    view = clampView({
      ...view,
      pan: view.pan - event.movementX * DEGREES_PER_PIXEL,
      tilt: view.tilt + event.movementY * DEGREES_PER_PIXEL,
    });
    draw();
  });
  return () => {
    isStopped = true;
    viewer.destroy();
  };
}

Export a flat video

createReframer({ width, height }) turns decoded 360 frames into flat views of them, with the same timestamps, ready for an encoder. Decoding, the audio and encoding are the caller's. Here Mediabunny's Conversion does all three, and the reframer draws each frame's view along the path:

// Export a flat 16:9 video from a 360 one along a keyframed path, with
// Mediabunny decoding, copying the audio and encoding.
import {
  createReframer,
  viewAt,
  type Reframer,
  type ViewKeyframe,
} from '@bvdmitri/web360';
import {
  ALL_FORMATS,
  BlobSource,
  BufferTarget,
  Conversion,
  Input,
  Mp4OutputFormat,
  Output,
  VideoSample,
} from 'mediabunny';

const WIDTH = 1920;
const HEIGHT = 1080;

// Each 360 frame in, its view at the frame's time (in seconds) out.
function reframed(
  reframer: Reframer,
  keyframes: readonly ViewKeyframe[],
): (sample: VideoSample) => Promise<VideoSample> {
  return async (sample) => {
    const frame = sample.toVideoFrame();
    const view = viewAt(keyframes, sample.timestamp);
    const flat = await reframer.reframe(frame, view);
    frame.close();
    return new VideoSample(flat);
  };
}

export async function reframeToMp4(
  file: File,
  keyframes: readonly ViewKeyframe[],
  onProgress: (share: number) => void,
): Promise<Blob> {
  const reframer = await createReframer({ width: WIDTH, height: HEIGHT });
  const output = new Output({
    format: new Mp4OutputFormat(),
    target: new BufferTarget(),
  });
  try {
    const source = new BlobSource(file);
    const conversion = await Conversion.init({
      input: new Input({ formats: ALL_FORMATS, source }),
      output,
      video: {
        processedWidth: WIDTH,
        processedHeight: HEIGHT,
        process: reframed(reframer, keyframes),
      },
    });
    conversion.onProgress = onProgress;
    await conversion.execute();
  } finally {
    reframer.destroy();
  }
  const bytes = output.target.buffer ?? new ArrayBuffer(0);
  return new Blob([bytes], { type: 'video/mp4' });
}

Both need WebGPU, and throw a Web360Error coded render-unavailable without it.