WebRTC
2020/04/15 PWA Night vol.15
RTCPeerConnection
RTCRtpTransceiver
RTCRtpSender
RTCRtpReceiver
RTCDataChannel
navigator.mediaDevices
MediaStream
MediaStreamTrack
MediaRecorder
WebAudio
HTMLMediaElement
Today's main theme
There are many different use-cases for WebRTC, from basic web apps that uses the camera or microphone, to more advanced video-calling applications and screen sharing.
from https://webrtc.org/
PWA
So today, I'll introduce the concepts only
x.x.x.x:yyyyy
// sender side const pc1 = new RTCPeerConnection(); // RTCRtpSender const sender = pc1.addTrack(track, stream); /* ... negotiation flows are omitted ... */ // receiver side pc2.addEventListener("track", ev => { // RTCRtpReceiver const { receiver } = ev; $video.srcObject = new MediaStream([receiver.track]); });
https://dontcallmedom.github.io/webrtc-impl-tracker/?webrtc
// for camera, microphone await navigator.mediaDevices.getUserMedia({ video: true, audio: true }); // for screen sharing await navigator.mediaDevices.getDisplayMedia({ video: true });
getDisplayMedia()
// from MediaStream videoElement.srcObject = stream; // deprecated // videoElement.src = URL.createObjectURL(stream); // from MediaStreamTrack audioElement.srcObject = new MediaStream([audioTrack]);
autoplay
playsinline
// from canvas or other HTMLMediaElement canvasElement.captureStream(fps); // from WebAudio audioContext.createMediaStreamDestination();
captureStream()
const devices = await navigator.mediaDevices.enumerateDevices(); const [audioIn] = devices.filter(d => d.kind === "audioinput"); const [videoIn] = devices.filter(d => d.kind === "videoinput"); // Chrome, ChromiumEdge only const [audioOut] = devices.filter(d => d.kind === "audiooutput");
default
label
getUserMedia()
const [videoIn] = devices.filter(d => d.kind === "videoinput"); await navigator.mediaDevices.getUserMedia({ video: { deviceId: { exact: videoIn.deviceId } } });
exact
const [audioOut] = devices.filter(d => d.kind === "audiooutput"); await audioElement.setSinkId(audioOut.deviceId);
const mediaRecorder = new MediaRecorder(stream); const chunks = []; mediaRecorder.ondataavailable = ev => chunks.push(ev.data); mediaRecorder.onstop = () => finalize(new Blob(chunks, { type: "audio/ogg; codecs=opus" })); mediaRecorder.start(); // mediaRecorder.stop();
https://github.com/alvestrand/webrtc-media-streams
const pc = new RTCPeerConnection({ forceEncodedVideoInsertableStreams: true }); const senderStreams = videoSender.getEncodedVideoStreams(); senderStreams.readable .pipeThrough(senderTransform) // your pipeline here .pipeTo(senderStreams.writable);
https://github.com/ricea/websocketstream-explainer
const wss = new WebSocketStream(url); const { readable } = await wss.connection; const reader = readable.getReader(); while (true) { const { value, done } = await reader.read(); if (done) break; await process(value); }
WebSocket
whatwg/streams
https://github.com/WICG/web-transport
const transport = new QuicTransport("example.com", 10001); setInterval(async () => { const stream = await transport.createSendStream(); const writer = stream.writable.getWriter(); writer.write(getSerializedGameState()); writer.close(); }, 100);
RTCQuicTransport
https://github.com/WICG/web-codecs
const videoDecoder = new VideoDecoder({ output(videoFrame) { const bitmap = videoFrame.transferToImageBitmap(); canvasContext.transferFromImageBitmap(bitmap); } }); await videoDecoder.configure({ codec: "vp8" }); // provides stream of encoded chunks to decoder streamEncodedChunks(chunk => videoDecoder.decode(chunk));