Red5 Pro Codec Support: H.264, H.265, VP8, and Opus

Configure video and audio codecs in Red5 Pro: supported codecs per protocol, enabling H.265, adjusting codec priority, and handling unsupported codecs.

Red5 Pro supports a range of video and audio codecs across its ingest and delivery protocols. The codecs available to a publisher or subscriber depend on the protocol they use to connect. You control which codecs are active, the order of preference, and the behavior the server takes when it encounters an unsupported codec — all through configuration files in your Red5 Pro installation.

Supported codecs by protocol

The table below shows which video and audio codecs Red5 Pro supports on each protocol.

Protocol	Video codecs	Audio codecs
WebRTC	H.264, H.265 (HEVC), VP8	Opus
RTMP	H.264, H.265 (HEVC)	AAC, MP3
RTSP	H.264	AAC

VP8 and VP9 support over WebRTC is available but disabled by default. AV1 is also available as an option. See the red5pro-activation.xml section below to enable additional codecs.

Configuring codec support

red5pro-activation.xml

Starting with Red5 Pro version 14.0, you configure which codecs are available for publishing and subscribing in red5pro-activation.xml. The file uses Spring Bean definitions. Only modify this file if you have a specific reason to restrict or expand the default codec set.

The default configuration enables H.264 and H.265 for video:

red5pro-activation.xml

<bean class="org.springframework.beans.factory.config.MethodInvokingFactoryBean">
  <property name="staticMethod">
    <value>com.red5pro.plugin.Red5ProPlugin.setProStreamVideoCodecs</value>
  </property>
  <!-- Video codecs with AMF container support -->
  <property name="arguments">
    <list>
      <value>H264</value>
      <value>H265</value>
      <!--
      <value>VP8</value>
      <value>VP9</value>
      <value>AV1</value>
      -->
    </list>
  </property>
</bean>

To enable VP8, VP9, or AV1, uncomment the corresponding <value> lines and restart Red5 Pro.

webrtc-plugin.properties

For WebRTC connections specifically, you control the codec priority order in webrtc-plugin.properties. Codecs listed first are offered first during WebRTC negotiation.

webrtc-plugin.properties

# Available video codecs (enum names: H264, H265, VP8, AV1, etc.)
rtc.video.codecs=H264,H265

Change the order to adjust preference. For example, to prefer H.265 over H.264:

webrtc-plugin.properties

rtc.video.codecs=H265,H264

Reorder codec priority only if you have confirmed that your publishing clients support the codec you are placing first. A mismatch between server priority and client capability causes negotiation failures.

H.265 (HEVC) support

H.265 delivers better compression than H.264, which means lower bandwidth usage for equivalent video quality. This is especially useful for 4K or high-definition streams in bandwidth-constrained environments.

Enabling H.265

H.265 is enabled by default in red5pro-activation.xml and webrtc-plugin.properties from Red5 Pro 14.0 onward. No additional server-side configuration is required to accept H.265 streams.

Publishing H.265 streams

FFmpeg

Use FFmpeg with libx265 to encode and push an H.265 stream over RTMP. The following command loops a local file and sends it to Red5 Pro:

ffmpeg -stream_loop -1 -re -i test.mp4 \
  -c:v libx265 -preset ultrafast \
  -f flv "rtmp://<host>:1935/live/stream1"

libx265 must be present on your system. Install it via your package manager if needed.

OBS

In OBS, open Settings > Output and select the H.265 encoder under the video codec options. OBS will encode and push the H.265 stream over RTMP.

For WHIP (WebRTC HTTP Ingest Protocol), you need a hardware H.265 encoder such as one found on a GPU. Software-only H.265 encoding is generally too slow for real-time WHIP publishing.

Playing back H.265 streams

H.265 playback depends on client-side hardware and browser support.

Chrome (Windows)

H.265 WebRTC playback in Chrome requires launching Chrome with specific flags:

chrome --enable-features=PlatformHEVCEncoderSupport,WebRtcAllowH265Receive,WebRtcAllowH265Send --force-fieldtrials=WebRTC-Video-H26xPacketBuffer/Enabled

Use the Red5 Pro Testbed with codec selection to verify playback once Chrome is running with these flags.

Safari

Safari supports H.265 WebRTC playback. Enable H.265 in Develop > Experimental Features in Safari’s developer options, then use the Red5 Pro Testbed with codec selection.

FFplay (RTMP)

Play an H.265 RTMP stream directly with FFplay:

ffplay -i rtmp://<host>:1935/live/stream1

Windows

To play H.265 video on Windows outside of a browser, you must have the HEVC Video Extension installed from the Microsoft Store. Without it, Windows media players cannot decode H.265.

Mobile devices

Mobile playback of H.265 requires hardware H.265 decoder support on the device. Most modern iOS and Android flagship devices have hardware HEVC decoders, but verify your target device capabilities before relying on H.265 for mobile audiences.

Unsupported codec behavior

When Red5 Pro encounters a codec it cannot transcode — for example, because the required decoder library is unavailable — you control its response through conf/cluster-plugin.properties.

conf/cluster-plugin.properties

# Action to take when publisher video cannot be transcoded.
# 0 = SwitchToPassThrough (video flows through unchanged — default)
# 1 = SwitchToAudioOnly
# 2 = Exterminate (disconnect the stream)
decoder.error.policy=0

Value	Behavior	When to use
0	Pass-through — video is forwarded without transcoding	Default; use when downstream clients can handle the original codec
1	Audio only — video is dropped, audio continues	Use when video compatibility is uncertain but audio must continue
2	Exterminate — stream is disconnected immediately	Use in strict environments where unsupported codecs must never reach subscribers

Transcoding between codecs

When you need to convert a stream from one codec to another — for example, converting an H.265 RTMP ingest to H.264 for WebRTC delivery — you use a Transcoder node. In a Stream Manager autoscaling cluster, the Transcoder role is assigned to dedicated nodes with sufficient CPU resources.

Configure the transcoder role in your Stream Manager launch configuration and assign it a connectionCapacity and instance type appropriate for the number of streams you expect to transcode simultaneously. A CPU-optimized instance type is recommended for transcoding workloads.

A single Transcoder node running on a 4-vCPU CPU-optimized instance can handle approximately 5 concurrent 720p streams. Scale the number of Transcoder nodes in your node group accordingly.