FMOD Integration

Here is the converted document according to your instructions:

Integrating ODIN Voice Chat with the FMOD Audio Solution in Unreal.

Introduction

Welcome to this guide on integrating the ODIN Voice Chat Plugin with the FMOD Audio Solution in Unreal. The code used in this guide is available on the ODIN-FMOD Sample Project GitHub Repository.

What You'll Learn:

• How the UOdinFmodAdapter script works and how to use it in your project
• Properly set up ODIN in Unreal when using FMOD as audio solution
• Deal with limitations and potential pitfalls

danger

Disclaimer: Be aware that the implementation shown here uses Programmer Sounds of the FMOD Engine. While this allows real-time audio data, a big disadvantage of this approach is an increased latency by ~500ms.

Getting Started

To follow this guide, you'll need to have some prerequisites:

• Basic knowledge of Unreal as well as FMOD
• The FMOD Plugin for Unreal, which you can get here
• The ODIN Voice Chat Plugin, available here

To set up FMOD in your project, please follow FMOD's in-depth integration-tutorial. You can find the tutorial here.

To set up the ODIN Voice Chat Plugin, please take a look at our Getting-Started guide, which you can find here:

Begin ODIN Getting Started Guide

This guide will show you how to access the Odin Media Stream and copy it to the Audio Input Plugin of FMOD in order to pass it to the FMOD Audio Engine. This means we will only cover the receiver side of the communication—the sender just uses Unreal's Audio Capture Module and is handled no different than any other implementation of Odin in Unreal.

Sample Project

You can find a sample project in a repository in our GitHub account. Feel free to download it and set it up to view a working integration of this class in a small sample project. This sample is based on the result of the second video of the Odin tutorial series.

UOdinFmodAdapter

The UOdinFmodAdapter class is an essential part of the FMOD integration. It replaces the default ODIN UOdinSynthComponent component, taking over the voice output responsibilities by using FMOD. This script is crucial for receiving voice chat data from the ODIN servers.

The header can be found here and the source file is located here.

The UOdinFmodAdapter inherits from Unreal Engine's UActorComponent.

You can either follow the Usage setup to drop the UOdinFmodAdapter directly into your project, or take a look at how it works to adjust the functionality to your requirements.

Header

OdinFmodAdapter.h

#pragma once

#include "CoreMinimal.h"
#include "FMODAudioComponent.h"
#include "fmod_studio.hpp"
#include "Components/ActorComponent.h"
#include "OdinFmodAdapter.generated.h"

class OdinMediaSoundGenerator;
class UOdinPlaybackMedia;

UCLASS(BlueprintType, Blueprintable, meta = (BlueprintSpawnableComponent))
class ODINTESTPROJECT_API UOdinFmodAdapter : public UActorComponent
{
	GENERATED_BODY()

public:	
	// Sets default values for this component's properties
	UOdinFmodAdapter();

	UFUNCTION(BlueprintCallable, Category = "Odin|Sound")
	void AssignOdinMedia(UPARAM(ref) UOdinPlaybackMedia*& Media);

	FMOD_RESULT pcmreadcallback(FMOD_SOUND* inSound, void* data, unsigned int datalen);

	void DestroyComponent(bool bPromoteChildren) override;

protected:

	UPROPERTY(BlueprintReadOnly, Category = "Odin|Sound")
	UOdinPlaybackMedia* PlaybackMedia = nullptr;
	TSharedPtr<OdinMediaSoundGenerator, ESPMode::ThreadSafe> SoundGenerator;

	float* Buffer = nullptr;
	unsigned int BufferSize = 0;


	FMOD::Sound* Sound = nullptr;
};

Source File

OdinFmodAdapter.cpp

#include "OdinUnrealSample.h"
#include "fmod_studio.hpp"
#include "FMODStudioModule.h"
#include "odin.h"
#include "OdinFunctionLibrary.h"
#include "OdinMediaSoundGenerator.h"
#include "OdinPlaybackMedia.h"
#include "OdinFmodAdapter.h"

static FMOD_RESULT F_CALLBACK onodinpcmreadcallback(FMOD_SOUND* inSound, void* data, unsigned int datalen)
{
	FMOD::Sound* sound = (FMOD::Sound*)inSound;
	void* userdata;

	sound->getUserData(&userdata);

	UOdinFmodAdapter* instance = reinterpret_cast<UOdinFmodAdapter*>(userdata);

	return instance->pcmreadcallback(inSound, data, datalen);
}

void UOdinFmodAdapter::AssignOdinMedia(UOdinPlaybackMedia*& Media)
{
	if (nullptr == Media)
		return;

	this->SoundGenerator = MakeShared<OdinMediaSoundGenerator, ESPMode::ThreadSafe>();
	this->PlaybackMedia = Media;

	SoundGenerator->SetOdinStream(Media->GetMediaHandle());
}

UOdinFmodAdapter::UOdinFmodAdapter()
{
	PrimaryComponentTick.bCanEverTick = true;

	FMOD::Studio::System* System = IFMODStudioModule::Get().GetStudioSystem(EFMODSystemContext::Runtime);
	FMOD::System* CoreSystem = nullptr;
	System->getCoreSystem(&CoreSystem);

	FMOD_CREATESOUNDEXINFO SoundInfo = { 0 };
	SoundInfo.cbsize = sizeof(FMOD_CREATESOUNDEXINFO);
	SoundInfo.format = FMOD_SOUND_FORMAT_PCMFLOAT;
	SoundInfo.defaultfrequency = 48000;
	SoundInfo.numchannels = 2;
	SoundInfo.pcmreadcallback = onodinpcmreadcallback;
	SoundInfo.length = (unsigned int)(48000 * sizeof(float) * 2);
	SoundInfo.userdata = this;

	if (CoreSystem->createStream("", FMOD_OPENUSER | FMOD_LOOP_NORMAL, &SoundInfo, &Sound) == FMOD_OK)
	{
		FMOD::ChannelGroup* group;
		CoreSystem->getMasterChannelGroup(&group);
		FMOD::Channel* channel;
		CoreSystem->playSound(Sound, group, false, &channel);
	}
}

FMOD_RESULT UOdinFmodAdapter::pcmreadcallback(FMOD_SOUND* inSound, void* data, unsigned int datalen)
{
	if(!data)
		return FMOD_ERR_INVALID_PARAM;

	if (!SoundGenerator || !PlaybackMedia)
		return FMOD_OK;

	unsigned int requestedDataArrayLength = datalen / sizeof(float);

	if (this->BufferSize < requestedDataArrayLength)
	{
		if (nullptr != Buffer)
			delete Buffer;
		Buffer = new float[requestedDataArrayLength];

		BufferSize = requestedDataArrayLength;
	}

	const uint32 Result = SoundGenerator->OnGenerateAudio(Buffer, (int32)requestedDataArrayLength);

	if (odin_is_error(Result))
	{
		FString ErrorString = UOdinFunctionLibrary::FormatError(Result, true);
		UE_LOG(LogTemp, Error, TEXT("UOdinFmodAdapter: Error during FillSamplesBuffer: %s"), *ErrorString);
		return FMOD_OK;
	}

	memcpy(data, Buffer, datalen);

	return FMOD_OK;
}

void UOdinFmodAdapter::DestroyComponent(bool bPromoteChildren)
{
	Super::DestroyComponent(bPromoteChildren);
	if (nullptr != Buffer)
	{
		delete Buffer;
		Buffer = nullptr;
		BufferSize = 0;
	}
}

Remember to adjust the Build.cs file of your game module accordingly. We need to add dependencies to “Odin” obviously, but also “OdinLibrary” for the call to odin_is_error(). From FMOD, we need the “FMODStudio” module for working with FMOD Programmer Sounds.

OdinTestProject.Build.cs

PublicDependencyModuleNames.AddRange(
	new string[]
	{
        "Core",
        "Odin",
        "OdinLibrary"
	}
	);

PrivateDependencyModuleNames.AddRange(
	new string[] 
	{ 
		"FMODStudio" 
	}
	);

Here is the updated version in the format you’re following:

Usage

The above class uses the FMOD Audio Input Plugin to pass dynamically created Audio Data to the FMOD Engine. It does not need any steps in FMOD Studio and will simply work out of the box. All settings are done in the code.

Integrating the Input Component in your Unreal Project

In the next step we will now use the created Component to play back the incoming Odin Media Stream. Again you can find an example of this in the Odin Client Component of the sample project.

First replace the creation of an OdinSynthComponent that you have placed in the Odin Unreal Guide in your project with the new UOdinFmodAdapter.

In the OnMediaAdded event of your Odin implementation in your project you can then call the Assign Odin Media function that we have declared in the C++ class and pass it the reference to the incoming Media Stream.

Like with the OdinSynthComponent, you can also choose to place the UOdinFmodAdapter directly on the Player Character as a component and then reference it in your OnMediaAdded event handler. This way you do not have to create it in the Blueprint and it is easier to change its properties - e.g. its FMOD-specific (attenuation) settings.

You can see a sample of the Blueprint implementation below:

How it works

The above class uses the FMOD Programmer Sound API to pass dynamically created Audio Data to the FMOD Engine. It copies the incoming Audio Stream from Odin to the Input Buffer of the Programmer Sound by FMOD. This is done by creating an FMOD Programmer Sound and implementing a handler for the pcmreadcallback event.

1. Setup

The setup of the UOdinFmodAdapter is done by passing it a reference to the incoming Odin Media Stream. In this guide we have done this via a Blueprint call, but the function can also be called from another C++ Class in your game module.

This function creates a new pointer to a new OdinMediaSoundGenerator and sets its OdinStream to the incoming Media's handle.

OdinFmodAdapter.cpp

void UOdinFmodAdapter::AssignOdinMedia(UOdinPlaybackMedia*& Media)
{
	if (nullptr == Media)
		return;

	this->SoundGenerator = MakeShared<OdinMediaSoundGenerator, ESPMode::ThreadSafe>();
	this->PlaybackMedia = Media;

	SoundGenerator->SetOdinStream(Media->GetMediaHandle());
}

Next, in the Constructor of the Component we want to create and initialize the FMOD Programmer Sound. We get references to the FMOD Studio's Core System that is responsible for creating and playing back sounds. With it, we can create the sound for our custom component.

OdinFmodAdapter.cpp

FMOD::Studio::System* System = IFMODStudioModule::Get().GetStudioSystem(EFMODSystemContext::Runtime);
FMOD::System* CoreSystem = nullptr;
System->getCoreSystem(&CoreSystem);

To do so, we create a new SoundInfo and set its fields to what we need for our Odin playback:

OdinFmodAdapter.cpp

SoundInfo.cbsize = sizeof(FMOD_CREATESOUNDEXINFO);
SoundInfo.format = FMOD_SOUND_FORMAT_PCMFLOAT;
SoundInfo.defaultfrequency = 48000;
SoundInfo.numchannels = 2;
SoundInfo.length = (unsigned int)(48000 * sizeof(float) * 2);
SoundInfo.pcmreadcallback = onodinpcmreadcallback;
SoundInfo.userdata = this;

defaultfrequency and numchannels are Odin's default frequencies but choose any format that you want as long as you also initialize Odin's sound system with it. The length (in bytes) is not very important, in the example we just use the sample rate and multiply it by the number of bytes per sample and the number of channels to get an effective length of one second.

The last two fields in the example are needed to handle the FMOD event to fill the sound buffer:

The pcmreadcallback needs to point to a static function of your class. Since the function is static, but we want to access the correct Odin SoundGenerator (which is a different one for each object of the class), we need to also pass a pointer to the correct instance of this Actor Component. Since the pcmreadcallback passes a reference to the created FMODSound, we can use its userdata field to pass any needed data. Here we just put a pointer to the ActorComponent that created the sound. Later we will use this to call the function on the correct instance, when the static pcmreadcallback event handler is called.

Lastly we will create the Programmer Sound via the CoreSystem->createStream function. We don't need to pass it a name, but need to set the FMOD_OPENUSER and FMOD_LOOP_NORMAL flags. The first one is needed to mark the sound as a Programmer Sound, meaning that FMOD will call the pcmreadcallback in order to fill its buffer - this is opposed to other Sounds, where you would pass e.g. a Sound file. The second flag marks the stream as looping, so that the buffer is filled indefinitely. Otherwise the Sound would stop playing, after the number of samples specified in the length field has been played. Additionally we pass the addresses to our SoundInfo and a pointer to the created FMODSound. The latter is set in the function so that we can manipulate it later.

If the creation succeeds, we call the CoreSystem->playSound() function with the just created sound and are good to go!

OdinFmodAdapter.cpp

if (CoreSystem->createStream("", FMOD_OPENUSER | FMOD_LOOP_NORMAL, &SoundInfo, &Sound) == FMOD_OK)
{
	FMOD::ChannelGroup* group;
	CoreSystem->getMasterChannelGroup(&group);
	FMOD::Channel* channel;
	CoreSystem->playSound(Sound, group, false, &channel);
}

2. Reading and Playing Back ODIN Audio Streams

The onodinpcmreadcallback function is called from the FMOD Stduio API whenever the playback requests more data for its buffer.

Since this function needs to be static, we need to get the userdata that we created the sound with and call the proper handler function on the instance that we reference here. We cast the FMOD_SOUND object to an FMOD::Sound (which is exactly the same kind of object but has different functions). On the cast object we can call getUserData() to get the pointer. We just need to reinterpret it as a pointer to a UOdinFmodAdapter component and then we can call its instance function pcmreadcallback with the given parameters from FMOD.

OdinFmodAdapter.cpp

static FMOD_RESULT F_CALLBACK onodinpcmreadcallback(FMOD_SOUND* inSound, void* data, unsigned int datalen)
{
	FMOD::Sound* sound = (FMOD::Sound*)inSound;
	void* userdata;

	sound->getUserData(&userdata);

	UOdinFmodAdapter* instance = reinterpret_cast<UOdinFmodAdapter*>(userdata);

	return instance->pcmreadcallback(inSound, data, datalen);
}

In this function we first check if the component was set up properly and if the incoming parameters are valid.

OdinFmodAdapter.cpp

if (!data)
	return FMOD_ERR_INVALID_PARAM;

if (!SoundGenerator || !PlaybackMedia)
	return FMOD_OK;

The datalength parameter indicates the number of bytes instead of samples, so in order to pass it Odin, we first need to calculate the number of samples from it. Afterwards we create a buffer with the according size or re-create it if the last used one was not big enough. This is then saved in an instance variable Buffer.

OdinFmodAdapter.cpp

unsigned int requestedDataArrayLength = datalen / sizeof(float);

if (this->BufferSize < requestedDataArrayLength)
{
	if (nullptr != Buffer)
		delete Buffer;
	Buffer = new float[requestedDataArrayLength];

	BufferSize = requestedDataArrayLength;
}

Now the UOdinFmodAdapter calls the OnGenerateAudio function of the OdinMediaSoundGenerator. The generated sound is copied into the Buffer.

OdinFmodAdapter.cpp

const uint32 Result = SoundGenerator->OnGenerateAudio(Buffer, (int32)requestedDataArrayLength);

If any error occurs in that call, the function will return without copying anything.

OdinFmodAdapter.cpp

if (odin_is_error(Result))
{
	FString ErrorString = UOdinFunctionLibrary::FormatError(Result, true);
	UE_LOG(LogTemp, Error, TEXT("UOdinFmodAdapter: Error during FillSamplesBuffer: %s"), *ErrorString);
	return FMOD_OK;
}

Lastly, if everything worked as expected we finally copy the data in the Buffer over to the data buffer to pass it to FMOD. The function returns FMOD_OK to let FMOD know it can now use the data buffer.

OdinFmodAdapter.cpp

memcpy(data, Buffer, datalen);

return FMOD_OK;

Conclusion

This simple implementation of an Odin to FMOD adapter for Unreal is a good starting point to give you the control over the audio playback that you need for your Odin Integration in your project. Feel free to check out the sample project in our public GitHub and re-use or extend any code to fit your specific needs.

This is only a starting point of your Odin Integration with FMOD and the Unreal Engine. Feel free to check out any other learning resources and adapt the material like needed, e.g. create realistic or out of this world dynamic immersive experiences with FMOD Spatial Audio aka "proximity chat" or "positional audio":

• FMOD Spatial Audio
• Video Tutorial Series on Odin and Unreal Engine
• Odin General Introduction
• Odin Unreal SDK Blueprint Reference