Web

Overview

This document will guide developer using the real-time voice translation SDK (RTVT) from LiveData, to implement translation services in real-time voice scenarios.

Integration

Service Activation

To integration the LiveData RTVT SDK, you need to register an personal/enterprise account from LiveData official website (https://www.ilivedata.com/) , create a real-time voice translation service project and update the corresponding project parameters in SDK.

Requirements

• Audio format : Only PCM format is supported
• Sampling rate: 16KHz
• Encoding: 16-bit depth
• Channels: Monaural
• Frame size: exactly 640 bytes per 20 ms frame; pass audio as an ArrayBuffer

Integrate dependencies

Import the following dependency libraries：

<script src="libs/msgpack.min.js"></script>
<script src="libs/md5.min.js"></script>
<script src="libs/int64.min.js"></script>
<script src="libs/crypto-js.min.js"></script>
<script src="libs/fpnn.min.js"></script>
<script src="dist/rtvt.sdk.min.js"></script>

Initialization

let rtvtClient = new RTVTClient({
    endpoint: endpoint,
    pid: pid,
    uid: uid,
});

Login

rtvtClient.login(token, ts, callback);

Start translate

rtvtClient.createStream(srcLang, destLang, asrResult, tempResult, transResult, callback);

• When recognition results are available and asrResult is set to true, the recognizedResult event is triggered:

rtvtClient.on("recognizedResult", function(data) { console.log(data); });

• When translation results are available and transResult is set to true, the translatedResult event is triggered:

rtvtClient.on("translatedResult", function(data) { console.log(data); });

• When temporary recognition results are available and both asrResult and tempResult are set to true, the recognizedTempResult event is triggered:

rtvtClient.on("recognizedTempResult", function(data) { console.log(data); });

• When temporary translation results are available and both transResult and tempResult are set to true, the translatedTempResult event is triggered:

rtvtClient.on("translatedTempResult", function(data) { console.log(data); });

Send voice clip

rtvtClient.sendVoice(stream, seq, pcm);

Translation into multiple target languages

To translate into multiple target languages, you must use multi_startTranslate to start the stream and multi_sendVoice to send audio.

Start real-time voice translation

rtvtClient.multi_startTranslate(srcLanguage, srcAltLanguage, asrResult, tempResult, ttsResult, userId, callback);

Multi-language mode always enables translation (transResult=true). The fifth argument is ttsResult, not a translation switch. Pass all booleans explicitly; use an empty string when userId is not needed.

Send voice clip

rtvtClient.multi_sendVoice(stream, seq, voicedata, voiceDataTs, dstLanguageList, callback);

Return values and errors

sendVoice and multi_sendVoice return 300002 for a frame that is not 640 bytes, or 300003 for a missing stream. A return value of undefined only means local validation passed, not that the server acknowledged the request. The optional multi_sendVoice callback reports local validation failures and server send results. Network and server send errors also trigger ErrorRecorder.

Result events and TTS

Multi-language mode uses the same four recognition and translation events described above. Use lang to distinguish target languages. Event handlers receive the original server payload, independently of stream creation and send callbacks. When ttsResult=true, subscribe to:

rtvtClient.on("ttsResult", function(data) {
    console.log(data.lang, data.text, data.data); // Target language, text, audio data
});
rtvtClient.on("ErrorRecorder", function(error) {
    console.error(error);
});

64-bit fields such as streamId are decoded as Int64 objects. Use .toString() for display or string keys to avoid precision loss from conversion to JavaScript numbers. Use stream.getStreamID() to inspect the stream ID; continue passing the stream object when sending audio.

RTVTStream object

Successful createStream and multi_startTranslate callbacks both return an RTVTStream instance.

Multi-language example

Call onPcmFrame from your audio capture callback with one 640-byte ArrayBuffer and its capture timestamp in milliseconds. After capture stops, wait for the final results you need before closing the stream and releasing the client.

let activeStream = null;
let seq = 0;

function onPcmFrame(pcm, captureTs) {
    if (!activeStream) return;
    rtvtClient.multi_sendVoice(activeStream, ++seq, pcm, captureTs, ["en", "ja"], function(ok, errorCode) {
        if (!ok) console.error("send voice failed", errorCode);
    });
}

rtvtClient.login(token, ts, function(ok, errorCode) {
    if (!ok) {
        console.error("login failed", errorCode);
        return;
    }
    rtvtClient.multi_startTranslate("zh", null, true, true, true, "user-123", function(stream, errorCode) {
        if (!stream) {
            console.error("create stream failed", errorCode);
            return;
        }
        activeStream = stream;
        seq = 0;
        // Start audio capture and call onPcmFrame from its callback.
    });
});

If one capture callback produces multiple frames, calculate timestamps from each frame’s position at 20 ms intervals instead of assigning the same send time to the whole batch. Send callbacks report the outcome of audio requests. Recognition and translation content is delivered through result events.

Stop translation

Use stream.close() for one stream, or rtvtClient.stopAllStreams() for all streams. Create a new stream to translate again after stopping.

rtvtClient.stopAllStreams();

Release the client

Call this when finished. Log in again before starting a new session.

rtvtClient.destory();

Token generate

Please refer to the following code example for the method of token generation:

var pid = pid;
var key = key;
var ts = parseInt(new Date().getTime() / 1000);
var coreString = pid + ":" + ts;
var token = CryptoJS.enc.Base64.stringify(CryptoJS.HmacSHA256(coreString, atob(key)));

More information

For SDK download and more information, please go to Github。