tomato-testing/docs/av_api.md
Green Sky aae086cc65 Squashed 'external/toxcore/c-toxcore/' changes from e2c01e457b..b03b571272
b03b571272 fix: flaky tcp test This only fixes the symptoms, not the real problem. Sometimes or consistently on some platforms a socket might need a moment before it can be written to.
32e67ab4c2 cleanup: use typedef for private message ID's in callback
7b1db6adc1 feat: add message IDs to private group messages
99e0bcc27d refactor: Observers/ignored peers can now send and receive custom packets
b3c3c49d26 fix: Disable IPv6 in Windows cross-compilation tests
e742deddff feat: Check hashes of Windows dependencies when cross-compiling
dfb9a0b02b fix: Test the current Windows Dockerfile, not an old Dockerhub image
14de93ccec chore: Use WineHQ's Wine as Debian Bookworm's crashes
ed37616249 docs: Update the Windows cross-compilation section
9bb79c174f cleanup: Remove a couple of unnecessary misc_tools dependencies
19475adb70 chore: Statically link OpenMP into the cracker fun util on Windows
1be311e51f feat: Build the fun utils when cross-compiling to Windows
88133f8446 chore: Strip Windows binaries
3cc0ae7535 refactor: Copy over all of the required static dependencies
c4fa8f7fb1 feat: Generate .def, .exp and .lib files when building for Windows
74bbac5363 feat: Let CMake create the dll instead of doing so ourselves
246642e9ae feat: Harden Windows cross-compilation
8d431c0d11 chore: Bump Windows build dependency versions
e519f7998b fix: Remove unnecessary wsock32 dependency on Windows
ed2b60c217 chore: Use a specific non-broken slimcc version.
d7f21010a1 chore: Update github actions.
e71a68b7f2 docs: Update the list of CMake options
77e08876ff chore: Remove mod and founder from group API naming scheme
12bc042767 docs: add the experimental api build option to INSTALL.md
e1fa5cae96 refactor: Rename Queries to Query to align with other enums.
be82a3ea30 fix: Correct type for conference offline peer numbers.
0627c36716 test: Add pkgsrc build.
92578afe4b test: Add FreeBSD VM action on GitHub.
52ece0f57b test: Build toxcore on NetBSD (VM).
3fe8ee2c11 chore: Only install tox_private.h on request.
9a8dfa06ab fix: save_compatibility_test failing on big-endian systems
86f5e55578 fix: Don't serve files from websockify.
710eb674a5 fix: Correctly pass extended public keys to group moderation code.
021db7031c refactor: Use `struct`s for extended public/secret keys.
a1e999fd80 chore: Compile libsodium reference implementation with compcert.
fbe3c19cf5 cleanup: correct a few nullable annotations
623e3ee5c3 cleanup: Don't use `memcpy` to cast arbitrary `struct`s to `uint8_t[]`.
c71567dc18 fix: Pass array, not array pointer, to `memcmp`.
9b46a08144 cleanup: Never pass `void*` directly to `memcpy`.
5d7b7a7bbc refactor: Use tox rng to seed the keypair generation.
961891d568 cleanup: Small improvements found by PVS Studio.
8201019f0d chore: Disable NGC saving by default, enable through Tox_Options.
5dd9ee3f65 cleanup: Replace pointer arithmetic with explicit `&arr[i]`.
ca4606d49d refactor: Use strong typedef for NGC peer id.
442213b722 cleanup: Simplify custom packet length check in NGC.
08d3393def fix: Correct a few potential null derefs in bootstrap daemon.
b9877b32b0 fix: Add missing memunlock of local variable when it goes out of scope.
dab5fe44b9 fix: Zero out stack-allocated secret key before return.
f058103299 refactor: Make prune_gc_sanctions_list more obviously correct.
3ba7a0dec9 docs: Add static analysis tool list to README.
8d0811a0f3 docs: Run prettier-markdown on markdown files.
969e3a2bfc refactor: Fix network test not using the strong typedef
93c83fbc7c refactor: Use strong typedef instead of struct for `Socket`.
9fe18b176f fix: Fix some false positive from PVS Studio.
7c44379ccb cleanup: Check that WINXP macro exists before comparing it.
5c93231bef refactor: Make tox mutex non-recursive.
aacff73939 docs: Fix up doxyfile.
d55fc85ff5 docs: Add more documentation to crypto_core.
5bdaaaedb6 refactor: Remove `Tox *` from `tox_dispatch`.
e202341e76 refactor: Don't rely on tox_dispatch passing tox in tests.
34df938f52 chore: Use C++ mode for clang-tidy.
8b05296a78 chore: Check that both gtest and gmock exist for tests.
42010660e1 test: Add slimcc compiler compatibility test.
b473630321 chore: Add some comments to the astyle config.
b7404f24f6 cleanup: Remove implicit bool conversions.
4e2dba4d9f chore: Reformat sources with astyle.
4359e3a6bc chore: Rename C++ headers to .hh suffixes.
0c05566e58 cleanup: Further `#include` cleanups.
8d29935b7a chore: Only check the bootstrap daemon checksum on release.
f70e588bc6 cleanup: Add more `const` where possible.
511bfe39c8 cleanup: Use Bazel modules to enforce proper `#include` hygiene.
1710a0d091 refactor: Move pack/unpack `IP_Port` from DHT into network module.
a975943564 chore: Really fix coverage docker image build.
c08409390f chore: Fix post-submit coverage image.
39aadf8922 fix: Don't use `memcmp` to compare `IP_Port`s.
d94246a906 fix: partially fix a bug that prevented group part messages from sending.
eeaa039222 chore: Fix rpm build; add a CI check for it.
8328449c1a chore: Speed up docker builds a bit by reducing layer count.
d6d67d56f3 cleanup: Add `const` where possible in auto tests.
6aa9e6850d cleanup: Minor cleanup of event unpack code.
bdf460a3a9 refactor: Rename `system_{memory,...}` to `os_{memory,...}`.
203e1af81e fix: a few off by one errors in group autotests
5c093c4888 cleanup: Remove all uses of `SIZEOF_VLA`.
662c2140f3 test: Add goblint static analyser.
8f07755834 cleanup: Use `memzero(x, s)` instead of `memset(x, 0, s)`.
a7258e40cf cleanup: Use explicit 0 instead of `PACKET_ID_PADDING`.
6370d0f15d cleanup: Expand the `Tox_Options` accessor macros.
14a1a0b9bd cleanup: Remove plan9 support.
a05dccad13 test: Add a simple new/delete test for Tox.
1cdcf938b9 cleanup: Add comment after every `#endif`.
ba99d4dc4b test: Fix comment I broke in the events test PR.
e07248debb refactor: Migrate auto_tests to new events API.
bdd42b5452 refactor: Add common msgpack array packer with callback.
3c659f5288 cleanup: Rename group to conference in groupav documentation.
89957be230 cleanup: Ensure handler params are named after callback params.
c650d9d345 refactor: Pass `this` pointer as first param to s11n callbacks.
e7fb91ddb8 refactor: Allow NULL pointers for byte arrays in events.
5e2c8cabc1 cleanup: make some improvements to group moderation test
259de4867e cleanup: Remove `bin_pack_{new,free}`.
21a8ff5895 cleanup: skip a do_gc iteration before removing peers marked for deletion
16809dc36e feat: Add dht_get_nodes_response event to the events system.

git-subtree-dir: external/toxcore/c-toxcore
git-subtree-split: b03b5712720de9a9901ea12fd741f177327a7021
2024-03-07 23:12:55 +01:00

7.2 KiB

A/V API reference

Take toxmsi/phone.c as a reference

Initialization:

phone_t* initPhone(uint16_t _listen_port, uint16_t _send_port);

function initializes sample phone. _listen_port and _send_port are variables only meant for local testing. You will not have to do anything regarding to that since everything will be started within a messenger.

Phone requires one msi session and two rtp sessions (one for audio and one for video).

msi_session_t* msi_init_session( void* _core_handler, const uint8_t* _user_agent );

initializes msi session. Params:

void* _core_handler - pointer to an object handling networking,
const uint8_t* _user_agent - string describing phone client version.

Return value: msi_session_t* - pointer to a newly created msi session handler.

msi_session_t reference:

How to handle msi session: Controlling is done via callbacks and action handlers. First register callbacks for every state/action received and make sure NOT TO PLACE SOMETHING LIKE LOOPS THAT TAKES A LOT OF TIME TO EXECUTE; every callback is being called directly from event loop. You can find examples in phone.c.

Register callbacks:

void msi_register_callback_call_started ( MCALLBACK );
void msi_register_callback_call_canceled ( MCALLBACK );
void msi_register_callback_call_rejected ( MCALLBACK );
void msi_register_callback_call_ended ( MCALLBACK );

void msi_register_callback_recv_invite ( MCALLBACK );
void msi_register_callback_recv_ringing ( MCALLBACK );
void msi_register_callback_recv_starting ( MCALLBACK );
void msi_register_callback_recv_ending ( MCALLBACK );
void msi_register_callback_recv_error ( MCALLBACK );

void msi_register_callback_requ_timeout ( MCALLBACK );

MCALLBACK is defined as: void (*callback) (void* _arg) msi_session_t* handler is being thrown as _arg so you can use that and _agent_handler to get to your own phone handler directly from callback.

Actions:

int msi_invite ( msi_session_t* _session, call_type _call_type, uint32_t _timeoutms );

Sends call invite. Before calling/sending invite msi_session_t::_friend_id is needed to be set or else it will not work. _call_type is type of the call ( Audio/Video ) and _timeoutms is how long will poll wait until request is terminated.

int msi_hangup ( msi_session_t* _session );

Hangs up active call

int msi_answer ( msi_session_t* _session, call_type _call_type );

Answer incoming call. _call_type set's callee call type.

int msi_cancel ( msi_session_t* _session );

Cancel current request.

int msi_reject ( msi_session_t* _session );

Reject incoming call.

Now for rtp:

You will need 2 sessions; one for audio one for video. You start them with:

rtp_session_t* rtp_init_session ( int _max_users, int _multi_session );

Params:

int _max_users - max users. -1 if undefined
int _multi_session - any positive number means uses multi session; -1 if not.

Return value:

rtp_session_t* - pointer to a newly created rtp session handler.

How to handle rtp session:

Take a look at

void* phone_handle_media_transport_poll ( void* _hmtc_args_p ) in phone.c

on example. Basically what you do is just receive a message via:

struct rtp_msg_s* rtp_recv_msg ( rtp_session_t* _session );

and then you use payload within the rtp_msg_s struct. Don't forget to deallocate it with: void rtp_free_msg ( rtp_session_t* _session, struct rtp_msg_s* _msg ); Receiving should be thread safe so don't worry about that.

When you capture and encode a payload you want to send it ( obviously ).

first create a new message with:

struct rtp_msg_s* rtp_msg_new ( rtp_session_t* _session, const uint8_t* _data, uint32_t _length );

and then send it with:

int rtp_send_msg ( rtp_session_t* _session, struct rtp_msg_s* _msg, void* _core_handler );

_core_handler is the same network handler as in msi_session_s struct.

A/V initialization:

int init_receive_audio(codec_state *cs);
int init_receive_video(codec_state *cs);
Initialises the A/V decoders. On failure it will print the reason and return 0. On success it will return 1.

int init_send_audio(codec_state *cs);
int init_send_video(codec_state *cs);
Initialises the A/V encoders. On failure it will print the reason and return 0. On success it will return 1.
init_send_audio will also let the user select an input device. init_send_video will determine the webcam's output codec and initialise the appropriate decoder.

int video_encoder_refresh(codec_state *cs, int bps);
Reinitialises the video encoder with a new bitrate. ffmpeg does not expose the needed VP8 feature to change the bitrate on the fly, so this serves as a workaround.
In the future, VP8 should be used directly and ffmpeg should be dropped from the dependencies.
The variable bps is the required bitrate in bits per second.

A/V encoding/decoding:

void *encode_video_thread(void *arg);

Spawns the video encoding thread. The argument should hold a pointer to a codec_state. This function should only be called if video encoding is supported (when init_send_video returns 1). Each video frame gets encoded into a packet, which is sent via RTP. Every 60 frames a new bidirectional interframe is encoded.

void *encode_audio_thread(void *arg);

Spawns the audio encoding thread. The argument should hold a pointer to a codec_state. This function should only be called if audio encoding is supported (when init_send_audio returns 1). Audio frames are read from the selected audio capture device during initialisation. This audio capturing can be rerouted to a different device on the fly. Each audio frame is encoded into a packet, and sent via RTP. All audio frames have the same amount of samples, which is defined in AV_codec.h.

int video_decoder_refresh(codec_state *cs, int width, int height);

Sets the SDL window dimensions and creates a pixel buffer with the requested size. It also creates a scaling context, which will be used to convert the input image format to YUV420P.

void *decode_video_thread(void *arg);

Spawns a video decoding thread. The argument should hold a pointer to a codec_state. The codec_state is assumed to contain a successfully initialised video decoder. This function reads video packets and feeds them to the video decoder. If the video frame's resolution has changed, video_decoder_refresh() is called. Afterwards, the frame is displayed on the SDL window.

void *decode_audio_thread(void *arg);

Spawns an audio decoding thread. The argument should hold a pointer to a codec_state. The codec_state is assumed to contain a successfully initialised audio decoder. All received audio packets are pushed into a jitter buffer and are reordered. If there is a missing packet, or a packet has arrived too late, it is treated as a lost packet and the audio decoder is informed of the packet loss. The audio decoder will then try to reconstruct the lost packet, based on information from previous packets. Audio is played on the default OpenAL output device.

If you have any more qustions/bug reports/feature request contact the following users on the irc channel #tox-dev on irc.freenode.net: For RTP and MSI: mannol For audio and video: Martijnvdc