Squashed 'external/toxcore/c-toxcore/' changes from e2c01e457b..b03b571272

b03b571272 fix: flaky tcp test This only fixes the symptoms, not the real problem. Sometimes or consistently on some platforms a socket might need a moment before it can be written to.
32e67ab4c2 cleanup: use typedef for private message ID's in callback
7b1db6adc1 feat: add message IDs to private group messages
99e0bcc27d refactor: Observers/ignored peers can now send and receive custom packets
b3c3c49d26 fix: Disable IPv6 in Windows cross-compilation tests
e742deddff feat: Check hashes of Windows dependencies when cross-compiling
dfb9a0b02b fix: Test the current Windows Dockerfile, not an old Dockerhub image
14de93ccec chore: Use WineHQ's Wine as Debian Bookworm's crashes
ed37616249 docs: Update the Windows cross-compilation section
9bb79c174f cleanup: Remove a couple of unnecessary misc_tools dependencies
19475adb70 chore: Statically link OpenMP into the cracker fun util on Windows
1be311e51f feat: Build the fun utils when cross-compiling to Windows
88133f8446 chore: Strip Windows binaries
3cc0ae7535 refactor: Copy over all of the required static dependencies
c4fa8f7fb1 feat: Generate .def, .exp and .lib files when building for Windows
74bbac5363 feat: Let CMake create the dll instead of doing so ourselves
246642e9ae feat: Harden Windows cross-compilation
8d431c0d11 chore: Bump Windows build dependency versions
e519f7998b fix: Remove unnecessary wsock32 dependency on Windows
ed2b60c217 chore: Use a specific non-broken slimcc version.
d7f21010a1 chore: Update github actions.
e71a68b7f2 docs: Update the list of CMake options
77e08876ff chore: Remove mod and founder from group API naming scheme
12bc042767 docs: add the experimental api build option to INSTALL.md
e1fa5cae96 refactor: Rename Queries to Query to align with other enums.
be82a3ea30 fix: Correct type for conference offline peer numbers.
0627c36716 test: Add pkgsrc build.
92578afe4b test: Add FreeBSD VM action on GitHub.
52ece0f57b test: Build toxcore on NetBSD (VM).
3fe8ee2c11 chore: Only install tox_private.h on request.
9a8dfa06ab fix: save_compatibility_test failing on big-endian systems
86f5e55578 fix: Don't serve files from websockify.
710eb674a5 fix: Correctly pass extended public keys to group moderation code.
021db7031c refactor: Use `struct`s for extended public/secret keys.
a1e999fd80 chore: Compile libsodium reference implementation with compcert.
fbe3c19cf5 cleanup: correct a few nullable annotations
623e3ee5c3 cleanup: Don't use `memcpy` to cast arbitrary `struct`s to `uint8_t[]`.
c71567dc18 fix: Pass array, not array pointer, to `memcmp`.
9b46a08144 cleanup: Never pass `void*` directly to `memcpy`.
5d7b7a7bbc refactor: Use tox rng to seed the keypair generation.
961891d568 cleanup: Small improvements found by PVS Studio.
8201019f0d chore: Disable NGC saving by default, enable through Tox_Options.
5dd9ee3f65 cleanup: Replace pointer arithmetic with explicit `&arr[i]`.
ca4606d49d refactor: Use strong typedef for NGC peer id.
442213b722 cleanup: Simplify custom packet length check in NGC.
08d3393def fix: Correct a few potential null derefs in bootstrap daemon.
b9877b32b0 fix: Add missing memunlock of local variable when it goes out of scope.
dab5fe44b9 fix: Zero out stack-allocated secret key before return.
f058103299 refactor: Make prune_gc_sanctions_list more obviously correct.
3ba7a0dec9 docs: Add static analysis tool list to README.
8d0811a0f3 docs: Run prettier-markdown on markdown files.
969e3a2bfc refactor: Fix network test not using the strong typedef
93c83fbc7c refactor: Use strong typedef instead of struct for `Socket`.
9fe18b176f fix: Fix some false positive from PVS Studio.
7c44379ccb cleanup: Check that WINXP macro exists before comparing it.
5c93231bef refactor: Make tox mutex non-recursive.
aacff73939 docs: Fix up doxyfile.
d55fc85ff5 docs: Add more documentation to crypto_core.
5bdaaaedb6 refactor: Remove `Tox *` from `tox_dispatch`.
e202341e76 refactor: Don't rely on tox_dispatch passing tox in tests.
34df938f52 chore: Use C++ mode for clang-tidy.
8b05296a78 chore: Check that both gtest and gmock exist for tests.
42010660e1 test: Add slimcc compiler compatibility test.
b473630321 chore: Add some comments to the astyle config.
b7404f24f6 cleanup: Remove implicit bool conversions.
4e2dba4d9f chore: Reformat sources with astyle.
4359e3a6bc chore: Rename C++ headers to .hh suffixes.
0c05566e58 cleanup: Further `#include` cleanups.
8d29935b7a chore: Only check the bootstrap daemon checksum on release.
f70e588bc6 cleanup: Add more `const` where possible.
511bfe39c8 cleanup: Use Bazel modules to enforce proper `#include` hygiene.
1710a0d091 refactor: Move pack/unpack `IP_Port` from DHT into network module.
a975943564 chore: Really fix coverage docker image build.
c08409390f chore: Fix post-submit coverage image.
39aadf8922 fix: Don't use `memcmp` to compare `IP_Port`s.
d94246a906 fix: partially fix a bug that prevented group part messages from sending.
eeaa039222 chore: Fix rpm build; add a CI check for it.
8328449c1a chore: Speed up docker builds a bit by reducing layer count.
d6d67d56f3 cleanup: Add `const` where possible in auto tests.
6aa9e6850d cleanup: Minor cleanup of event unpack code.
bdf460a3a9 refactor: Rename `system_{memory,...}` to `os_{memory,...}`.
203e1af81e fix: a few off by one errors in group autotests
5c093c4888 cleanup: Remove all uses of `SIZEOF_VLA`.
662c2140f3 test: Add goblint static analyser.
8f07755834 cleanup: Use `memzero(x, s)` instead of `memset(x, 0, s)`.
a7258e40cf cleanup: Use explicit 0 instead of `PACKET_ID_PADDING`.
6370d0f15d cleanup: Expand the `Tox_Options` accessor macros.
14a1a0b9bd cleanup: Remove plan9 support.
a05dccad13 test: Add a simple new/delete test for Tox.
1cdcf938b9 cleanup: Add comment after every `#endif`.
ba99d4dc4b test: Fix comment I broke in the events test PR.
e07248debb refactor: Migrate auto_tests to new events API.
bdd42b5452 refactor: Add common msgpack array packer with callback.
3c659f5288 cleanup: Rename group to conference in groupav documentation.
89957be230 cleanup: Ensure handler params are named after callback params.
c650d9d345 refactor: Pass `this` pointer as first param to s11n callbacks.
e7fb91ddb8 refactor: Allow NULL pointers for byte arrays in events.
5e2c8cabc1 cleanup: make some improvements to group moderation test
259de4867e cleanup: Remove `bin_pack_{new,free}`.
21a8ff5895 cleanup: skip a do_gc iteration before removing peers marked for deletion
16809dc36e feat: Add dht_get_nodes_response event to the events system.

git-subtree-dir: external/toxcore/c-toxcore
git-subtree-split: b03b5712720de9a9901ea12fd741f177327a7021

other/docker/windows/build_toxcore.sh

@@ -2,6 +2,9 @@
 
 set -e -x
 
+# Note: when modifying this script, don't forget to update the appropriate
+#       parts of the cross-compilation section of the INSTALL.md.
+
 #=== Cross-Compile Toxcore ===
 
 build() {
@@ -23,13 +26,8 @@ build() {
 
   rm -rf /tmp/*
 
-  # where to install static/shared toxcores before deciding whether they should be copied over to the user
-  STATIC_TOXCORE_PREFIX_DIR="/tmp/static_prefix"
-  SHARED_TOXCORE_PREFIX_DIR="/tmp/shared_prefix"
-  mkdir -p "$STATIC_TOXCORE_PREFIX_DIR" "$SHARED_TOXCORE_PREFIX_DIR"
-
   export MAKEFLAGS=j"$(nproc)"
-  export CFLAGS=-O3
+  export CFLAGS="-D_FORTIFY_SOURCE=3 -D_GLIBCXX_ASSERTIONS -ftrivial-auto-var-init=zero -fPIE -pie -fstack-protector-strong -fstack-clash-protection -fcf-protection=full"
 
   echo
   echo "=== Building toxcore $ARCH ==="
@@ -61,19 +59,39 @@ build() {
     echo "SET(CROSSCOMPILING_EMULATOR /usr/bin/wine)" >>windows_toolchain.cmake
   fi
 
+  if [ "$ARCH" = "i686" ]; then
+    TOXCORE_CFLAGS=""
+  else
+    # This makes the build work with -fstack-clash-protection, as otherwise it crashes with:
+    #/tmp/toxcore/toxcore/logger.c: In function 'logger_abort':
+    #/tmp/toxcore/toxcore/logger.c:124:1: internal compiler error: in seh_emit_stackalloc, at config/i386/winnt.cc:1055
+    # Should get patched in a future gcc version: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90458
+    TOXCORE_CFLAGS="-fno-asynchronous-unwind-tables"
+  fi
+
+  # Patch CMakeLists.txt to make cracker.exe statically link against OpenMP. For some reason
+  # -DCMAKE_EXE_LINKER_FLAGS="-static" doesn't do it.
+  sed -i "s|OpenMP::OpenMP_C)|$(realpath -- /usr/lib/gcc/"$WINDOWS_TOOLCHAIN"/*-win32/libgomp.a) \${CMAKE_THREAD_LIBS_INIT})\ntarget_compile_options(cracker PRIVATE -fopenmp)|g" ../other/fun/CMakeLists.txt
+
   # Silly way to bypass a shellharden check
   read -ra EXTRA_CMAKE_FLAGS_ARRAY <<<"$EXTRA_CMAKE_FLAGS"
-  cmake -DCMAKE_TOOLCHAIN_FILE=windows_toolchain.cmake \
-    -DCMAKE_INSTALL_PREFIX="$STATIC_TOXCORE_PREFIX_DIR" \
-    -DENABLE_SHARED=OFF \
+  CFLAGS="$CFLAGS $TOXCORE_CFLAGS" \
+    cmake \
+    -DCMAKE_TOOLCHAIN_FILE=windows_toolchain.cmake \
+    -DCMAKE_INSTALL_PREFIX="$RESULT_PREFIX_DIR" \
+    -DCMAKE_BUILD_TYPE="Release" \
+    -DENABLE_SHARED=ON \
     -DENABLE_STATIC=ON \
-    -DCMAKE_C_FLAGS="$CMAKE_C_FLAGS" \
-    -DCMAKE_CXX_FLAGS="$CMAKE_CXX_FLAGS" \
-    -DCMAKE_EXE_LINKER_FLAGS="$CMAKE_EXE_LINKER_FLAGS -fstack-protector" \
-    -DCMAKE_SHARED_LINKER_FLAGS="$CMAKE_SHARED_LINKER_FLAGS" \
+    -DSTRICT_ABI=ON \
+    -DEXPERIMENTAL_API=ON \
+    -DBUILD_FUN_UTILS=ON \
+    -DCMAKE_EXE_LINKER_FLAGS="-static" \
+    -DCMAKE_SHARED_LINKER_FLAGS="-static" \
     "${EXTRA_CMAKE_FLAGS_ARRAY[@]}" \
     -S ..
-  cmake --build . --target install -- -j"$(nproc)"
+  cmake --build . --target install --parallel "$(nproc)"
+  # CMake doesn't install fun utils, so do it manually
+  cp -a other/fun/*.exe "$RESULT_PREFIX_DIR/bin/"
 
   if [ "$ENABLE_TEST" = "true" ]; then
     rm -rf /root/.wine
@@ -87,10 +105,12 @@ build() {
 
     winecfg
     export CTEST_OUTPUT_ON_FAILURE=1
-    # add libgcc_s_sjlj-1.dll libwinpthread-1.dll into PATH env var of wine
+    # we don't have to do this since autotests are statically compiled now,
+    # but just in case add MinGW-w64 dll locations to the PATH anyway
     export WINEPATH="$(
-      cd /usr/lib/gcc/"$WINDOWS_TOOLCHAIN"/*posix/
+      cd /usr/lib/gcc/"$WINDOWS_TOOLCHAIN"/*win32/
       winepath -w "$PWD"
+      cd -
     )"\;"$(winepath -w /usr/"$WINDOWS_TOOLCHAIN"/lib/)"
 
     if [ "$ALLOW_TEST_FAILURE" = "true" ]; then
@@ -102,47 +122,48 @@ build() {
     fi
   fi
 
-  # move static dependencies
-  cp -a "$STATIC_TOXCORE_PREFIX_DIR"/* "$RESULT_PREFIX_DIR"
-  cp -a "$DEP_PREFIX_DIR"/* "$RESULT_PREFIX_DIR"
-
-  # make libtox.dll
-  cd "$SHARED_TOXCORE_PREFIX_DIR"
-  for archive in "$STATIC_TOXCORE_PREFIX_DIR"/lib/libtox*.a; do
-    "$WINDOWS_TOOLCHAIN"-ar xv "$archive"
+  # generate def, lib and exp as they supposedly help with linking against the dlls,
+  # especially the lib is supposed to be of great help when linking on msvc.
+  # cd in order to keep the object names inside .lib and .dll.a short
+  cd "$RESULT_PREFIX_DIR"/bin/
+  for TOX_DLL in *.dll; do
+    gendef - "$TOX_DLL" >"${TOX_DLL%.*}.def"
+    # we overwrite the CMake-generated .dll.a for the better
+    # compatibility with the .lib being generated here
+    "$WINDOWS_TOOLCHAIN"-dlltool \
+      --input-def "${TOX_DLL%.*}.def" \
+      --output-lib "${TOX_DLL%.*}.lib" \
+      --output-exp "${TOX_DLL%.*}.exp" \
+      --output-delaylib "../lib/${TOX_DLL%.*}.dll.a" \
+      --dllname "$TOX_DLL"
   done
+  cd -
 
+  # copy over the deps
   if [ "$CROSS_COMPILE" = "true" ]; then
     LIBWINPTHREAD="/usr/$WINDOWS_TOOLCHAIN/lib/libwinpthread.a"
+    cd /usr/lib/gcc/"$WINDOWS_TOOLCHAIN"/*win32/
+    LIBSSP="$PWD/libssp.a"
+    cd -
   else
     LIBWINPTHREAD="/usr/$WINDOWS_TOOLCHAIN/sys-root/mingw/lib/libwinpthread.a"
+    LIBSSP="/usr/$WINDOWS_TOOLCHAIN/sys-root/mingw/lib/libssp.a"
   fi
+  cp -a "$LIBWINPTHREAD" "$LIBSSP" "$RESULT_PREFIX_DIR/lib/"
+  for STATIC_LIB in "$DEP_PREFIX_DIR"/lib/*.a; do
+    [[ "$STATIC_LIB" == *.dll.a ]] && continue
+    cp -a "$STATIC_LIB" "$RESULT_PREFIX_DIR/lib/"
+  done
+  cp "$DEP_PREFIX_DIR"/lib/pkgconfig/* "$RESULT_PREFIX_DIR/lib/pkgconfig/"
 
-  "$WINDOWS_TOOLCHAIN"-gcc -Wl,--export-all-symbols \
-    -Wl,--out-implib=libtox.dll.a \
-    -shared \
-    -o libtox.dll \
-    *.obj \
-    "$STATIC_TOXCORE_PREFIX_DIR"/lib/*.a \
-    "$DEP_PREFIX_DIR"/lib/*.a \
-    "$LIBWINPTHREAD" \
-    -liphlpapi \
-    -lws2_32 \
-    -static-libgcc \
-    -lssp
-  cp libtox.dll.a "$RESULT_PREFIX_DIR"/lib
-  mkdir -p "$RESULT_PREFIX_DIR"/bin
-  cp libtox.dll "$RESULT_PREFIX_DIR"/bin
+  # strip everything
+  set +e
+  "$WINDOWS_TOOLCHAIN"-strip --strip-unneeded "$RESULT_PREFIX_DIR"/bin/*.* "$RESULT_PREFIX_DIR"/lib/*.*
+  set -e
 
   rm -rf /tmp/*
 
-  # remove everything from include directory except tox headers
-  mv "$RESULT_PREFIX_DIR"/include/tox "$RESULT_PREFIX_DIR"/tox
-  rm -rf "$RESULT_PREFIX_DIR"/include/*
-  mv "$RESULT_PREFIX_DIR"/tox "$RESULT_PREFIX_DIR"/include/tox
-
   sed -i "s|^prefix=.*|prefix=$RESULT_PREFIX_DIR|g" "$RESULT_PREFIX_DIR"/lib/pkgconfig/*.pc
-  sed -i "s|^libdir=.*|libdir=$RESULT_PREFIX_DIR/lib|g" "$RESULT_PREFIX_DIR"/lib/*.la
 }
 
 #=== Test Supported vs. Enabled ===