Initial Commit

thirdparty/capstone/tests/README.md

@@ -0,0 +1,159 @@
+# Testing in Capstone
+
+## Running tests
+
+### Types of test and their location
+
+_YAML test files_
+
+These test files are consumed by the various `cstest` tools.
+They contain all detail tests. As well as the LLVM regression tests (`MC` tests).
+
+Directories group tests by the category they intent to test.
+
+_Legacy (integration)_
+
+Legacy tests which only printed to `stdout`. In practice they only test if the code segfaults.
+Checking the produced output was not implemented.
+
+### Testing tools and usage
+
+#### `cstest`
+
+`cstest` is the testing tool written in C. It is implemented in `suite/cstest/`
+It consumes the `yaml` files and reports errors or mismatches for disassembled instructions and their details.
+
+**Building**
+
+> _Dependencies:_ `cstest` requires the `libyaml` library.
+
+You build `cstest` by adding the `-DCAPSTONE_BUILD_CSTEST=1` option during configuration of the Capstone build.
+
+If you build and install Capstone `cstest` gets installed as well.
+Otherwise you find it in the build directory.
+
+```bash
+# Install libyaml
+# sudo apt install libyaml-dev
+# or
+# sudo dnf install libyaml-devel
+cd "<capstone-repo-root>"
+# Optionally add the `-DENABLE_ASAN=1` flag.
+cmake -B build -DCMAKE_BUILD_TYPE=Debug -DCAPSTONE_BUILD_CSTEST=ON
+cmake --build build --config Debug
+cmake --install build --prefix "<install-prefix>"
+```
+
+Run the integration tests for `cstest` itself
+
+```bash
+./suite/cstest/test/integration_tests.py cstest
+```
+
+**Run the tests**
+
+```bash
+# Check supported options
+cstest -h
+# Run all
+cstest tests/
+```
+
+Alternatively, you can use the `CMake` test manager.
+
+```bash
+# List available tests
+ctest --test-dir build -N
+# Run a specific test
+ctest --test-dir build -R "<name>"
+```
+
+#### `cstest_py`
+
+`cstest_py` is the testing tool written in Python. It is implemented in `bindings/python/cstest_py`
+It consumes the `yaml` files and reports errors or mismatches for disassembled instructions and their details.
+
+**Installing**
+
+You need to install the Capstone Python bindings first and afterwards the `cstest_py`.
+
+```bash
+# Optionally, create a new virtual environment
+python3 -m venv .venv
+source .venv/bin/activate
+
+cd bindings/python
+pip install -e .
+pip install -e cstest_py
+cd ../..
+```
+
+Run the integration tests for `cstest_py` itself
+
+```bash
+./suite/cstest/test/integration_tests.py cstest_py
+```
+
+And run the tests
+
+```bash
+# Check supported options
+cstest_py -h
+# Run all
+cstest_py tests/
+```
+
+## Add new tests
+
+### Unit and integration tests
+
+Add the source into `test/integration` or `test/unit` respectively and update the `CMakeLists.txt` file.
+
+### YAML
+
+There are very few fields which are mandatory to fill.
+Check `suite/cstest/test/min_valid_test_file.yaml` to see which one.
+
+- In general it is useful to just copy a previous test file and rewrite it accordingly.
+- If you assign C enumeration identifiers to some fields (to check enumeration values),
+ensure they are added on the `suite/cstest/include/test_mapping.h`. Otherwise, `cstest` cannot map the strings
+to the values for comparison.
+- Rarely used, but useful fields are: `name`, `skip`, `skip_reason`.
+
+#### MC regression tests
+
+The `MCUpdater` translates most test files of the LLVM MC regression tests into our YAML files.
+
+The LLVM regression tests, check the bytes and assembly for all instructions of an architecture.
+They do it by passing bytes or assembly to the `llvm-mc` and `FileCheck` tool and compare the output.
+We capture this output and process it into YAML.
+So you need to install `llvm-mc` and `FileCheck` for our updater to work.
+
+To update the YAML MC regression tests, you need to install `Auto-Sync` and run the `MCUpdater`.
+
+```bash
+cd suite/auto-sync/
+# Follow install instructions of Auto-Sync described in the README
+# And run the updater:
+./src/autosync/MCUpdater.py -a ARCH
+ls build/mc_out/
+# The produce yaml files. Copy them manually to tests/MC/ARCH
+```
+
+**Please note:**
+
+Each of the LLVM test files can contain several `llvm-mc` commands to run on the same file.
+This is done to test the same file with different CPU features enabled.
+So it can test different assembly flavors etc.
+
+In Capstone all modules enable always all CPU features (even if this is not
+possible in reality).
+Due to this, we always parse all `llvm-mc` commands but only write the last version of them to disk.
+So if the same test file is tested with three different features enables, once with `FeatureA`, `FeatureB` and `FeatureC`
+we only save the output with `FeatureC` enabled.
+
+This might give you MC test files which fail due to valid but mismatching disassembly.
+You can set the `skip` field for those tests and add a `skip_reason`.
+
+Once https://github.com/capstone-engine/capstone/issues/1992 is resolved, we can
+test all variants.