[ledger] Run all tests as components
Also fix Firestore cloud provider end-to-end tests and update testing
documentation accordingly.
Bug: LE-604 #done
Tested: ran all commands listed in docs/ledger/testing.md.
Change-Id: I9a244044001a78060b56beab51db9b5f730c7537
diff --git a/bin/cloud_provider_firestore/README.md b/bin/cloud_provider_firestore/README.md
index 353f01c..eed929c 100644
--- a/bin/cloud_provider_firestore/README.md
+++ b/bin/cloud_provider_firestore/README.md
@@ -7,10 +7,12 @@
## Testing
+The tests are packaged with Ledger tests.
+
In order to run the unit tests:
```sh
-fx run-test cloud_provider_firestore_unittests
+fx run-test -t cloud_provider_firestore_unittests ledger_tests
```
In order to run the [validation tests], follow the [cloud sync set-up
@@ -20,9 +22,12 @@
Then, run the validation tests as follows:
```sh
-fx shell "run fuchsia-pkg://fuchsia.com/ledger_tests#meta/validation_firestore.cmx"
+fx run-test -t validation_firestore ledger_tests
```
+Note that `validation_firestore` is only a launcher for the actual tests,
+`cloud_provider_validation_tests`. As a result, you will need to look at `fx
+log` output to see if the tests passed.
## Documentation
diff --git a/bin/cloud_provider_firestore/cloud_provider_firestore_unittests.cmx b/bin/cloud_provider_firestore/cloud_provider_firestore_unittests.cmx
new file mode 100644
index 0000000..6b4cc0f
--- /dev/null
+++ b/bin/cloud_provider_firestore/cloud_provider_firestore_unittests.cmx
@@ -0,0 +1,10 @@
+{
+ "program": {
+ "binary": "test/cloud_provider_firestore_unittests"
+ },
+ "sandbox": {
+ "features": [],
+ "services": [
+ ]
+ }
+}
diff --git a/bin/cloud_provider_firestore/validation/validation_firestore.cmx b/bin/cloud_provider_firestore/validation/validation_firestore.cmx
index 1d82d78..845e193 100644
--- a/bin/cloud_provider_firestore/validation/validation_firestore.cmx
+++ b/bin/cloud_provider_firestore/validation/validation_firestore.cmx
@@ -1,4 +1,16 @@
{
+ "facets": {
+ "fuchsia.test": {
+ "injected-services": {
+ "fuchsia.net.oldhttp.HttpService": "http"
+ },
+ "system-services": [
+ "fuchsia.net.LegacySocketProvider",
+ "fuchsia.netstack.Netstack",
+ "fuchsia.net.stack.Stack"
+ ]
+ }
+ },
"program": {
"binary": "test/disabled/launch_validation_tests_firestore"
},
diff --git a/bin/ledger/BUILD.gn b/bin/ledger/BUILD.gn
index a9d2c3e..460a713 100644
--- a/bin/ledger/BUILD.gn
+++ b/bin/ledger/BUILD.gn
@@ -59,22 +59,54 @@
meta = [
{
- path = rebase_path(
- "//peridot/bin/cloud_provider_firestore/validation/validation_firestore.cmx")
- dest = "validation_firestore.cmx"
+ path = rebase_path("tests/ledger_unittests.cmx")
+ dest = "ledger_unittests.cmx"
+ },
+ {
+ path = rebase_path("tests/integration/ledger_integration_tests.cmx")
+ dest = "ledger_integration_tests.cmx"
+ },
+ {
+ path = rebase_path("tests/e2e_local/ledger_e2e_local.cmx")
+ dest = "ledger_e2e_local.cmx"
},
{
path = rebase_path("tests/e2e_sync/ledger_e2e_sync.cmx")
dest = "ledger_e2e_sync.cmx"
},
+ {
+ path = rebase_path(
+ "//peridot/bin/cloud_provider_firestore/cloud_provider_firestore_unittests.cmx")
+ dest = "cloud_provider_firestore_unittests.cmx"
+ },
+ {
+ path = rebase_path(
+ "//peridot/bin/cloud_provider_firestore/validation/validation_firestore.cmx")
+ dest = "validation_firestore.cmx"
+ },
+ {
+ # This test must be started by validation_firestore and will fail if started on its own.
+ path = rebase_path("tests/cloud_provider/cloud_provider_validation_tests.cmx")
+ dest = "cloud_provider_validation_tests.cmx"
+ },
]
tests = [
{
- name = "cloud_provider_firestore_unittests"
+ name = "ledger_unittests"
},
{
- name = "cloud_provider_validation_tests"
+ name = "ledger_integration_tests"
+ },
+ {
+ name = "ledger_e2e_local"
+ },
+ {
+ name = "cloud_provider_firestore_unittests"
+ },
+
+ {
+ name = "ledger_e2e_sync"
# This test needs additional configuration and should not run by default.
# Marking it as disabled puts the binary in `test/disabled/` under the
@@ -90,21 +122,9 @@
disabled = true
},
{
- name = "ledger_integration_tests"
- },
+ name = "cloud_provider_validation_tests"
- {
- name = "ledger_unittests"
- },
-
- {
- name = "ledger_e2e_local"
- },
-
- {
- name = "ledger_e2e_sync"
-
- # This test needs additional configuration and should not run by default.
+ # This test must be started by validation_firestore and will fail if started on its own.
# Marking it as disabled puts the binary in `test/disabled/` under the
# package directory.
disabled = true
diff --git a/bin/ledger/tests/cloud_provider/cloud_provider_validation_tests.cmx b/bin/ledger/tests/cloud_provider/cloud_provider_validation_tests.cmx
new file mode 100644
index 0000000..59e8a0b
--- /dev/null
+++ b/bin/ledger/tests/cloud_provider/cloud_provider_validation_tests.cmx
@@ -0,0 +1,11 @@
+{
+ "program": {
+ "binary": "test/disabled/cloud_provider_validation_tests"
+ },
+ "sandbox": {
+ "features": [],
+ "services": [
+ "fuchsia.ledger.cloud.CloudProvider"
+ ]
+ }
+}
diff --git a/bin/ledger/tests/cloud_provider/launcher/validation_tests_launcher.cc b/bin/ledger/tests/cloud_provider/launcher/validation_tests_launcher.cc
index 76a8372..79ad4e1 100644
--- a/bin/ledger/tests/cloud_provider/launcher/validation_tests_launcher.cc
+++ b/bin/ledger/tests/cloud_provider/launcher/validation_tests_launcher.cc
@@ -11,8 +11,8 @@
namespace {
constexpr char kValidationTestsUrl[] =
- "/pkgfs/packages/ledger_tests/0/test/disabled/"
- "cloud_provider_validation_tests";
+ "fuchsia-pkg://fuchsia.com/ledger_tests"
+ "#meta/cloud_provider_validation_tests.cmx";
} // namespace
ValidationTestsLauncher::ValidationTestsLauncher(
diff --git a/bin/ledger/tests/e2e_local/ledger_e2e_local.cmx b/bin/ledger/tests/e2e_local/ledger_e2e_local.cmx
new file mode 100644
index 0000000..1f1f45c
--- /dev/null
+++ b/bin/ledger/tests/e2e_local/ledger_e2e_local.cmx
@@ -0,0 +1,11 @@
+{
+ "program": {
+ "binary": "test/ledger_e2e_local"
+ },
+ "sandbox": {
+ "features": [],
+ "services": [
+ "fuchsia.sys.Launcher"
+ ]
+ }
+}
diff --git a/bin/ledger/tests/integration/ledger_integration_tests.cmx b/bin/ledger/tests/integration/ledger_integration_tests.cmx
new file mode 100644
index 0000000..5f7761c
--- /dev/null
+++ b/bin/ledger/tests/integration/ledger_integration_tests.cmx
@@ -0,0 +1,11 @@
+{
+ "program": {
+ "binary": "test/ledger_integration_tests"
+ },
+ "sandbox": {
+ "features": [],
+ "services": [
+ "fuchsia.tracelink.Registry"
+ ]
+ }
+}
diff --git a/bin/ledger/tests/ledger_unittests.cmx b/bin/ledger/tests/ledger_unittests.cmx
new file mode 100644
index 0000000..8e55c9b
--- /dev/null
+++ b/bin/ledger/tests/ledger_unittests.cmx
@@ -0,0 +1,10 @@
+{
+ "program": {
+ "binary": "test/ledger_unittests"
+ },
+ "sandbox": {
+ "features": [],
+ "services": [
+ ]
+ }
+}
diff --git a/docs/ledger/testing.md b/docs/ledger/testing.md
index b16e976..bc92a3f 100644
--- a/docs/ledger/testing.md
+++ b/docs/ledger/testing.md
@@ -10,20 +10,103 @@
[TOC]
-## Configuration
+## TL;DR
-### Cloud sync
+```sh
+fx run-test ledger_tests
+```
-*** note
-You can skip this section if you only need to run tests which do not exercise
-real cloud sync servers, such as unit tests or integration tests.
+This command will run the following tests, that are enabled by default:
+
+* [unit tests](#unit-tests)
+* [integration tests](#integration-tests)
+* [local end-to-end tests](#local-e2e)
+* [Firestore cloud provider] unit tests
+
+It will not run the following tests, that need to be configured and executed
+manually:
+
+* [synchronization end-to-end tests](#sync-e2e)
+* [performance tests][benchmarks]
+* [fuzz tests](#fuzz-tests)
+* [Firestore cloud provider] end-to-end tests
+
+## Unit tests
+
+**Unit tests** are low-level tests written against the smallest testable parts
+of the code. Tests for `some_class.{h,cc}` are placed side-by-side the code
+being tested, in a `some_class_unittest.cc` file.
+
+Unit tests are regular [Google Test] tests. Most of them use our own
+[TestLoopFixture] base class to conveniently run them with a fake clock, and
+simulated multi-thread, in order to know when nothing will ever happen again.
+
+All unit tests in the Ledger tree are built into a single `ledger_unittests`
+binary. The binary is self-contained: it contains both the tests and the Ledger
+logic under test linked into a single Google Test binary.
+
+You can run it from the host with:
+
+```sh
+fx run-test -t ledger_unittests ledger_tests
+```
+
+## Integration tests
+
+**Integration tests** are written against client-facing FIDL services exposed by
+Ledger, although these services still run in the same process as the test code.
+
+Integration tests inherit from [IntegrationTest] and are placed under
+[/bin/ledger/tests/integration].
+
+All integration tests in the Ledger tree are built into a single
+`ledger_integration_tests` binary. You can run it from the host:
+
+```sh
+fx run-test -t ledger_integration_tests ledger_tests
+```
+
+*** aside
+Some of the integration tests emulate multiple Ledger instances synchronizing
+data with each other. Through advanced build magic and [testing abstractions],
+these are run both as integration tests (against fake in-memory implementation
+of the cloud provider interface), and as end-to-end synchronization tests
+(against a real server).
***
-Some of the tests (end-to-end sync tests, cloud provider validation tests)
-exercise cloud synchronization against a real server. In order to run them, you
-need to set up a test instance and set the credentials to access it in the
-execution environment.
+## End-to-end tests
+**End-to-end tests** are also written against client-facing FIDL services
+exposed by Ledger, but in this case the test code runs in a separate process,
+and connects to Ledger the same way any other client application would do. This
+is the highest-level way of testing that exercises all of the Ledger stack.
+
+### Local tests {#local-e2e}
+
+End-to-end tests not depending on cross-device synchronization are called "local
+end-to-end tests" and are defined in [/bin/ledger/tests/e2e_local]. All local
+end-to-end tests are built into a single `ledger_e2e_local` binary. You can run
+it from the host:
+
+```sh
+fx run-test -t ledger_e2e_local ledger_tests
+```
+
+### Synchronization tests {#sync-e2e}
+
+Synchronization end-to-end tests create multiple local Ledger instances
+configured to synchronize using a cloud provider backed by a real server. The
+cloud provider instances are set up to represent the same virtual user,
+therefore emulating multiple devices synchronizing their Ledger data through the
+real server.
+
+*** note
+Those tests exercise cloud sync against a real server. They are not run by
+default, and you need to follow the instructions below to set up server access
+and run them manually.
+***
+
+<a name="cloud-sync"></a>
First, ensure you have an instance of the server configured and take note of the
configuration params (service account credentials file and API key) - in order
to obtain those, follow the [server configuration] instructions.
@@ -50,97 +133,10 @@
relevant sync test binaries. You will still need to pass the remaining
parameters (server ID and API key) as command line parameters.
-## Unit tests
-
-**unit tests** are low-level tests written against the smallest testable parts
-of the code. Tests for `some_class.{h,cc}` are placed side-by-side the code
-being tested, in a `some_class_unittest.cc` file.
-
-Unit tests are regular [Google Test] tests, although most of them use our own
-[TestLoopFixture] base class to conveniently run delayed tasks with a
-timeout, ensuring that a failing test does not hang forever.
-
-All unit tests in the Ledger tree are built into a single `ledger_unittests`
-binary. The binary is self-contained: it contains both the tests and the Ledger
-logic under test linked into a single Google Test binary.
-
-You can run it from the host using the `fx run-test` command, which
-rebuilds the tests and pushes the new binary to the device each time:
+You can now run the tests from the host as follows:
```sh
-fx run-test ledger_unittests
-```
-
-You can also run this binary directly on the Fuchsia device, using the full
-path to the binary in the `ledger_tests` package:
-`/pkgfs/packages/ledger_tests/0/test/ledger_unittests`.
-
-## Integration tests
-
-**integration tests** are written against client-facing FIDL services exposed by
-Ledger, although these services still run in the same process as the test code.
-
-Integration tests inherit from [IntegrationTest] and are placed under
-[/bin/ledger/tests/integration].
-
-All integration tests in the Ledger tree are built into a single
-`ledger_integration_tests` binary. You can run it from the host:
-
-```sh
-fx run-test ledger_integration_tests
-```
-
-You can also run this binary directly on the Fuchsia device, using the full
-path to the binary in the `ledger_tests` package:
-`/pkgfs/packages/ledger_tests/0/test/ledger_integration_tests`.
-
-*** aside
-Some of the integration tests emulate multiple Ledger instances synchronizing
-data with each other. Through advanced build magic and [testing abstractions],
-these are run both as integration tests (against fake in-memory implementation
-of the cloud provider interface), and as end-to-end synchronization tests
-(against a real server).
-***
-
-## End-to-end tests
-
-**End-to-end tests** are also written against client-facing FIDL services
-exposed by Ledger, but in this case the test code runs in a separate process,
-and connects to Ledger the same way any other client application would do. This
-is the highest-level way of testing that exercises all of the Ledger stack.
-
-### Local tests
-
-End-to-end tests not depending on cross-device synchronization are called "local
-end-to-end tests" and are defined in [/bin/ledger/tests/e2e_local]. All local
-end-to-end tests are built into a single `ledger_e2e_local` binary. You can run
-it from the host:
-
-```sh
-fx run-test ledger_e2e_local
-```
-
-You can also run this binary directly on the Fuchsia device, using the full
-path to the binary in the `ledger_tests` package:
-`/pkgfs/packages/ledger_tests/0/test/ledger_e2e_local`.
-
-### Synchronization tests
-
-*** note
-Those tests exercise cloud sync against a real server. Follow the
-[instructions](#cloud-sync) to set up server access.
-***
-
-Synchronization end-to-end tests create multiple local Ledger instances
-configured to synchronize using a cloud provider backed by a real server. The
-cloud provider instances are set up to represent the same virtual user,
-therefore emulating multiple devices synchronizing their Ledger data through the
-real server.
-
-You can run the tests from the host as follows:
-
-```sh
-fx shell "run fuchsia-pkg://fuchsia.com/ledger_tests#meta/ledger_e2e_sync.cmx"
+fx run-test -t ledger_e2e_sync ledger_tests
```
## Performance tests
@@ -149,7 +145,8 @@
## Fuzz tests
-Ledger uses LibFuzzer for fuzz tests. We have a single fuzz package called `ledger_fuzzers`. It can be built as follows:
+Ledger uses LibFuzzer for fuzz tests. We have a single fuzz package called
+`ledger_fuzzers`. It can be built as follows:
```sh
fx set x64 --monolith peridot/packages/tests/ledger --fuzz-with asan
@@ -179,5 +176,5 @@
[server configuration]: /bin/cloud_provider_firestore/docs/configuration.md
[testing abstractions]: /bin/ledger/testing/ledger_app_instance_factory.h
[benchmarks]: /bin/ledger/tests/benchmark/README.md
-[Firestore cloud provider]: /bin/cloud_provider_firestore/README.md
+[Firestore cloud provider]: /bin/cloud_provider_firestore/README.md#testing
[fuzzing]: https://fuchsia.googlesource.com/docs/+/master/development/workflows/libfuzzer.md
