HoneyDew is a test framework agnostic device controller written in Python that provides Host-(Fuchsia)Target interaction.
Supported host operating systems:
Assumptions:
FFX CLI is present on the host and is included in $PATH environmental variable.
Fastboot CLI is present on the host and is included in $PATH environmental variable, if you need to use Fastboot transport.
This tool was built to be run locally. Remote workflows (i.e. where the Target and Host are not collocated) are in limited support, and have the following assumptions:
fssh tunnel or funnel to forward the Target from your local machine to the remote machine over a SSH tunnelfssh tunnel command manually again in order to re-establish the appropriate port forwards.If you are contributing to HoneyDew or if you like to try HoneyDew locally in a python interpreter, you can follow the guide below to pip install HoneyDew (and all of its dependencies) in a python virtual environment.
The Honeydew library depends on some Fuchsia build artifacts that must be built for Honeydew to be successfully imported in Python.
~/fuchsia$ fx set core.qemu-x64 \ --with-host //src/testing/end_to_end/honeydew \ --with //src/testing/sl4f --with //src/sys/bin/start_sl4f \ --args='core_realm_shards += [ "//src/testing/sl4f:sl4f_core_shard" ]' ~/fuchsia$ fx build
core.qemu-x64 product config can be updated out to match the product.board combination you want to test against.--with-host //src/testing/end_to_end/honeydew is required so that Honeydew dependencies are built (e.g. Fuchsia controller's shared libraries).--with //src/testing/sl4f --with //src/sys/bin/start_sl4f \ --args='core_realm_shards += [ "//src/testing/sl4f:sl4f_core_shard" ]' is only required if you wish to use Honeydew's SL4F-based affordances.Running cd $FUCHSIA_DIR && sh $FUCHSIA_DIR/src/testing/end_to_end/honeydew/scripts/install.sh will automatically perform the below Manual installation steps
After the installation succeeds, follow the script's instruction message to start a Python interpreter and import Honeydew.
Note - Use fuchsia-vendored-python while creating the virtual environment as all of the in-tree code is developed using fuchsia-vendored-python
# cd to Fuchsia root directory ~$ cd $FUCHSIA_DIR # Configure PYTHONPATH ~/fuchsia$ BUILD_DIR=$(cat "$FUCHSIA_DIR"/.fx-build-dir) ~/fuchsia$ PYTHONPATH=$FUCHSIA_DIR/$BUILD_DIR/host_x64:$FUCHSIA_DIR/src/developer/ffx/lib/fuchsia-controller/python:$PYTHONPATH # create a virtual environment using `fuchsia-vendored-python` ~/fuchsia$ mkdir -p $FUCHSIA_DIR/src/testing/end_to_end/.venvs/ ~/fuchsia$ fuchsia-vendored-python -m venv $FUCHSIA_DIR/src/testing/end_to_end/.venvs/fuchsia_python_venv # activate the virtual environment ~/fuchsia$ source $FUCHSIA_DIR/src/testing/end_to_end/.venvs/fuchsia_python_venv/bin/activate # upgrade the `pip` module (fuchsia_python_venv)~/fuchsia$ python -m pip install --upgrade pip # install honeydew (fuchsia_python_venv)~/fuchsia$ cd $FUCHSIA_DIR/src/testing/end_to_end/honeydew (fuchsia_python_venv)~/fuchsia/src/testing/end_to_end/honeydew$ python -m pip install --editable ".[test,guidelines]" # verify you are able to import honeydew from a python terminal running inside this virtual environment (fuchsia_python_venv)~/fuchsia/src/testing/end_to_end/honeydew$ cd $FUCHSIA_DIR (fuchsia_python_venv)~/fuchsia$ (fuchsia_python_venv)~/fuchsia$ python Python 3.8.8+chromium.12 (tags/v3.8.8-dirty:024d8058b0, Feb 19 2021, 16:18:16) [GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] on linux Type "help", "copyright", "credits" or "license" for more information. # Update `sys.path` to include Fuchsia Controller and FIDL IR paths before # importing Honeydew. >>> import os >>> import sys >>> import subprocess >>> FUCHSIA_ROOT = os.environ.get("FUCHSIA_DIR") >>> OUT_DIR = subprocess.check_output( \ "echo $(cat \"$FUCHSIA_DIR\"/.fx-build-dir)", \ shell=True).strip().decode('utf-8') >>> sys.path.append(f"{FUCHSIA_ROOT}/src/developer/ffx/lib/fuchsia-controller/python") >>> sys.path.append(f"{FUCHSIA_ROOT}/{OUT_DIR}/host_x64") # You can now import honeydew >>> import honeydew
Running cd $FUCHSIA_DIR && sh $FUCHSIA_DIR/src/testing/end_to_end/honeydew/scripts/uninstall.sh will automatically perform the below mentioned uninstallation steps
To fully uninstall HoneyDew, delete the virtual environment that was crated
(fuchsia_python_venv)~/fuchsia$ deactivate ~/fuchsia$ rm -rf $FUCHSIA_DIR/src/testing/end_to_end/.venvs/fuchsia_python_venv
Before proceeding further, please complete the Installation and activate the python virtual environment created during the installation.
# Update `sys.path` to include Fuchsia Controller and FIDL IR paths before # importing Honeydew. >>> import os >>> import sys >>> import subprocess >>> FUCHSIA_ROOT = os.environ.get("FUCHSIA_DIR") >>> OUT_DIR = subprocess.check_output( \ "echo $(cat \"$FUCHSIA_DIR\"/.fx-build-dir)", \ shell=True).strip().decode('utf-8') >>> sys.path.append(f"{FUCHSIA_ROOT}/src/developer/ffx/lib/fuchsia-controller/python") >>> sys.path.append(f"{FUCHSIA_ROOT}/{OUT_DIR}/host_x64") # Enable Info logging >>> import logging >>> logging.basicConfig(level=logging.INFO) # You can now import honeydew >>> import honeydew # Setup HoneyDew to run using isolated FFX and collect the logs # Call this first prior to calling any other HoneyDew API >>> from honeydew.transports import ffx >>> ffx.setup(logs_dir="/tmp/foo/logs") # honeydew.create_device() will look for a specific Fuchsia device class implementation that matches the device type specified and if it finds, it returns that specific device type object, else returns GenericFuchsiaDevice object. # In below examples, # * "fuchsia-54b2-038b-6e90" is a x64 device whose implementation is present in HoneyDew. Hence returning honeydew.device_classes.x64.X64 object. # * "fuchsia-d88c-79a3-aa1d" is Google's 1p device whose implementation is not present in HoneyDew. Hence returning a generic_fuchsia_device.GenericFuchsiaDevice object. # * "fuchsia-emulator" is an emulator device whose implementation is not present in HoneyDew. Hence returning a generic_fuchsia_device.GenericFuchsiaDevice object. # * "[::1]:8022" is a fuchsia device whose SSH port is proxied via SSH from a local machine to a remote workstation. >>> fd_1p = honeydew.create_device("fuchsia-ac67-847a-2e50", ssh_private_key=os.environ.get("SSH_PRIVATE_KEY_FILE")) INFO:honeydew:Registered device classes with HoneyDew '{<class 'honeydew.device_classes.generic_fuchsia_device.GenericFuchsiaDevice'>, <class 'honeydew.device_classes.x64.X64'>, <class 'honeydew.device_classes.fuchsia_device_base.FuchsiaDeviceBase'>}' INFO:honeydew:Didn't find any matching device class implementation for 'fuchsia-ac67-847a-2e50'. So returning 'GenericFuchsiaDevice' INFO:honeydew.device_classes.fuchsia_device_base:Starting SL4F server on fuchsia-ac67-847a-2e50... >>> type(fd_1p) honeydew.device_classes.generic_fuchsia_device.GenericFuchsiaDevice >>> ws = honeydew.create_device("fuchsia-54b2-038b-6e90", ssh_private_key=os.environ.get("SSH_PRIVATE_KEY_FILE")) INFO:honeydew:Registered device classes with HoneyDew '{<class 'honeydew.device_classes.generic_fuchsia_device.GenericFuchsiaDevice'>, <class 'honeydew.device_classes.x64.X64'>, <class 'honeydew.device_classes.fuchsia_device_base.FuchsiaDeviceBase'>}' INFO:honeydew:Found matching device class implementation for 'fuchsia-54b2-038b-6e90' as 'X64' INFO:honeydew.device_classes.fuchsia_device_base:Starting SL4F server on fuchsia-54b2-038b-6e90... >>> type(ws) honeydew.device_classes.x64.X64 >>> emu = honeydew.create_device("fuchsia-emulator", ssh_private_key=os.environ.get("SSH_PRIVATE_KEY_FILE")) INFO:honeydew:Registered device classes with HoneyDew '{<class 'honeydew.device_classes.generic_fuchsia_device.GenericFuchsiaDevice'>, <class 'honeydew.device_classes.x64.X64'>, <class 'honeydew.device_classes.fuchsia_device_base.FuchsiaDeviceBase'>}' INFO:honeydew:Didn't find any matching device class implementation for 'fuchsia-emulator'. So returning 'GenericFuchsiaDevice' INFO:honeydew.device_classes.fuchsia_device_base:Starting SL4F server on fuchsia-emulator... >>> type(emu) honeydew.device_classes.generic_fuchsia_device.GenericFuchsiaDevice >>> fd_remote = honeydew.create_device("fuchsia-d88c-796c-e57e", ssh_private_key=os.environ.get("SSH_PRIVATE_KEY_FILE"), device_ip_port=custom_types.IpPort.parse("[::1]:8022")) INFO:honeydew:Registered device classes with HoneyDew '{<class 'honeydew.device_classes.fuchsia_device_base.FuchsiaDeviceBase'>, <class 'honeydew.device_classes.x64.X64'>, <class 'honeydew.device_classes.generic_fuchsia_device.GenericFuchsiaDevice'>}' INFO:honeydew:Didn't find any matching device class implementation for 'fuchsia-d88c-796c-e57e'. So returning 'GenericFuchsiaDevice' INFO:honeydew.transports.ssh:Waiting for fuchsia-d88c-796c-e57e to allow ssh connection... Warning: Permanently added '[::1]:8022' (ED25519) to the list of known hosts. INFO:honeydew.transports.ssh:fuchsia-d88c-796c-e57e is available via ssh. INFO:honeydew.transports.sl4f:Starting SL4F server on fuchsia-d88c-796c-e57e... >>> type(fd_remote) <class 'honeydew.device_classes.generic_fuchsia_device.GenericFuchsiaDevice'>
>>> emu.name 'fuchsia-emulator' >>> emu.device_type 'qemu-x64' >>> emu.product_name 'default-fuchsia' >>> emu.manufacturer 'default-manufacturer' >>> emu.model 'default-model' >>> emu.serial_number
>>> emu.firmware_version '2023-02-01T17:26:40+00:00'
>>> emu.reboot() INFO:honeydew.device_classes.base_fuchsia_device:Rebooting fuchsia-emulator... INFO:honeydew.device_classes.base_fuchsia_device:Waiting for fuchsia-emulator to go offline... INFO:honeydew.device_classes.base_fuchsia_device:fuchsia-emulator is offline. INFO:honeydew.device_classes.base_fuchsia_device:Waiting for fuchsia-emulator to go online... INFO:honeydew.device_classes.base_fuchsia_device:fuchsia-emulator is online. INFO:honeydew.transports.sl4f:Starting SL4F server on fuchsia-emulator... >>> emu.log_message_to_device(message="This is a test INFO message logged by HoneyDew", level=honeydew.custom_types.LEVEL.INFO) >>> emu.snapshot(directory="/tmp/") INFO:honeydew.device_classes.fuchsia_device_base:Snapshot file has been saved @ '/tmp/Snapshot_fuchsia-emulator_2023-03-01-01-09-43-PM.zip' '/tmp/Snapshot_fuchsia-emulator_2023-03-01-01-09-43-PM.zip'
>>> emu.close() >>> del emu
Running cd $FUCHSIA_DIR && sh $FUCHSIA_DIR/src/testing/end_to_end/honeydew/scripts/conformance.sh will automatically ensure you have followed the guidelines. Run this script and fix any errors it suggests.
Once the script has completed successfully, it will print output similar to the following:
INFO: Honeydew code has passed all of the conformance steps
These guidelines need to be run at the least on the following patchsets:
Below guidelines requires certain dependencies that are not yet available in Fuchsia third-party. So for the time being you need to pip install these dependencies inside a python virtual environment.
Follow Installation to install HoneyDew which will also install all of these dependencies.
Before proceeding further, ensure you are inside the virtual environment created by above installation step.
# activate the virtual environment ~/fuchsia$ source $FUCHSIA_DIR/src/testing/end_to_end/.venvs/fuchsia_python_venv/bin/activate (fuchsia_python_venv)~/fuchsia$
Running cd $FUCHSIA_DIR && sh $FUCHSIA_DIR/src/testing/end_to_end/honeydew/scripts/format.sh will automatically perform all of the below mentioned style guide checks. Run this script and fix any errors it suggests.
HoneyDew code follows Google Python Style Guide and it is important to ensure any new code written continue to follow these guidelines.
At this point, we do not have an automated way (in CQ) for identifying this and alerting the CL author prior to submitting the CL. Until then CL author need to follow the below instructions every time HoneyDew code is changed.
(fuchsia_python_venv)~/fuchsia$ autoflake --in-place --remove-unused-variables --remove-all-unused-imports --remove-duplicate-keys --recursive $FUCHSIA_DIR/src/testing/end_to_end/honeydew/
(fuchsia_python_venv)~/fuchsia$ isort $FUCHSIA_DIR/src/testing/end_to_end/honeydew/
fx format-code underneath uses yapf for formatting the python code.(fuchsia_python_venv)~/fuchsia$ fx format-code
pylint is properly installed by running pylint --helppylint(fuchsia_python_venv)~/fuchsia$ pylint --rcfile=$FUCHSIA_DIR/src/testing/end_to_end/honeydew/linter/pylintrc $FUCHSIA_DIR/src/testing/end_to_end/honeydew/honeydew/ (fuchsia_python_venv)~/fuchsia$ pylint --rcfile=$FUCHSIA_DIR/src/testing/end_to_end/honeydew/linter/pylintrc $FUCHSIA_DIR/src/testing/end_to_end/honeydew/tests/
mypy is properly installed by running mypy --helpmypy(fuchsia_python_venv)~/fuchsia$ mypy --config-file=$FUCHSIA_DIR/src/testing/end_to_end/honeydew/pyproject.toml $FUCHSIA_DIR/src/testing/end_to_end/honeydew/
Running cd $FUCHSIA_DIR && sh $FUCHSIA_DIR/src/testing/end_to_end/honeydew/scripts/coverage.sh will automatically perform the below mentioned coverage steps which shows comprehensive coverage on the entire Honeydew codebase.
For a targeted report on only the locally modified files , run the command above with the --affected flag.
It is a hard requirement that HoneyDew code is well tested using a combination of unit and functional tests.
Broadly this is how we can define the scope of each tests:
<device>.reboot() actually reboots Fuchsia device) which can’t be ensured using unit test casesWe use coverage tool for measuring the code coverage requirement of HoneyDew.
At this point, we do not have an automated way (in CQ) for identifying this and alerting the CL author prior to submitting the CL. Until then CL author need to follow the below instructions every time HoneyDew code is changed:
coverage is properly installed by running coverage --helpparameterized is properly installed by running pip show parameterized# Set up environment for running coverage tool (fuchsia_python_venv)~/fuchsia$ BUILD_DIR=$(cat "$FUCHSIA_DIR"/.fx-build-dir) (fuchsia_python_venv)~/fuchsia$ OLD_PYTHONPATH=$PYTHONPATH (fuchsia_python_venv)~/fuchsia$ PYTHONPATH=$FUCHSIA_DIR/$BUILD_DIR/host_x64:$FUCHSIA_DIR/src/developer/ffx/lib/fuchsia-controller/python:$PYTHONPATH # Run unit tests using coverage tool (fuchsia_python_venv)~/fuchsia$ coverage run -m unittest discover --top-level-directory $FUCHSIA_DIR/src/testing/end_to_end/honeydew --start-directory $FUCHSIA_DIR/src/testing/end_to_end/honeydew/tests/unit_tests --pattern "*_test.py" # Run below command to generate code coverage stats (fuchsia_python_venv)~/fuchsia$ coverage report -m # Delete the .coverage file (fuchsia_python_venv)~/fuchsia$ rm -rf .coverage # Restore environment (fuchsia_python_venv)~/fuchsia$ PYTHONPATH=$OLD_PYTHONPATH
Finally, follow Uninstallation for the cleanup.
One of the primary goal while designing HoneyDew was to make it easy to contribute for anyone working on Host-(Fuchsia)Target interactions.
HoneyDew is meant to be the one stop solution for any Host-(Fuchsia)Target interactions. We can only make this possible when more people contribute to HoneyDew and add more and more interactions that others can also benefit.
Here are some of the best practices that should be followed while contributing to HoneyDew:
group("tests") section in the BUILD.gn file (located in the same directory as unit test)group("tests") section in the top level HoneyDew unit tests BUILD filegroup("tests") section in the top level HoneyDew functional tests BUILD fileHere are some of the things to follow during HoneyDew CL review process as a CL author/contributor (or) CL reviewer/approver:
... Verified the following on Patchset: <initial patchset number> * HoneyDew code guidelines * Successfully ran the impacted functional tests using [LocalRun|InfraRun]
... Verified the following on Patchset: <final patchset number> * HoneyDew code guidelines * Successfully ran the impacted functional tests using [LocalRun|InfraRun] * Successfully ran Honeydew builders using try-jobs
