resultdb/proto/v1/test_result.proto

// Copyright 2019 The LUCI Authors.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//      http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

syntax = "proto3";

package luci.resultdb.v1;

import "google/protobuf/duration.proto";
import "google/protobuf/timestamp.proto";

import "resultdb/proto/v1/common.proto";
import "resultdb/proto/v1/test_metadata.proto";
import "resultdb/proto/v1/failure_reason.proto";

option go_package = "go.chromium.org/luci/resultdb/proto/v1;resultpb";

// A result of a functional test case.
// Often a single test case is executed multiple times and has multiple results,
// a single test suite has multiple test cases,
// and the same test suite can be executed in different variants
// (OS, GPU, compile flags, etc).
//
// This message does not specify the test id.
// It should be available in the message that embeds this message.
message TestResult {
  reserved 11;  // test_location

  // Can be used to refer to this test result, e.g. in ResultDB.GetTestResult
  // RPC.
  // Format:
  // "invocations/{INVOCATION_ID}/tests/{URL_ESCAPED_TEST_ID}/results/{RESULT_ID}".
  // where URL_ESCAPED_TEST_ID is test_id escaped with
  // https://golang.org/pkg/net/url/#PathEscape See also https://aip.dev/122.
  //
  // Output only.
  string name = 1;

  // Test id, a unique identifier of the test in a LUCI project.
  // Regex: ^[[::print::]]{1,512}$
  //
  // If two tests have a common test id prefix that ends with a
  // non-alphanumeric character, they considered a part of a group. Examples:
  // - "a/b/c"
  // - "a/b/d"
  // - "a/b/e:x"
  // - "a/b/e:y"
  // - "a/f"
  // This defines the following groups:
  // - All items belong to one group because of the common prefix "a/"
  // - Within that group, the first 4 form a sub-group because of the common
  //   prefix "a/b/"
  // - Within that group, "a/b/e:x" and "a/b/e:y" form a sub-group because of
  //   the common prefix "a/b/e:".
  // This can be used in UI.
  // LUCI does not interpret test ids in any other way.
  string test_id = 2;

  // Identifies a test result in a given invocation and test id.
  // Regex: ^[a-z0-9\-_.]{1,32}$
  string result_id = 3;

  // Description of one specific way of running the test,
  // e.g. a specific bucket, builder and a test suite.
  Variant variant = 4;

  // Whether the result of test case execution is expected.
  // In a typical Chromium CL, 99%+ of test results are expected.
  // Users are typically interested only in the unexpected results.
  //
  // An unexpected result != test case failure. There are test cases that are
  // expected to fail/skip/crash. The test harness compares the actual status
  // with the expected one(s) and this field is the result of the comparison.
  bool expected = 5;

  // Machine-readable status of the test case.
  // MUST NOT be STATUS_UNSPECIFIED.
  TestStatus status = 6;

  // Human-readable explanation of the result, in HTML.
  // MUST be sanitized before rendering in the browser.
  //
  // The size of the summary must be equal to or smaller than 4096 bytes in
  // UTF-8.
  //
  // Supports artifact embedding using custom tags:
  // * <text-artifact> renders contents of an artifact as text.
  //   Usage:
  //   * To embed result level artifact: <text-artifact
  //   artifact-id="<artifact_id>">
  //   * To embed invocation level artifact: <text-artifact
  //   artifact-id="<artifact_id>" inv-level>
  string summary_html = 7;

  // The point in time when the test case started to execute.
  google.protobuf.Timestamp start_time = 8;

  // Duration of the test case execution.
  // MUST be equal to or greater than 0.
  google.protobuf.Duration duration = 9;

  // Metadata for this test result.
  // It might describe this particular execution or the test case.
  // A key can be repeated.
  repeated StringPair tags = 10;

  // Hash of the variant.
  // hex(sha256(sorted(''.join('%s:%s\n' for k, v in variant.items())))).
  //
  // Output only.
  string variant_hash = 12;

  // Information about the test at the time of its execution.
  TestMetadata test_metadata = 13;

  // Information about the test failure. Only present if the test failed.
  FailureReason failure_reason = 14;
}

// Machine-readable status of a test result.
enum TestStatus {
  // Status was not specified.
  // Not to be used in actual test results; serves as a default value for an
  // unset field.
  STATUS_UNSPECIFIED = 0;

  // The test case has passed.
  PASS = 1;

  // The test case has failed.
  // Suggests that the code under test is incorrect, but it is also possible
  // that the test is incorrect or it is a flake.
  FAIL = 2;

  // The test case has crashed during execution.
  // The outcome is inconclusive: the code under test might or might not be
  // correct, but the test+code is incorrect.
  CRASH = 3;

  // The test case has started, but was aborted before finishing.
  // A common reason: timeout.
  ABORT = 4;

  // The test case did not execute.
  // Examples:
  // - The execution of the collection of test cases, such as a test
  //   binary, was aborted prematurely and execution of some test cases was
  //   skipped.
  // - The test harness configuration specified that the test case MUST be
  //   skipped.
  SKIP = 5;
}

// Indicates the test subject (e.g. a CL) is absolved from blame
// for an unexpected result of a test variant.
// For example, the test variant fails both with and without CL, so it is not
// CL's fault.
message TestExoneration {
  // Can be used to refer to this test exoneration, e.g. in
  // ResultDB.GetTestExoneration RPC.
  // Format:
  // invocations/{INVOCATION_ID}/tests/{URL_ESCAPED_TEST_ID}/exonerations/{EXONERATION_ID}.
  // URL_ESCAPED_TEST_ID is test_variant.test_id escaped with
  // https://golang.org/pkg/net/url/#PathEscape See also https://aip.dev/122.
  //
  // Output only.
  string name = 1;

  // Test identifier, see TestResult.test_id.
  string test_id = 2;

  // Description of the variant of the test, see Variant type.
  // Unlike TestResult.extra_variant_pairs, this one must be a full definition
  // of the variant, i.e. it is not combined with Invocation.base_test_variant.
  Variant variant = 3;

  // Identifies an exoneration in a given invocation and test id.
  // It is server-generated.
  string exoneration_id = 4;

  // Reasoning behind the exoneration, in HTML.
  // MUST be sanitized before rendering in the browser.
  string explanation_html = 5;

  // Hash of the variant.
  // hex(sha256(sorted(''.join('%s:%s\n' for k, v in variant.items())))).
  string variant_hash = 6;
}