test-patch

Docker Support

By default, test-patch runs in the same shell where it was launched. It can alternatively use Docker to launch itself in a container. This is particularly useful if running under a QA environment that does not provide all the necessary binaries. For example, if the patch requires a newer version of Java than what is installed on a Jenkins instance.

The --docker parameter tells test-patch to run in Docker mode. The --dockerfile parameter allows one to provide a custom Dockerfile. The Dockerfile should contain all of the necessary binaries and tooling needed to run the test. test-patch will copy this file up until the text YETUS CUT HERE to a different directory and then append its necessary hooks to re-launch itself prior to executing docker.

If a custom Dockerfile cannot be used or the docker executable does not work, test-patch will attempt to recover by switching to its bundled Dockerfile or disabling docker support and running locally. This behavior can be changed with the --dockeronfail option. It takes a list of comma-delimited settings:

The ‘fail’ setting is always the last option that test-patch will use and may be omitted unless it is the only option.

For example, --dockeronfail=continue means if the Dockerfile can’t be found, just turn off Docker support and continue running. --dockeronfail=fallback will switch to the bundled Dockerfile and then fail the build if docker fails to work. --dockeronfail=fail means to just fail the build and do not try any other mechanisms of recovery. The default is ‘fallback,continue,fail’ which will allow test-patch to try to continue executing as much as it possibily can.

Be aware that if the Dockerfile is found and the docker command works, test-patch will always fail the build if the Dockerfile itself fails the build. It will not attempt to continue in the non-Docker mode.

Dockerfile images will be named with a test-patch prefix and suffix with either a date or a git commit hash. By using this information, test-patch will automatically manage broken/stale container images that are hanging around if it is run in –robot mode. In this way, if Docker fails to build the image, the disk space should eventually be cleaned and returned back to the system. The docker mode can also be run in a safe mode that prevents deletions via the --dockerdelrep option. Specifying this option will cause test-patch to only report what it would have deleted, but not actually remove anything.

Resource Controls

Docker’s --memory flag is supported via the --dockermemlimit option. This enables the container’s memory size to be limited. This may be important to set to prevent things like broken unit tests bringing down the entire build server. See the Docker documentation for more details. Apache Yetus also sets the –oom-score-adj to 500 in order to offer itself as the first processes to be killed if memory is low.

Additionally, if bash v4 and Linux is in use, a separate process is launched to keep a rolling count of the maximum number of threads (not processes!) in use at one time. This number will be reported at the end of the test-patch run. Depending upon the build, languages, features enabled, etc, this number may be helpful in determining what the value of --proclimit

Process Reaper

A common problem is the ‘stuck’ unit test. If bash v4.0 or higher is in use, Apache Yetus may be told to turn on the process reaper functionality. Using the --reapearmode option, this feature may be configured to either report and even kill left over processes that match provided regular expressions.

WARNING: Using --reapermode outside of Docker will report or kill ALL matching processes on the system. It is recommended to only use those options whilst in Docker mode.

The reaper will run after every ‘external’ command that is printed on the console. This includes almost all build tool commands and individual test commands.

Plug-ins

test-patch allows one to add to its basic feature set via plug-ins. There is a directory called test-patch.d inside the directory where test-patch.sh lives. Inside this directory one may place some bash shell fragments that, if setup with proper functions, will allow for test-patch to call it as necessary. Different plug-ins have specific functions for that particular functionality. In this document, the common functions available to all/most plug-ins are covered. Test plugins are covered below. See other documentation for pertinent information for the other plug-in types.

Common Plug-in Functions

Every plug-in must have one line in order to be recognized, usually an ‘add’ statement. Test plug-ins, for example, have this statement:

add_test_type <pluginname>

This function call registers the pluginname so that test-patch knows that it exists. Plug-in names must be unique across all the different plug-in types. Additionally, the ‘all’ plug-in is reserved. The pluginname also acts as the key to the custom functions that you can define. For example:

function pluginname_filefilter

defines the filefilter for the pluginname plug-in.

Similarly, there are other functions that may be defined during the test-patch run:

HINT: It is recommended to make the pluginname relatively small, 10 characters at the most.  Otherwise, the ASCII output table may be skewed.

Plug-in Importation

Plug-ins are imported from several key directories:

If the --skip-system-plugins flag is passed, then only core.d is imported.

Test Plug-ins

Plug-ins geared towards independent tests are registered via:

add_test_type <pluginname>

NOTE: If the plug-in has support for maven, the maven_add_install pluginname should be executed. See more information in Custom Maven Tests in the build tool documentation.

Personalities

Configuring for Other Projects

It is impossible for any general framework to be predictive about what types of special rules any given project may have, especially when it comes to ordering and Maven profiles. In order to direct test-patch to do the correct action, a project personality should be added that enacts these custom rules.

A personality consists of two functions. One that determines which test types to run and another that allows a project to dictate ordering rules, flags, and profiles on a per-module, per-test run.

There can be only one of each personality function defined.

Global Definitions

Globals for personalities should be defined in the personality_globals function. This function is called after the other plug-ins have been imported. This allows one to configure any settings for plug-ins that have been imported safely:

function personality_globals
{
  PATCH_BRANCH_DEFAULT=master
  GITHUB_REPO="apache/yetus"
}

Additionally, a personality may require some outside help from the user. The personality_parse_args function is called almost immediately after the personality is loaded and plug-ins parse arguments.

function personality_parse_args
{
  echo "$*"
}

It is important to note that this function is called AFTER personality_globals.

Test Determination

The personality_file_tests function determines which tests to turn on based upon the file name. It is relatively simple. For example, to turn on a full suite of tests for Java files:

function personality_file_tests
{
  local filename=$1

  if [[ ${filename} =~ \.java$ ]]; then
    add_test findbugs
    add_test javac
    add_test javadoc
    add_test mvninstall
    add_test unit
  fi

}

The add_test function is used to activate the standard tests. Additional plug-ins (such as checkstyle), will get queried on their own.

Module & Profile Determination

Once the tests are determined, it is now time to pick which modules should get used. That’s the job of the personality_modules function.

function personality_modules
{

    clear_personality_queue

...

    personality_enqueue_module <module> <flags>

}

It takes exactly two parameters repostatus and testtype.

The repostatus parameter tells the personality function exactly what state the source repository is in. It can only be in one of two states: branch or patch. branch means the patch has not been applied. The patch state is after the patch has been applied.

The testtype state tells the personality exactly which test is about to be executed.

In order to communicate back to test-patch, there are two functions for the personality to use.

The first is clear_personality_queue. This removes the previous test’s configuration so that a new module queue may be built. Custom personality_modules will almost always want to do this as the first action.

The second is personality_enqueue_module. This function takes two parameters. The first parameter is the name of the module to add to this test’s queue. The second parameter is an option list of additional flags to pass to Maven when processing it. personality_enqueue_module may be called as many times as necessary for your project.

NOTE: A module name of . signifies the root of the repository.

For example, let’s say your project uses a special configuration to skip unit tests (-DskipTests). Running unit tests during a javadoc build isn’t very useful and wastes a lot of time. We can write a simple personality check to disable the unit tests:

function personality_modules
{
    local repostatus=$1
    local testtype=$2

    if [[ ${testtype} == 'javadoc' ]]; then
        personality_enqueue_module . -DskipTests
        return
    fi
    ...

This function will tell test-patch that when the javadoc test is being run, do the documentation build at the base of the source repository and make sure the -DskipTests flag is passed to our build tool.

Enabling Plug-ins

Personalities can set the base list of plug-ins to enable and disable for their project via the personality_plugins function. Just call it with the same pattern as the --plugins command line option:

personality_plugins "all,-checkstyle,-findbugs,-asflicense"

This list is used if the user does not provide a list of plug-ins.

Important Variables

There are a handful of extremely important system variables that make life easier for personality and plug-in writers. Other variables may be provided by individual plug-ins. Check their development documentation for more information.