Introduction

Target group of this document and the accompanying cookbook are developers who need to create new assemblies and maintain existing ones. This document assumes that you are familiar with Concept - Continuous Delivery Tools (valid to 7.10), at least up to chapter Assemblies.

Glossary

References

• Concept - Continuous Delivery Tools (valid to 7.10)
• Cookbook - Gradle Assembly Tools (valid to 7.8)

Built Assembly And Deployment

Like any component the assembly has a source form (located in the version control system) and a built form (located in an artifact repository). Also there is a process called assembly process transforming the sources into the published from.

This chapter describes the published form and how it is interpreted by the deployment. The next chapter covers the process of creating it and the source form.

Each binary assembly has an ivy.xml file. The following example demonstrates basic structure and typical contents of an assembly's ivy.xml file:

<ivy-module version="2.0" xmlns:e="http://ant.apache.org/ivy/extra">
  <info organisation="com.intershop.assembly" module="primetech" revision="7.5.0.0.20140207101035" status="integration" 
    publication="20140207124937" e:productName="Intershop 7" e:copyrightOwner="Intershop Communications" e:copyrightFrom="2005">
    <e:hostType name="appserver" includeCartridges="false" includeShare="false" includeLocal="true"/>
    <e:hostType name="share" includeCartridges="true" includeShare="true" includeLocal="false"/>
    <e:hostType name="webserver" includeCartridges="false" includeShare="false" includeLocal="true"/>
    <e:environment name="development"/>
    <e:environment name="production"/>
    <e:environment name="test"/>
  </info>
  <configurations>
    <conf name="appserver" visibility="public"/>
    <conf name="default" visibility="public"/>
    <conf name="deploy-plugins" visibility="public"/>
    <conf name="development-cartridges" visibility="public"/>
    <conf name="init-cartridges" visibility="public"/>
    <conf name="production-cartridges" visibility="public"/>
    <conf name="share" visibility="public"/>
    <conf name="test-cartridges" visibility="public"/>
    <conf name="webserver" visibility="public"/>
  </configurations>
  <publications>
    <artifact name="deploy-settings" type="deploy-settings-gradle" ext="gradle" conf="default"/>
    <artifact name="deploy" type="deploy-gradle" ext="gradle" conf="default"/>
    <artifact name="database-dump" type="dump" ext="dmp" conf="default"/>
    <artifact name="branding" type="share" ext="zip" conf="default"/>
  </publications>
  <dependencies defaultconfmapping="*->;default">
    <dependency org="com.intershop" name="3rd_oracle" rev="11.2.0.3.0.2" conf="appserver, development-cartridges, production-cartridges, test-cartridges" transitive="false"/>
    <dependency org="com.intershop" name="runtime" rev="7.5.0.0.20140207101035" conf="appserver, development-cartridges, production-cartridges, test-cartridges, webserver" transitive="false"/>
    <dependency org="com.intershop" name="tools" rev="7.5.0.0.20140207101035" conf="development-cartridges, production-cartridges, test-cartridges" transitive="false"/>
    <dependency org="com.intershop" name="pf_cartridge" rev="7.5.0.0.20140207101035" conf="development-cartridges, production-cartridges, test-cartridges" transitive="false"/>
    <dependency org="com.intershop" name="pf_objectgraph" rev="7.5.0.0.20140207101035" conf="development-cartridges, production-cartridges, test-cartridges" transitive="false"/>
    ...
  </dependencies>
</ivy-module>

The dependencies element lists all contained components as dependencies. Each dependency declares an exact version number (not version expressions like 7.5.0.0.+).

Also an assembly has a number of named subsets, technically represented by what are called Ivy configurations. Each assembly subset may contain an arbitrary amount of components and a component can be contained in a huge variety of subsets, as indicated by the conf attribute of the dependency element.

Examples of such subsets are components to be deployed on different host types, like components for appserver and webserver hosts. It is possible to use this technique for any aspect in deployment, where it is necessary to mark components for a certain purpose, also custom ones. The (kinds of) subsets predefined by Intershop are described in section Assembly Subsets.

Besides dependencies to other components, the assembly's ivy.xml file also describes artifacts of the assembly. Artifacts of the assembly are contained as files near the ivy.xml file in the artifact repository. It is useful to have artifacts not only in infrastructure components and cartridges, but also in assemblies, for artifacts that can only be created once all contained components and their versions are known. An example is the database dump (useful for demo-purposes) of all cartridges. It is possible to add custom artifacts. The (types of) artifacts predefined by Intershop are described in the section Generated Assembly Artifacts.

Lastly the assembly contains additional meta-data using Ivy 'extra attributes' (attributes of the info element) or 'extra elements' (nested elements in the info element). This also leaves room for arbitrary customization. Types of meta-data predefined by Intershop, like copyright information, that are displayed on login screens, are described in the Cookbook - Gradle Assembly Tools (valid to 7.8).

Cartridges and Infrastructure Components

The assembly distinguishes between two kinds of assigned system components: cartridges and infrastructure components. Whether a system component is considered as a cartridge or an infrastructure component is therefore determined by the assembly subsets it is assigned to. It is neither defined by any meta-data (i.e., the ivy.xml file) in the system component itself nor by any convention or rule.

Cartridges

Cartridges are all these system components that are assigned to one or more environments or the subset init-cartridges (see descriptions below). A system component that is marked as a cartridge in the way it is built and deployed:

• Using its deploy.gradle artifact, if it has one - i.e., if has a custom deployment script.
• Using the CartridgeDeploymentPlugin otherwise.

Also system components marked as cartridges are added to the CARTRIDGES_DIR/cartridgelist.properties file.

Besides the ivy.xml file the CartridgeDeploymentPlugin also deploys:

• all artifacts of type cartridge to the cartridge folder (CARTRIDGES_DIR/<cartridge>)
• all artifacts of type jar to the cartridge lib folder (CARTRIDGES_DIR/<cartridge>/release/lib)
• all artifacts of type share to IS_SHARE directory

Infrastructure Components

Infrastructure components are all components that are assigned to one or more host types (see below).

Most of these scripts apply the InfrastructureDeploymentPlugin which deploys:

• all artifacts of type local into IS_HOME direcotry
• all artifacts of type share into IS_SHARE directory

Assembly Subsets

Environments

Environments are different stages in the life-cycle of an assembly, like development, demo, quality assurance and production. At time of deployment exactly one environment is selected.

An environment is declared using an environment element nested in the info element.

An environment has a name, which is an arbitrary string.

Also it has an associated Ivy configuration by the name <environment>-cartridges. Components assigned to this configuration are only included in the deployment if the environment is selected. Also they are considered cartridges (see above).

Host Types

Host types are types of nodes in a cluster deployment. They are declared using a host-type element nested in the info element of the assemblies ivy.xml file. At time of deployment exactly one host type is selected possessing a name, which is an arbitrary string.

Also it has an associated Ivy configuration with the same name. System components assigned to this configuration are only included in the deployment if the host type is selected. Also they are considered infrastructure components (see above).

Components may be assigned to multiple host types. The host type's flags control indicates which artifacts are to be deployed for each component. There are four flags for each host type:

• includeShare specifies share artifacts (like sites content) to be deployed into the shared file system (IS_SHARE).
• includeLocal specifies instance-specific artifacts (like executable files) to be deployed locally (IS_HOME).
• includeCartridges specifies the cartridge directory's artifacts to be deployed (the cartridge's ivy.xml and artifacts of type cartridge and jar).
• includeJavadoc specifies the Javadoc will be deployed for all components (including cartridges) as well. (since Gradle Tools 2.0)

See section Cartridges below for what cartridge means in the context of a deployment. Depending on the set flag of the host type (includeCartridges, includeShare or includeJavadoc), the deployment will load cartridges in addition to the host type's own components.

There is an implicit host type all, that can be selected upon deployment despite not being declared in the assemblies ivy.xml file. The host type all is used for the single-machine-installation. It contains all components of all other host types, and has all four flags set to true. (Which also explains why includeShare and includeLocal are independent flags and not neceessarily the opposite of each other.)

It is possible to define custom host types. Predefined are: webserver, appserver, share, javadoc (until 7.5) and solr (since 7.5) (see for the Concept - Continuous Delivery Tools | Assembly Subsset Host Type details). Predefined in this context means, that most assemblies delivered by Intershop declares them.

Ivy Configuration 'init-cartridges'

The Ivy configuration init-cartridges contains system components that should be deployed as cartridges, but only be loaded by the DBInit tool and not by the application server.

This is available as a safety measure against running potentially unsecure DBInit code (like special pipelines) in the application server. If a cartridge contains both production and DBinit code it can instead be added to an environment.

Selected Cartridges and Infrastructure Components

Based on the selected host type and environment, components and cartridges are selected for deployment.

The set of cartridges to be deployed is calculated the following way:

• Cartridges of the selected environment plus
• Init-Cartridges

The IS_SHARE/system/cartridges/cartridgelist.properties is generated during the deployment by:

• Filling the property cartridges (for the application server) with all cartridges of the selected environment
• Filling the property cartridges.dbinit (for the dbinit) with all cartridges of the selected environment plus all init-cartridges

The set of components to be deployed in total is calculated the following way:

• All components contained in the selected host-type plus
• Cartridges as calculated above, if the host-type has the flag includeCartridges set, or no cartridges otherwise.

The following picture shows an assembly with a few subsets, cartridges and infrastructure components:

The next diagram shows, which components get deployed at deployment time:

• the host-type appserver and
• the environment production are selected.

(Actually any environment would result in the same deployment here, because the host-type appserver does not include cartridges.)

The last diagram shows, which components get deployed if at deployment time

• the host-type share and
• the environment production are selected

Ivy Configuration Deploy-Plugins

The Ivy configuration deploy-plugins contains the components needed to run the deployment. They are listed using an exact version (instead of a version expression like 1.0.0.+) to ensure that running a deployment on the same assembly (using the same configuration) will always have the same result.

You should always inherit the configuration deploy-plugins from an assembly supplied by Intershop/your vendor. (You may add additional deploy-plugins, see the  Cookbook - Gradle Assembly Tools (valid to 7.8) for details.)

Generated Assembly Artifacts

The assembly process generates or copies artifacts, that are required for the deployment process. These artifacts are described in this section.

The Deployment Configuration: deploy-settings.gradle and deploy.gradle

Assemblies must contain two artifacts to control the overall flow of the deployment:

• deploy-settings.gradle is interpreted like Gradle's settings.gradle files for multi-project builds. It controls which sub-projects are included in the deployment, i.e., which components have to be deployed.
• deploy.gradle is interpreted like a build.gradle file of the top-level project of a multi-project build. It provides DSL extensions on which the user can configure the deployment and assembly-specific deployment tasks (like generation of the cartridgelist.properties or the deployment of the database dump)

For more information on the contents and roles of these files see the Concept - Gradle Deployment Tools.

Database Dump

Assemblies may contain a database dump that is generated during the assembly process. This dump can be used in certain environments (like demo servers or for developers) to initialize the database and save time compared to running DBInit.

Shared Artifacts

The assembly may contain arbitrary artifacts of type share. These artifacts must be zip-files and are extracted to IS_SHARE directory. An example of such an artifact is a package of branding installations in the <IS_SHARE>/sites folder. This package is generated during the DBInit as a byproduct when running a BrandingPreparer and must be deployed, so a dump import can fully replace a DBInit.

Assembly Process

Steps

The assembly process is a build process, running a build.gradle file. The main Gradle task to run the assembly process is to publish. The following picture shows the steps of the assembly process.

The steps filled with grey are optional:

• The step run smoke test is not a dependency of publish and must be triggered explicitly.
• The step export dump and its dependencies dbinit and deploy server are only executed if the project property releaseWithDump is set to true (default: false).

On continuous Integration servers, the  releaseWithDump property should be set, also some tests should be executed.

In a developer environment  releaseWithDump is typically not set. The remaining two steps when running the task publish are lightweight and fast, reducing the assembly process to the bare minimum. They can still be executed by running dedicated Gradle tasks, see below.

Generate ivy.xml File

The core of the assembly process is to generate the  ivy.xml file of the assembly from declarations in the  build.gradle file. Dependencies and configurations in the Gradle project result in dependencies and configurations in the assembly's ivy.xml file. If transitive dependencies are specified, they are resolved transitively at first and all resulting components are added as dependencies in the  ivy.xml.

You can use Gradle's DSL for dependency management to the full extent to add/modify/exclude dependencies. For most common tasks (like adding cartridges to an environment) we offer a more convenient custom DSL. See the Cookbook - Gradle Assembly Tools (valid to 7.8) for details.

For generating the cartridgelist.properties during deployment, cartridges are sorted in the resulting ivy.xml using a file named cartridge-order.txt.

Deploy Server

As a pre-requisite of later steps the assembly must be deployed. The deployment during the assembly is as close to a deployment in production as it is possible, i.e., it uses the same deployment scripts and plugins like the real deployment does. 

Before running the deployment the assembly process must:

1. Make the assembly available in a repository. For this a preliminary version of the assembly is published - not containing the database dump, which will be generated in a later step.
2. Generate a  settings.gradle file for running the deployment. Configuration values that cannot be determined automatically, like ports and the database account, must be provided by the user in an  environment.properties file.

The preliminary version of the assembly is called the build assembly publication. It

1. has a special version number build(for example com.intershop.assembly:intershop7:build) and
2. is published to a temporary repository located at <assembly source directory>\target\build-repo.

Developers can trigger this step by running the Gradle task deployServer. The resulting server is sufficient for running most manual and automatic tests during development.

The deployment during the assembly process always specifies host type all, i.e., a single machine deployment.

DBInit and Export Dump

The DBInit is run inside the deployed server. The assembly process observes the console output/log files of the DBInit and fails if any errors occur.

Developers can trigger this step by running the task dbinit.

After running the DBInit the legacy ant export script is used to generate a database dump file.

Smoke Tests

Optionally some some tests can be executed by the test task, including:

• starting an application server
• starting a web-server

and observing log-files for errors/warnings. The assembly process fails if an error occurs.

Integration Tests

Starting with Gradle Tools 2.3, the assembly process also executes certain integration tests, that are:

• Located in a cartridge, that is
  • Newly added by the current assembly (i.e., not inherited)
  • Part of the environment test,
• Residing in the package tests.unit for test cases, that are starting an embedded server instance.
  These tests are executed by the task ishUnitTest.

Integration test execution is based on Gradle's Test task, so that all standard functionalities also apply for integration tests, e.g., selecting only a subset of tests for execution.

The overall check task depends on all kinds of test tasks. In the future, more types of integration tests may be executable during the assembly process. Call the test task instead of check to execute only smoke tests.

The assembly process fails if an error occurs.

Publish

Finally a non-preliminary ivy.xml file is generated, containing the database dump as an artifact if releaseWithDump is true. The ivy.xml file and all declared artifacts are then uploaded/copied to a repository. From there they can be deployed or have other assemblies inherit from them.

Assembly Plugins

The steps described above are provided by two Gradle Plugins:

• Step 1 along with the DSL for host-types, environments and inheritance is contained in com.intershop.build.gradle:assembly. This functionality is very general and not limited to be used to deploy Intershop 7 based projects.
• All other steps are contained in com.intershop.build.gradle:ish-assembly and are specific to Intershop 7 based projects and runtime architecture.

Source Files

Typical source files used to build an assembly are:

Inheritance

It is possible to inherit declarations/contents from existing binary assemblies automatically. The following things can be inherited:

• Ivy configurations and their contained components
• Environments and their contained cartridges
• Host Types and their contained components
• Assembly artifacts
• Database dumps from inherited assemblies as basis for a derived database dump by running an import and a partial DBInit

See the  Cookbook - Gradle Assembly Tools (valid to 7.8) for how the custom DSL for assembly inheritance works.