Introduction

This cookbook gives practical advise on how to develop using Gradle and Intershop build and deployment tools. The target audience are developers of Intershop customer projects and partners. Knowledge of Concept - Continuous Delivery Tools (valid to 7.10) is assumed.The cookbook presents a directory setup and workflow, that is optimized for a single system component set and assembly.

Preconditions

Before a single developer can set up an environment, a few things have to be set up first for the team. The required team setup is out of scope of this cookbook. Such information is included in:

• Cookbook - Setup CI Infrastructure (valid to GradleTools 1.1), resp.Cookbook - Setup CI Infrastructure (valid to GradleTools 2.1) - infrastructure setup for continuous integration (especially, setting up a preconfigured Gradle user home- developer configuration - available from the VCS)
• Cookbook - Gradle Build Tools - create a system component set

• Cookbook - Gradle Assembly Tools (valid to 7.8) - create an assembly

Glossary

Recipe: Set Up a Development Environment

Problem

I want to develop on Intershop 7.4 CI or later. How get I a Gradle-based development environment?

Solution

Setting up an development environments consists of the following steps:

1. Set up a Gradle user home.
2. Check out system component set and assembly using your VCS.
3. Build system component set and publish to local repository.
4. Deploy a server.
5. Initialize the database.
6. Set up an Intershop Studio workspace.

The following steps provide you with a jump start without further ado. See discussion for further details.

Since command line syntax on Windows and Linux slightly differ, the steps are provided once for Windows and once for Linux. Except for the command line syntax both are equivalent.

Windows

* Commands provided by gradle_environment.bat created with Gradle Tools >= 2.0

Starting services and the application server are out of scope of this document.

Linux

* Commands provided by gradle_environment.bat created with Gradle Tools >= 2.0

Starting services and the application server are out of scope of this document.

Discussion

Gradle User Home and Developer Home

Gradle tools version 1.0-1.1:

The Gradle user home (<GRADLE_USER_HOME>) is a folder that stores all files associated with a certain development context. It contains:

• An environment script gradle_environment.bat/ gradle_environment.sh
  
• Developer specific configuration in gradle.properties
• Caches for downloads from remote artifact repositories

Gradle tools version >= 2.0:

The developer home is a folder containing an environment script (gradle_environment.bat/ gradle_environment.sh), see next section. The developer home is checked into the VCS.

The Gradle user home (<GRADLE_USER_HOME>) is a folder that stores temporary files associated with a certain development context. It contains:

• Developer specific configuration in gradle.properties
• Caches for downloads from remote artifact repositories

The Gradle user home is not checked into the VCS.

Developer Console

When setting up your CI environment a script gradle_environment.bat | gradle_environment.sh is generated. This script configures the directory layout you work with, by setting a few environment variables.

We refer to a console with these environment variables in place as a developer console. Run all gradlew commands and Intershop Studio from a developer console.

To start a developer console:

1. Open a console or use an existing one
2. Go to
   
   1. Gradle tools 1.0-1.1: Gradle user home folder
   2. Gradle tools >= 2.0: Developer home folder
3. Run
   1. gradle_environment.bat on Windows or
   2. source ./gradle_environment.sh on Linux


The following table lists the most important environment locations provided from gradle_environment.bat | gradle_environment.sh represented by environment variables.

These environment variables are partly for your convenience and partly to configure Gradle/Intershop Studio.

with Gradle Tools >= 2.0:

The environment script provides the following alias commands:

Build, Publish and Repositories

Running gradlew publish in <COMPONENTSET> actually consists of of two steps, first build, then publish. It is necessary to not only build components, but also to publish them to a repository (e.g., copy them to a folder with predefined structure) for two reasons:

1. Gradle needs to find dependencies for compiling Java. Gradle knows two kinds of dependencies:
   
   1. External dependencies are all dependencies across component set boundaries (e.g., the cartridge bc_foundation in the component set p_business to the cartridge core in the component set p_core). They are denoted by compile group: 'com.intershop', name: '<name>' in a component's build.gradle file. They must be resolved from (a local or remote) repository.
   2. Dependencies inside a component set (e.g., core to isml, both inside p_core) are called project dependencies and are denoted by compile project(':<name>'). (Project dependencies resolved directly from the target directories.)
2. The deployment needs to find all components to deploy in a (local or remote) repository.

The useSetAsFilter task of the component set adds a filter property in the <GRADLE_USER_HOME>/gradle.properties:

filter.<organization of component set>.<name of component set> = <version>-local

This property ensures that the assembly process uses local versions for all cartridges from this component set. (So you deploy local versions of these in the initial and later re-deployments.) See also Concept - Gradle Build Tools | Configuring Dependency Versions.

Initial Deployment

The developer deployment first generates a <SERVER>/settings.gradle file from the environment.properties configuration. Based on this file it then runs standard deployment procedure as describes in Concept - Gradle Deployment Tools specifying the host type all, e.g., deploying all host types of the assembly. As deployment source it uses the repositories declared in your corporate plugin as well as your local repository at <REPO>.

Besides being based on the same deployment scripts, the developer deployment is also tailored for development in a few ways:

• The properties IS_SOURCE in <SERVER>/local/intershop.properties and cartridges.source in <SERVER>/share/system/cartridges/cartridgelist.properties. This ensures that the application server uses the cartridges checkout directory to load static artifacts and classes.

• Deploy development.properties from <ASSEMBLY>/development.properties to <SERVER>/share/system/config/cluster/development.properties . This way reload and no-preload behavior is preconfigured for the application server.
• Deploy all cartridges exclusively to the assembly environment development (i.e., that are not included in the production environment), like the dev_lilith cartridge .

• Open debug ports in tomcat.bat / tomcat.sh and dbinit.bat / dbinit.sh as configured in <ASSEMBLY>/environment.properties. The configuration of debug ports depends on the configuration of the source directories in the gradle.properties in the developer home.

Post-install Scripts and Services

The post-install scripts perform all deployment operations for which administrator / root privileges are required. Currently this is only the creation of services. The developer deployment uses the current user as deployment user and– on Windows– also as user for executing the deployed applications / services. Therefore, you have to enter your password for every service to be created.

After the post-install scripts, running services / applications is completely up to you and works mostly as before. If starting any service fails on Windows, please refer to the Recipe (Windows) Starting a Service Fails in the Cookbook - Gradle Deployment Tools (7.4 CI - ICM 7.7)

Intershop Studio

Intershop Studio will find your sources, the deployed server etc. by relying on the environment variables in a developer console. Therefore, start Intershop Studio from a developer console. No more initial configuration in the preference menu is necessary.

To import your system component set into the workspace:

1. From the menu select File | Import...
   The Import dialog opens.
2. Choose Intershop Studio | Component Set Projects into Workspace (Gradle) wizard and click Next.
   The Import Component Set (Gradle) wizard is started.
3. Browse... for the directory of the system component set.
4. Click Build Model.
5. Select the desired projects (cartridges).
6. Click Select Required to also select all required projects.
7. Click Finish.
8. Wait till background actions have finished.

To import your assembly into the workspace, use the same steps, but instead select the import wizard Gradle | Gradle Project. I.e:

1. From the menu select File | Import...
   The Import dialog opens.
2. Choose Gradle | Gradle Project wizard and click Next.
   The Import Gradle Project wizard is started.
3. Browse... for the directory of the assembly.
4. Click Build Model.
5. Click Select All.
6. Click Finish.
7. Wait till background actions have finished.

Process / Data Flow

The following diagram shows how data flows between directories (blue) during the different steps (red).

Recipe: Run Gradle Tasks

Problem

Gradle tasks are the units of work you can trigger. This recipe describes how. For an introduction beyond this recipe, please refer to the Gradle user guide.

Solution

You may run Gradle tasks using Intershop Studio or the console. Generally you have to:

1. Specify which Gradle project to run tasks in.
2. Specify which Gradle task(s) to execute.

The assembly, all cartridges and the system component set itself are Gradle projects. The component set and the cartridges form a Gradle multi-project build. The component set works as the root project and cartridges are (direct) child-projects of the component set.

Tasks can have dependencies to other tasks. Executing a certain task will always also execute its dependencies.

You may specify multiple tasks to run in one execution of Gradle. The order of depending tasks is mostly determined automatically by Gradle.

• Depending tasks will always be executed after their dependencies.
• Tasks that do not depend on each other may still have a preconfigured order, if you run them together.

Tasks independent of each other are executed in the order in which you are passing the tasks in.

Running Tasks from Intershop Studio

In Intershop Studio each Gradle project is represented by an Eclipse project.

To execute tasks in a cartridge/assembly/component set:

1. Right-click the corresponding project in the Package Explorer.
   The context menu opens.
2. Click Run As | Gradle Build...
   For cartridges you can also reach that dialog by clicking Build Cartridge (Gradle Build) in the menu.
3. In the dialog Edit Configuration select the tasks to be executed.
4. Optionally, click Order to sort the tasks.
5. Click Apply to confirm your settings.
6. Click Run to execute the task(s).

You may also specify additional parameters in the dialog.

To reuse and switch between multiple configurations per project (e.g., different deploy or database tasks in the assembly), give them different names. Thus they can be found in the list of the external tool configurations .

Running Tasks via Console

To execute tasks via Console:

1. Open a developer console
2. Go to <ASSEMBLY> directory for executing tasks in the assembly or
   to the <COMPONENTSET> for executing tasks in the component set or a single cartridge
3. Run
   1. On Windows: gradlew :<task1> :<task2>, e.g., gradlew :deployServer :importDump
   2. On Linux: ./gradlew :<task1> :<task2>, e.g., ./gradlew :deployServer :importDump

To execute a task in a certain cartridge (i.e., a child-project), pass in :<cartridge>:<task>, e.g., gradlew :core:clean :core:build.

You can also combine tasks from different cartridges, e.g., gradlew :core:build :bc_foundation:build.

FAILURE:
Could not determine which tasks to execute.

* What went wrong:
Task 'build' not found in root project 'projectComponentSet'.

* Try:
Run gradlew
tasks to get a list of available tasks. 

Discussion

The following is a loose collection of helpful hints to get you started with Gradle. Run all of them in a developer console.

To display all cartridges in the component set execute the following inside <COMPONENTSET>:

gradlew :projects

To display tasks of a (root) project (component set or assembly):

gradlew :tasks

To display the tasks of a certain subproject (e.g., cartridge) run:

gradlew :<name>:tasks

To find out which tasks are actually going to be executed (e.g., which dependencies are automatically pulled in) and in which order:

gradlew <tasks> --dry-run

Recipe: Initialize the Database

Problem

The database must be initialized before testing (or as part of the test, i.e., to see if the database can be initialized).

Solution

There are two ways to initialize your database.

Import Database Dump of Release

You may import the database dump generated for the released version of your assembly. This is only useful if your local changes have no effect on the initial database state. To import the dump run the task importDump on the assembly.

The release version from which the dump is taken is determined by the property dump.<assembly organization>.<assembly name> in <GRADLE_USER_HOME>/gradle.properties. This way rerunning the importDump task always imports the same dump, unless you change the property.

The dump property is automatically created upon first run of the importDump task. To update the property to the latest released version run the updateDumpVersion task.

Execute DBInit

You may run the DBInit through assembly. To do so:

1. Open a developer console and go to <ASSEMBLY>
2. Run
   
   1. On Windows: gradlew dbinit -PskipDeployment
   2. On Linux: ./ gradlew dbinit -PskipDeployment

Discussion

Import Database Dump of Release

The dump is downloaded from the release repository and handed over to ant import at <SERVER>/local/tools/misc.

Also branding installations, which are usually created by running DBInit are downloaded from the release repository and extracted into <SERVER>/share/sites.

You can still use ant import at <SERVER>/local/tools/misc to import other dumps.

Execute DBInit

The command first executes ant dbschema-clean at <SERVER>/local/tools/misc to purge your database account. After that it executes dbinit --classic in <SERVER>/local/bin, i.e., a full DBInit over all cartridges without interactions.

You may still run these commands manually if you like - or need to pass in other options.

Recipe: Make Changes Effective on the Server

Problem

After the initial setup, the developer changes the sources of the component set. This may happen because of changes made by the present developer or updates to the VCS working copies of assembly and component set. The developer must be able to see this changes in the deployed server.

Solution

Depending on the kind of change, different actions are necessary to make the changes visible in the running server.

See discussion below for how to perform these actions.

Discussion

Please, refer to the Recipe: Run Gradle Tasks on how to run these tasks. This recipe also mentions how to execute a task on multiple or all cartridges at once.

Rebuild

To rebuild (e.g., recompile) a cartridge, run the build task on that cartridge.

The buildDependents task builds the cartridge itself and all cartridges in your component set that depend on it. This is useful to see if Java changes cause compilation errors in depending components.

When not cleaning the build results before building / publishing, Gradle performs an incremental build. Gradle only re-execute Gradle tasks whose inputs have changed (or whose outputs have been tampered with). This is somewhat safer and more reliable than re-executing an Ant build. Using the non-cleaning, incremental build should suffice for developers most of the time.

To execute a clean build run the tasks clean and build on the cartridge(s).

Using these tasks does not touch the contents of your local repository. If you later redeploy your changes, make sure to publish them before.

Publish

To republish a cartridge to your local repository, run the publish task on that cartridge. This overwrites the local version of that cartridge inside your local repository at <REPO>.

Redeploy

All deployment tasks are run on the assembly project. Before you can deploy any change, you must have published it.

To redeploy all cartridges of your component set run the task deployCartridges.

To redeploy only a few cartridges:

1. Open a developer console and go to <ASSEMBLY>
2. Run
   
   1. On Windows: gradlew deployCartridges -Pcartridges="<space separated list of cartridges>", e.g., gradlew deployCartridges -Pcartridges="core bc_foundation"
   2. On Linux: ./gradlew deployCartridges -Pcartridges="<space separated list of cartridges>", e.g., ./gradlew deployCartridges -Pcartridges="core bc_foundation"

To redeploy the whole server run the task deployServer.

Running the post-install scripts again is not necessary after redeployment.

Recipe: Run Automatic Tests

Problem

A developer needs to execute and debug tests.

Solution

There are currently three ways to execute tests.

Run Tests Inside Intershop Studio

This should be the preferred way to execute tests for the developer.

Before being able to run the system or integration tests, you must set up your workspace by:

1. From the menu select Window | Preferences.
   You may run into a problem when opening the preference dialog the first time, see the box Issue with Intershop Studio Preferences above.
2. Go to preference page Intershop Studio | JUnit Cartridge Test | Classpath.
3. Check both Add Intershop Suite Server entries and All components.
4. Click Apply to confirm your settings.
5. Click Ok.

To run or debug a single test:

1. Select a test class, a Java package containing test classes or a method of a test class
2. Open the context menu and select
   
   1. Run as | JUnit Test for unit tests (located in the tested cartridge, see discussion) or
      Run as | JUnit Cartridge Test for integration or system tests or
   2. Debug as | JUnit Test for unit tests or
      Debug as | JUnit Cartridge Test for integration or system tests

This way of execution applies to all types of tests.

Execute Tests with Gradle

You may run all unit tests of a cartridge (i.e., not a _test-cartridge) by running the Gradle test task on that cartridge. A report is generated per cartridge at <COMPONENTSET>/<cartridge>/target/reports/tests/index.html.

You may also run the Gradle test task on the component set. This executes the test tasks of all cartridges and generates an overall report. The report can be found at <COMPONENTSET>/target/reports/allTests/index.html.

Execute Tests with Test Runner

You may run multiple integration or system tests with the Intershop Test-Runner by:

1. Open a developer console.
2. If your prompt does not yet start with ES<n>|...(e.g., ES1|D:\is7\trunk\), run <SERVER>/local/bin/environment.bat.
3. Go to <SERVER>/local/tools/misc.
4. Run ant test.

Discussion

There are three kinds of tests that are treated differently.

Unit Tests

Unit tests are tests that require nothing but a properly populated class path to run. Especially, they cannot require a pre-created files or directories (like through deployment) or a database. (They may however start up a pretty complete server in their initialization - as long as this server does not access any file system resources, like configuration files.)

Super classes/annotations for Unit tests are:

• Base class junit.framework.TestCase (legacy test cases written for JUnit version <= 4)

• Classes with @RunWith annotation (newly written test cases starting with JUnit 4)

Unit tests are located in the same cartridge as the functionality they test.

Run unit tests using:

• the Intershop Studio or
• via Gradle (developer tools)

Integration Tests

Integration tests require a fully deployed server and are executed within that server, so they have direct access to the classes and objects in the JVM.

Super classes for Integration tests are:

• com.intershop.tools.etest.server.UnitTestCase(sic!)
• com.intershop.tools.etest.remote.CAPITestCase

Integration tests are located in the accompanying test cartridge of the cartridge being tested (bc_pricing_test for bc_pricing).

Run integration tests using:

• the Intershop Studio
• the Test Runner

System Tests

System tests run outside the server and access it via any remote interface (e.g., via HTTP). Technically they are run like unit tests, but they require additional Java properties to know how to connect to a server.

Super classes for system tests are:

• tests.remote.com.intershop.testfw.util.ucm.UCMHTTPTestCase

System tests are currently all located in ucm_tests in component set tests.

Run integration tests using:

• the Intershop Studio
• the Test Runner

Recipe: Create a Cartridge

Problem

Developers need to add new cartridges to an existing component set.

Solution

To add a new cartridge to an existing component set perform the following overall steps:

1. Check out an existing component set and import it to Intershop Studio workspace (see according recipes).
2. Create a new cartridge using Intershop Studio.
3. Fill the build.gradle of new cartridge.

4. Add the cartridge to build.gradle file of the assembly and specify the order.
5. Publish the component set and redeploy the assembly

Please see discussion below for details on how to perform these steps.

It is not necessary to register the cartridge inside the component set in any way. Instead all sub-folders not explicitly excluded in <COMPONENTSET>/settings.gradle are treated as components and included in the build. This way upon commit they are also included in the build of the continuous integration server automatically.

Other non-Gradle related steps, like adding the cartridge to an application type may still be necessary.

Discussion

Create a New Cartridge Using Intershop Studio

1. From the menu select File | New | Cartridge.
2. Enter cartridge and Java package name.
3. Click Next.
4. Use the template Default (Gradle Support) (should be already selected).
   Properties like organization and version are now specified by the component set.
5. Click Finish.

Fill build.gradle

The build.gradle file of a cartridge is both, the build script and the build configuration file for that cartridge. Reusable build functionality comes in Gradle build plugins. Different kinds of artifacts are handled by different plugins. The Gradle build will fail, if you apply a plugin, but there are no matching source files to process.

Intershop Studio generates an initial build.gradle file that is suitable for cartridges containing Java source code and static files. Please refer to Cookbook - Gradle Build Tools | Configure the Build of a Component / Cartridge on how to configure your build.gradle file.

...
Execution failed for task ':<cartridge>:isml2classMain'
> taskdef class com.intershop.beehive.isml.capi.ISML2JSP cannot be found using the classloader AntClassLoader
....

Declare Dependencies

When declaring dependencies you need to distinguish between:

1. Dependencies to cartridges in the same component set.
   These are also called project dependencies. Declare them using the syntax compile project(':<cartridge name>') inside the dependencies block.
2. Dependencies to cartridges in another component set.
   These are called external dependencies. Declare them using the syntax compile group: '<cartridge organization>', name: '<cartridge name>'

The following example demonstrates the different declarations.

dependencies {
    compile group: 'com.intershop', name: 'tools'
	compile project(':customer_base')
} 

As already used from dependency management with Ivy the compile classpath is resolved intransitively: If you specify a dependency to the core cartridge you will not automatically have a dependency to tools. So you will not have tools in your classpath unless you declare the dependency explicitly.

Add New Cartridge to Assembly

Please follow the Recipe in Cookbook - Gradle Assembly Tools | Add Cartridges to an Assembly.

Publish and Deploy

To let the changes take effect locally, publish the component set and redeploy your assembly, as described in Recipe: Make Changes Effective on the Server. Make sure to publish the whole system component set by running the publish task on the system component set, not just the added cartridge.

Recipe: Remove a Development Environment

Problem

The developer needs to remove the development environment when finished to free system resources.

Solution

1. Stop processes running on the deployed server. (Optional for services - as they are stopped automatically later.)
2. Run Gradle task undeploy on the assembly. This first execution generates an uninstall script for the services.
3. Run as administrator / with root privileges (as requested in the console output): <SERVER>\postInstall\uninstallServices.bat. This stops and removes the services.
4. Run Gradle task undeploy on the assembly again. This purges your server directory.
5. Delete the directories <REPO>, <WORKSPACE>, <JAVADOC>, your VCS working copies <ASSEMBLY> and <COMPONENTSET> and finally <GRADLE_USER_HOME>.

Recipe: Clean Gradle Cache

Problem

The Gradle cache located <GRADLE_USER_HOME>/cache currently grows indefinitely and may eventually fill the hard disk.

Solution

On Windows:

1. Close Intershop Studio if open.
2. Open a developer console and go to <COMPONENTSET>.
3. Run gradlew --stop.
4. Run SET GRADLE_USER_HOME=%USERPROFILE%\.gradle.
5. Run gradlew --stop (see discussion for why this has do be done twice).

6. Reset GRADLE_USER_HOME by running gradle_environment.bat.
7. Delete the folder <GRADLE_USER_HOME>/cache using an OS command.

On Linux:

1. Close Intershop Studio if open.
2. Open a developer console and go to <COMPONENTSET>.
3. Run ./gradlew --stop.
4. Run export GRADLE_USER_HOME=$HOME/.gradle.
5. Run ./gradlew --stop (see discussion for why this has do be done twice).

6. Reset GRADLE_USER_HOME by running gradle_environment.sh.
7. Delete the folder <GRADLE_USER_HOME>/cache using an OS command.

Discussion

Cache Contents and Cleaning

The Gradle cache contains:

• Artifacts downloaded from remote repositories, like binaries of cartridges used as dependencies, but also deployment and build plugins. This is by far the largest part.
• Precompiled gradle scripts.

Gradle has no feature for selectively cleaning the artifact cache so far: http://forums.gradle.org/gradle/topics/managing_artifact_cache_growth.

In the meantime, you have to revert to completely deleting the Gradle cache occasionally as described above. The disadvantage of this approach is that all artifacts required for further build and deployment processes will be downloaded again.

Stopping Daemons

Before deleting the cache directory all processes that hold file handles into it must be stopped, i.e., all Gradle daemons. You can do so by calling gradlew --stop.

You can see all running Gradle daemons (not just of the current Gradle user home) by running jps. Example of output:

19452 lilith.jar
22244 Jps
17448 GradleDaemon
13520
15852 GradleDaemon
16772
3468 GradleDaemon
22276