Introduction

This guide explains the configuration steps for configuring Intershop 7 to transfer system data to the Intershop Commerce Insight (ICI). It also provides solutions for possible problems that may arise during the configuration.

Since the ICI is a remote application, the system data on which the ICI reports are based must be collected in Intershop 7 and is then transferred to the ICI. The framework used to realize this process makes use of three basic techniques:

• Data aggregators to collect the data,
• SSH to protect the data during transfer and
• Scheduled pipelines to automate the data transfer.

The configuration of Intershop 7 for the ICI involves two main aspects:

• Preparing the SSH communication and
• Configuring the data aggregators.

References

• Reference - System Monitoring

Configuration Steps

Generate a Private/Public Key Pair

The key pair is used to ensure secure data transfer from the local installation to the remote ICI Web server.

Windows

To generate the private/public key pair when using Windows perform the following steps:

1. Open a command prompt (cmd.exe) and change to %IS_HOME%\bin, where %IS_HOME% represents the Intershop 7 root directory.
2. Run the batch file environment.bat to set the system environment.

3. Change to the directory %IS_HOME%\tools\misc and execute the following command:

   ant ssh_keygen -Dusername="<User>" -Dpassphrase="<MyPassphrase>"

   Info

   Replace <User> with a name of your choice, preferably the installation's name or the company name, and <MyPassphrase> with a passphrase of your choice. Keep this passphrase secret.

Unix

To generate the private/public key pair when using Unix perform the following steps:

1. Change to the isas# user, e.g., su - isas1
2. Change to the directory $IS_HOME/tools/misc/, where $IS_HOME represents the Intershop 7 root directory.

3. Execute the following command:

   ../ant/bin/ant ssh_keygen -Dusername="<User>" -Dpassphrase="<MyPassphrase>"

   Replace <User> with a name of your choice, preferably the installation's name or the company name, and <MyPassphrase> with a passphrase of your choice. Keep this passphrase secret.

Send Public Key and User Name (<User>) to Intershop Customer Support

You can find the key file report-ssh-key.pub in $IS_SHARE/system/config/cluster/.

Intershop Customer Support will create your ICI user account based on the public key.

Fine Tune Reporting-related Properties

1. Edit the file appserver.properties in $IS_SHARE/system/config/cluster/.
   The following settings are optional but can become necessary for large WA-<your_server>-<date>.log files:

   # 
   # WebAdapter log file pushing
   #
   # WA log file pushing is not allowed in the time
   # from HOUR (0-23) (to prevent excessive log file data transfer during high-traffic  
   # times over the Internet and on the hard disk)
   intershop.WaLogPush.notAllowedFrom=9
   # to HOUR (0-23)
   intershop.WaLogPush.notAllowedTo=19
   #
   # maximum WA log file transfer rate in MB/h
   intershop.WaLogPush.maxTransferRate=10000 

   Detailed information on this topic can be found in the reporting readme file report-config.txt located in $IS_SHARE/system/config/cluster/.

2. Go to the SMC and activate and adjust the reporting schedules after Intershop Customer Support has notified you that your ICI account has been created. Set the schedules as follows:

   • Report_AppServerLogs: start time around 1:00 am, interval every 1 day.

   • Report_WALogs: start time around 1:30 am, interval every 1 day.

   • Report_TransferData: start time around 3:00 am, interval every 1 day.

3. To ensure all three jobs work as expected, trigger the scheduled pipelines one by one manually and check their output.

Troubleshooting: Data Transfer

Sometimes data transfer doesn't work or stops working. Possible reasons are a firewall blocking the data transfer, the SSH keys are not correct or the corresponding jobs got disabled.

In case the data transfer does not work, please check the following points in the given order:

• Run the Job Report_AppServerLogs.
  Does the job write files to $IS_SHARE/reportingrepository/AppServerLogs/data? If no files are written, consult the error*.log file, and the job's log file in $IS_SHARE/system/log for possible error messages.

• Run the Job Report_WALogs.
  Does the job write files to $IS_SHARE/reportingrepository/WALogs/data? If no files are written, check if there exist recent WA-*.log files in $IS_SHARE/system/log.
  If there exists a WA-*.log, consult the error*.log file, and the job's log file in $IS_SHARE/system/log for possible error messages.

• Run the Job Report_TransferData.
  Were the files transferred to the ICI? If not, check the error*.log file, and the job's log file in $IS_SHARE/system/log for possible error messages. The most frequent error type is connection problems resulting from a closed firewall. Contact your system administrator to have the firewall opened at port 22 for outgoing SSH connections to IP 195.110.60.151, upload.insight.intershop.com.

• If log entries suggest authentication problems contact Intershop Customer Support.

FAQ

What Reporting-related Jobs Are Available?

There exist two types of reporting jobs: aggregator jobs and the transfer job.

Aggregators, defined in report-config.xml in $IS_SHARE/system/config/cluster/, are used to dynamically generate jobs in the SMC, named starting with Report_*. These aggregator jobs are responsible for collecting system data, e.g., log files or database content, that are then automatically copied to $IS_SHARE/reportingrepository/<aggregator name>/data/ where the data is packed into a *.zip file.

The transfer job Report_TransferData, which writes logging information to $IS_SHARE/system/log/job-*.log files, is responsible for transferring the collected and compressed data files. All new or changed files stored in $IS_SHARE/reportingrepository/ are transferred to the Intershop Commerce Insight. If the files do not exist locally anymore, the job will remove them from the ICI as well.

What Is the Data Transfer Volume?

As soon as the daily allowance of data transfer volume is reached - 1GB (zipped) for ICI Basic contracts - the transfer is automatically being cut.

The customer has to take care of the configuration of the daily volume of transfer data.
This limit will not affect the usefulness of the ICI as monitoring tool. If the transfer was cut, the scale values display a percentage of the total data. Tendencies will be valid, but not absolute in values.

Intershop Customer Support offers an extended ICI contract which includes a higher limit of daily data transfer volume. If you are interested in this offer, contact our Support Admin.

What is the Maximum Age of the Files That Can be Processed by the ICI?

The data in their raw form, i.e., which are not aggregated, are only kept for a limited time (=archive time) within the ICI.

The default value is 35 days. It can be less for large installations or during complex maintenance tasks (or more in mutual agreements).
The Intershop Commerce Insight does not save the log files after the import. This means, Intershop cannot provide backup of customer files which are not located on the upload server anymore.
As long as the file date is younger than the archive time, the files will be processed automatically.

In case you want to upload older files, please be aware that this produces an increased high load on our servers. A successful upload process should not miss any files and should be finished within 24h.

Can I Replace Incomplete Log Files?

Yes. The repair process is the same, except that the archive time does not apply; even for younger cases you will have to provide all files of the affected days.

What Should I do if the Hourly Transfer Size is Exceeded?

Another reason for a not running WebadapterAgent can be that the hourly transfer size is exceeded because of a big WA-*.log file. We recommend to increase it by setting 200 to 2000 MB/h: 

intershop.WaLogPush.maxTransferRate=2000 in the appserver.properties (appserver restart is required after change).