ANT Contrib & Custom Tasks

Contrib tasks

Lets see how we can create a contrib tasks in ANT first and then we proceed to custom tasks. To create contrib tasks in ANT first we have to create ant-contrib plug in and drop it in target platform's directory. This topic meant for advanced Eclipse Plugin Developers.

Download the ant-contrib jar to some location in local machine.

Create a new plugin project using eclipse.

Create a lib folder parallel to META-INF file and, place the ant-contrib jar over there.

Goto Eclipse plugin editor, and add lib/ant-contrib.jar  to the bundle classpath.

Once you are done here, add org eclipse ant core as dependent plugin by going to dependents tab.This will help in identifying extensions of ant core defined in ant core plugin.

We will be using ExtraClassPathEntries extension and AntTask extension.

Curretnly ant-contrib  defines the following tasks:

for   foreach  with parallel execution benefit

For each task that you want to use, define a mapping of task against classname defined in the ant-contrib jar for that task.

After defining all the need tasks that you want to use in your build system, add the lib/ant-contrib.jar in to the extra class path entries section of the manifest file.

Here is a sample and partial manifest file after above change.

<extension point="org.eclipse.ant.core.antTasks">

      <antTask 

           library="lib/ant-contrib.jar" 

           name="foreach" 

           class="net.sf.antcontrib.logic.ForEach"> 

      </antTask> 

     <antTask 

          library="lib/ant-contrib.jar" 

          name="for" 

          class="net.sf.antcontrib.logic.For">

     </antTask>

</extension>

<extension point="org.eclipse.ant.core.extraClasspathEntries"> 

     <extraClasspathEntry 

               library="lib/ant-contrib.jar"> 

      </extraClasspathEntry> 

</extension>

If everything is configured properly, you should be able to pick the defined ant tasks in the ant contrib jar from the Eclipse's plugin editor extension tab.

Once all of the tasks are added in the extension section of the manifest, the very important step is to remove the lib/ant-contrib.jar from the runtime bundle classpath. This is to make sure, we do not include ant-contrib.jar into the class path of the manifest classloader.  

Note that all the task defined in ant-contrib.jar need to be loaded by Ant class loader defined in org eclipse ant core plugin.   Otherwise  you will see all kinds weird error messages from the ant builder in eclipse plugins. 

At this point we are ready to export this project as ant contrib plugin. Goto File menu, select export and export the project as deployable plugin.  Select any destination directory. Once exported you will have plugin with ant-contrib jar inside.

How to use this plugin:

Drop this jar into the plugin's folder of your target platform's build system. Modify build.xml to define ant contrib xmlns as follows..

build.xml sample -

<project default="some.default.target" xmlns:ac="antlib:net.sf.antcontrib">

...

..

</project>

Note that   above defined as prefix namespace for antcontrib. 

Now use ac prefix to call the relevant ant contrib tasks anywhere in the build.xml 

Sample is like the following:

   <target name="iterate">

      <ac:for param="file">

         <fileset dir="."/>

         <sequential>

            <echo message="- @{file}"/>

         </sequential>

      </antcontrib:for>

   </target>

At this point, we should be able to run the build successfully calling tasks defined in ant contrib.

CUSTOM TASKS

Steps to create custom ANT  tasks.

1. Create a new Java Project in eclipse. Right click on Project->Properties->Java Build Path->Add external Jar. Browse to Ant.jar file in ANT_HOME/lib folder.

2.Make a new class say Hello which has to extend class org.apache.tools.ant.Task

import org.apache.tools.ant.BuildException;  

import org.apache.tools.ant.Project;  

import org.apache.tools.ant.Task;  

  

public class Hello extends Task 

{  

 private String retproperty = null;  

 private String msg;  

  

 public String getRetProperty() {  

  return retproperty;  

 }  

  

 public void setRetProperty(String property) {  

  this.retproperty = property;  

 }  

  

 public void setMsg(String msg) {  

  this.msg = msg;  

 }  

  

 public void execute() throws Exception {  

  System.out.println(msg);  

  Project project = getProject();  

  project.setProperty(retproperty,  

    "Value Returned By Custom task");  

 }  

}   

3. Once you have the java code ready in eclipse click on File=>Export=>Choose Jar and export to any suitable location on your system 

4. Now it’s time to write Ant script which will call this custom task pass a value and also read the value returned by the task.So write a similar Ant script. Here classpath should be location where jar in Step 3 was exported. 

<project name="CustomTask" default="build" basedir=".">  

  <taskdef name="Hello" classpath="../../CustomTask.jar" classname="HelloWorld">  

  <target name="build">  

   <helloworld retproperty="val" msg=" Custom Ant Task">  

   <echo message="${val}">  

  </echo></helloworld></target>  

</taskdef></project> 

5.Now we can run the Ant script through command prompt or eclipse as per convenience. 

<span style="font-family: Arial;">

<hello retproperty="val" msg="Custom Ant Task">

</hello>

</span>

This executes method of the HelloWorld class is invoked and msg variable is initialised with value passed. Similarly

project.setProperty(retproperty,"Value Returned By Custom task");  

The above line sets the value into property value  which is again defined in ret property. So that in Ant script variable val will hold the value returned or set inside the class so we can use it for further processing.

SVN Branching,Merging commands

           As most of used SVN on windows we would know mergeing, Branching etc on windows with the help of graphical user interface. But sometimes we also need to work on UNIX/LINUX environment and in that case if we want to use SVN we would need to know the commands to create branch, merge branch, check in, check out etc. In this post let me show some of commands used for creating branch, merging branch into trunk, updating branch.

1. Merge a Branch into Trunk

a. Check out the trunk:


svn co svn+ssh://server/path/to/trunk


b. Check out a copy of the branch you are going to merge:


svn co svn+ssh://server/path/to/branch/myBranch


c. Change your current working directory to “myBranch” .Find the revision “myBranch” began at:


svn log --stop-on-copy

This should display back to you the changes that have been made back to the point the branch was cut. Remember that number (should be rXXXX, where XXXX is the revision number).

d. Change your current working directory to trunk # Perform an SVN update:


svn up

This will update your copy of trunk to the most recent version, and tell you the revision you are at. Make note of that number as well (should say “At revision YYYY” where YYYY is the second number you need to remember).

e. Now we can perform an SVN merge:


svn merge -rXXXX:YYYY svn+ssh://server/path/to/branch/myBranch


This will put all updates into your current working directory for trunk.

f. Resolve any conflicts that arose during the merge and Check in the results:


svn ci -m "MERGE myProject myBranch [XXXX]:[YYYY] into trunk"


That is it. You have now merged “myBranch” with trunk.

2. Branch Creation

a. Get the head revision


svn info svn://server.com/svn/repository/trunk | grep Revision


b. Create a branch

svn cp svn://server.com/svn/repository/trunk \ svn://server.com/svn/repository/branches/your_branch \ -m "Branching from trunk to your_branch at HEAD_REVISION"


c. Swtich to new branch


svn switch --relocate \ svn://server.com/svn/repository/trunk \ svn://server.com/svn/repository/branches/your_branch

d. Update you current directory


svn info | grep URL svn up



e. commit your new changes

3. Updating the branch

a. Update your current branch

b. Search the subversion logs to check what revision number last merged.

svn log --limit 500 | grep -B 3 your_branch

c.Get the head revision

svn info svn://server.com/svn/repository/trunk | grep Revision

d. Merge the difference of the last merged revision on trunk and the head revision on trunk into the your branch working copy

svn merge -r LAST_MERGED_REVISION:HEAD_REVISION \ svn://server.com/svn/repository/trunk .
Replace LAST_MERGED_REVISION with the revision number you noted in step 2, and HEAD_REVISION with the revision number you noted in step 3.


Now look for errors in the output. Could all files be found? Did things get deleted that shouldn’t have been? Maybe you did it wrong. If you need to revert, run svn revert -R *.


e. Check for conflicts.


svn status | egrep '^C|^.C'


Resolve any conflicts. Make sure the application starts and the tests pass.


f. Commit merge changes.


svn ci -m "Merged changes from trunk to your_branch"

Datasource setup on Websphere

In my previous post about "Data sources and Advantages" I have explained about Datasources and It's advantages over Driver manger in JAVA/J2EE based applications. In this post I will show how to setup DataSources on Websphere application server, which can be used in applications using JNDI name.

Following are the steps to create Datasources on Websphere :

1.Start the WebSphere Application Server and login to administrative console.

2. Click Security -> Global security. Under Authentication, expand Java Authentication and Authorization Service and click J2C authentication data.

3. Click New and enter the Alias, User ID and Password,click Ok.(This will be the user name and password of DB).

4. On the administrative console, expand Resources. Expand JDBC then click JDBC Providers.In the Scope section, choose the appropriate scope depending on the usability of Datasource.

5.Click New to create a new JDBC driver. Select, the Database type, Provider type, Implementation type and Name. The Name automatically fills based on the implementation type chosen.

6.Click Next and configure the database class path. Click Next.

7.On the Summary page, click Finish.Click Save to save your selections to Master configurations. The JDBC providers page then appears.

8.On the administrative console, click Data sources. Click New to create a new data source. Enter the Data source name and the JNDI name, and choose the authentication alias from the drop-down list in Component-managed authentication alias. This JNDI name will be used in the applications.

9. Click Next and select the JDBC provider which was created earlier and click Next.

10. Enter the Database name and and server name where DB is located.Deselect the checkbox, Use this data source in container managed persistence (CMP). Click Next.

11. Select Component-managed authentication alias and Container-managed authentication alias.

12.On the Summary page, click Finish.

13.The Data sources page displays, Click Save. Click Test Connection. The message should indicate that the connection is successful. Ignore any warnings.

Continuous Integration with Jenkins

Jenkins is an open source tool which helps in Continuous integration. Continuous integration is a process in which all development work is integrated at a predefined time or at a regular defined interval or through a trigger and the resulting work is automatically tested and built. The basic functionality of Jenkins is to poll a version control tool, if any changes are there, build the code using a build tool like  Apache Ant or Maven. Jenkins monitors the build process reports success or errors to the administrator and /or the code comitters.

The functionality of Jenkins can be extended by using plug-ins for specific tasks, for example report generation.

Various tools that help in implementing the CI with Jenkins are:

Code Repositories :  SVN, Mercurial, Git

Test Frameworks : JUnit,Cucumber, CppUnit

Artifact Repositories : Nexus, Artifactory

Using Jenkins:

Download the jenkins.war file from Here. And copy to Webapps folder of tomcat. Start Tomcat, and Jenkins installation will be available under http://localhost:8080/jenkins

We need to have accessible code repository in a version control tool like SVN/GIT/ClearCase. A plugin is used to connect to the code repository. A working set of build scripts in Ant/Maven. Artifact repository like Nexus(if using Maven) and application deployment destination (WebSphere). Build jobs are set up in jenkins to cater to a integration branch(ideal scenario). In the build job, the poll time is defined (say 10 mins). 

Jenkins also supports master-slave setup. In which, there will be a master and multiple slaves to distribute the build job load. Slave set up has to be done on the systems requires. Cross OS set ups are also supported. For example: Master being on Linux, slaves can be set up on windows systems. Jobs can be linked to built by a specific jenkins slave. A “master” is an installation of Jenkins. When you weren’t using the master/slave support, a master was all you had. Even in the master/slave mode, the role of a master remains the same. It will serve all HTTP requests, and it can still build projects on its own.Slaves are computers that are set up to build projects for a master. Jenkins runs a separate program called “slave agent” on slaves. In other words, there is no need to install the full Jenkins (package or compiled binaries) on a slave node. 

When slaves are registered to a master, a master starts distributing loads to slaves. The exact delegation behavior depends on configuration of each project. Some projects may choose to “stick” to a particular machine for a build, while others may choose to roam freely between slaves. For people accessing Jenkins website, things works mostly transparently. You can still browse javadoc, see test results, download build results from a master, without ever noticing that builds were done by slaves.  In other words, the master becomes a sort of “portal” to the entire build farm.

Jenkins has a built-in SSH client implementation that it can use to talk to remote sshd and start a slave agent. This is the most convenient and preferred method for Unix slaves, which normally has sshd out-of-the-box. Click Manage Jenkins, then Manage Nodes, then click “New Node.” In this set up, you’ll supply the connection information (the slave host name, user name, and ssh credential). Note that the slave will need the master’s public ssh key copied to ~/.ssh/authorized_keys. Jenkins will do the rest of the work by itself, including copying the binary needed for a slave agent, and starting/stopping slaves. If your project has external dependencies (like a special ~/.m2/settings.xml, or a special version of java), you’ll need to set that up yourself, though.

This is the most convenient set up on Unix. However, if your are on Windows and you don’t have ssh commands with cygwin for example, you can use a tool like PuTTY and PuTTYgen to generate your private and public pair of keys.For Windows slaves, Jenkins can use the remote management facility built into Windows 2000 or later. In this set up, you’ll supply the username and the password of the user who has the administrative access to the system, and Jenkins will use that remotely create a Windows service and remotely start/stop them.This is the most convenient set up on Windows, but does not allow you to run programs that require display interaction.

Another way of doing this is to start a slave agent through Java Web Start (JNLP). In this approach, you’ll interactively logon to the slave node, open a browser, and open the slave page. You’ll be then presented with the JNLP launch icon. Upon clicking it, Java Web Start will kick in, and it launches a slave agent on the computer where the browser was running.This mode is convenient when the master cannot initiate a connection to slaves, such as when it runs outside a firewall while the rest of the slaves are in the firewall.

If you need display interaction (e.g. for GUI tests) on Windows and you have a dedicated (virtual) test machine, this is a suitable option. Create a jenkins user account, enable auto-login, and put a shortcut to the JNLP file in the Startup items (after having trusted the slave agent’s certificate). This allows one to run tests as a restricted user as well.

This launch mode uses a mechanism very similar to Java Web Start, except that it runs without using GUI, making it convenient for an execution as a daemon on Unix. To do this, configure this slave to be a JNLP slave, take slave.jar as discussed above, and then from the slave, run a command like this:

$ java -jar slave.jar -jnlpUrl http://yourserver:port/computer/slave-name/slave-agent.jnlp

Some slaves are faster, while others are slow. Some slaves are closer (network wise) to a master, others are far away. So doing a good build distribution is a challenge. Currently, Jenkins employs the following strategy:

1. If a project is configured to stick to one computer, that’s always good.
2. Jenkins tries to build a project on the same computer that it was previously built.
3. Jenkins tries to move long builds to slaves, because the amount of network interaction between a master and a slave tends to be logarithmic to the duration of a build (IOW, even if project A takes twice as long to build as project B, it won’t require double network transfer.) So this strategy reduces the network overhead.

Node Monitoring:

Jenkins has a notion of a “node monitor” which can check the status of a slave for various conditions, displaying the results and optionally marking the slave offline accordingly. Jenkins bundles several, checking disk space in the workspace; disk space in the temporary partition; swap space; clock skew (compared to the master); and response time.The builds can be triggered in many ways. As we have seen, due to SCM polling, url triggers, Fixed interval trigger etc.

Access to jenkins can also be made secure by using jenkins own database – by setting up user permissions by creating logins and using matrix security or by integrating with enterprise LDAP. Alternatively security can be delegated to the servlet container or use unix user/group database.It is recommended to secure Jenkins. Manage Jenkins and then Configure Global Security. Select the Enable securityflag. The easiest way is to use Jenkins own user database. Create at least the user “Anonymous” with read access. Also create entries for the users you want to add in the next step.

Quiet Period:

Commits often come in a burst. This seems to happen mainly for two reasons — people sometimes forget to commit some files, and in the tranquility of waiting for your SCM to finish a commit, people sometimes realize the problems in the commit and they quickly make follow-up changes. The conventional wisdom is that the CI server should wait for the burst to finish before attempting a build. This is said to reduce the chance of having broken build, and it is also sometimes useful in reducing the average turn-around time for builds that take longer.

As such, Hudson is capable of waiting for a commit burst to be over before it triggers a new build, and this feature is called “quiet period.” There are two parts in Hudson that interacts with the quiet period. One is the SCM polling behavior and the other is the queue.

The queue portion of the quiet period is straight-forward. When a build is scheduled into the queue with quiet period, the build will sit in the queue until the quiet period expires. If during this period, additional attempts are made to put the same build in the queue, the quiet period resets to its initial value. For example, if the quiet period is 5 minutes, and the build is put into the queue 9:00am and 9:03am, the actual build will only happen after 9:08am.

Finger Print Tracking:

When you have interdependent projects on Jenkins, it often becomes hard to keep track of which version of this is used by which version of that. Jenkins supports “file fingerprinting” to simplify this.

For example, suppose you have the TOP project that depends on the MIDDLE project, which in turn depends on the BOTTOM project. You are working on the BOTTOM project. The TOP team reported that bottom.jar that they are using causes an NPE, which you thought you fixed in BOTTOM #32. Jenkins can tell you which MIDDLE builds and TOP builds are using (or not using) your bottom.jar #32. The fingerprint of a file is simply a MD5 checksum. Jenkins maintains a database of md5sum, and for each md5sum, Jenkins records which builds of which projects used. This database is updated every time a build runs and files are fingerprinted.

To avoid the excessive disk usage, Jenkins does not store the actual file. Instead, it just stores md5sum and their usages. These files can be seen in $JENKINS_HOME/fingerprints.

Splitting Jobs:

A build is normally a fairly sequential process, and in a big project, a full execution can easily take hours. While one could bring such a job on Jenkins, a long turn-around time to the result tends to reduce the value of continuous integration. This page discusses a technique to cope with this problem.

The idea to is to split a big build into multiple stages. Each stage is executed sequentially for a particular build run, but this works like a CPU pipeline and increase the throughput of CI, and also reduces the turn-around time by reducing the time a build sits in the build queue.

In this situation, your earlier stage needs to pass files to later stages. A general way to do this is as follows:

1. An earlier stage archives all the files into a zip/tgz file at the end of the build.
2. Tell Jenkins to archive this zip/tgz file as a post-build action, take a fingerprint of it, then trigger the next stage.
3. The first thing the next stage does in its build is to obtain this bundle through the permalink for the last successful artifact, then unzip it. Do keep this archive file around because we’ll take a fingerprint of it here, too.
4. The build proceeds by using the files obtained from the earlier stage.
5. Tell Jenkins to fingerprint the zip/tgz file. This allows you to correlate executions of these stages to track the flow.

If you have more than 2 stages, you can repeat this process. In some cases, this “zip/tgz” file would have to contain the entire workspace.

Jenkins Backup:

Jenkins stores all the settings, logs and build artifacts in its home directory, for example, in /var/lib/jenkins under the default install location of Ubuntu.To create a backup of your Jenkins setup, just copy this directory.

The jobs directory contains the individual jobs configured in the Jenkins install. You can move a job from one Jenkins installation to another by copying the corresponding job directory. You can also copy a job directory to clone a job or rename the directory.

Click reload config button in the Jenkins web user interface to force Jenkins to reload configuration from the disk.

As we have understood by now, Jenkins is the most widely used CI tool and is an industry standard. This is supported by the widely available plugins which ease many tasks. Setting up and configuring jenkins is a breeze. Being in development for quite sometime, it is very stable and has a great online community support with some companies providing paid support as well.

One interesting feature is the multi-configuration project. A multi-configuration project is useful for instances where your builds will make many similar build steps, and you would otherwise be duplicating steps. The configuration matrix allows you to specify what steps to duplicate, and create a multiple-axis graph of the type of builds to create.

ClearCase UCM Project Creation

In my previous post I have explained about UCM project and different terminologies used in ClearCase. In this post I will show how to create a UCM project in IBM Clear Case. Creation of a UCM Project is done through the ClearCase Project Explorer. The Project Explorer can be started through the Start menu in Windows or through the ClearCase Explorer on the UCM page of the Toolbox tab.

Once the Project Explorer is started, right click on PVOB where the project is to be created and select New Project…This will start the New Project Wizard. (If the PVOB is not visible in the Project Explorer, select View -> Show all Project VOBs.).

In the New Project Wizard, enter the UCM Project name. It should be descriptive but should also be concise, since the play a role in the path to the elements in ClearCase. The Integration Stream Name will automatically be populated with UCM Project name and appended with “_Integration”. It is recommended to shorten this to “_Int” to keep the path name to the elements as short as possible. A short description should be entered that gives information about the purpose of the UCM Project.

Determine if this is to be a single stream (only an integration stream) project or a multistream project (Traditional). Select the option which corresponds to the development environment you are trying to create. Click Next.

At this point, it must be decided if the UCM Project is to be based on an existing project or created as a new entity. Select No to create a new UCM Project that is not based on an existing UCM Project, Click Next.

If the new UCM Project is to be based on the recommended baselines of an existing project, select Yes and highlight the integration stream of the UCM Project the new project will use as a base. If the PVOB that is needed does not appear in the window, select “Show all UCM Project VOBs”. Click Next.

In this Wizard, the components that will be included in the UCM Project will need to be selected.

Based on previous UCM Project:

If the new UCM Project was based on an existing UCM Project, the components and their baselines will be displayed in the window. Check these to make sure they are what are expected. Components can be added deleted if necessary by highlighting the component and selecting the Remove button. If the displayed component has the incorrect baseline, highlight the component and select Change. This will bring up a dialog similar to adding a component (see below). To add additional components, follow the steps for adding a component (below). If all components are correct, click Next.

New UCM Project:

If the UCM Project is not based on an existing UCM Project, components must be added to the project. To add a component to the UCM Project, select Add… This will bring up the Add Baseline window. 

From the drop down menu, select the component to be added to the UCM Project. Once the component is selected, select Change >> All Streams from “From Stream:” This will display all the available baselines for the component that was selected.

Highlight the baseline that is to be used as the basis for the UCM Project and then select OK. This will add the component to the UCM Project.

Repeat the above steps until all the components have been added to the UCM Project. Then click Next. This will display following wizard.

At this point, the components that are to be modifiable in the UCM Project must be selected. If a component is checked, the component is modifiable. If not checked, the component will be read-only in the UCM Project. 

Based on previous UCM Project:

If the UCM Project is based on a previous project, the components will be configured according to the old UCM Project. Modify this as necessary for the new UCM Project.

New UCM Project:

By default, all components are set to read-only. Check the components that will be modifiable in the UCM Project.

Once the components have been configured, select the Policies button. This will bring up the Project Policies window.

Enable the first to policies and select OK and then select Next.

Make sure No is selected and select Finish.

Review the information in the New Project – Confirmation window. If all the information is correct, select OK. This will create the new UCM Project.