Continuous Integration with Jenkins

Jenkins is an open source tool which helps in Continuous integration. Continuous integration is a process in which all development work is integrated at a predefined time or at a regular defined interval or through a trigger and the resulting work is automatically tested and built. The basic functionality of Jenkins is to poll a version control tool, if any changes are there, build the code using a build tool like  Apache Ant or Maven. Jenkins monitors the build process reports success or errors to the administrator and /or the code comitters.

The functionality of Jenkins can be extended by using plug-ins for specific tasks, for example report generation.

Various tools that help in implementing the CI with Jenkins are:

Code Repositories :  SVN, Mercurial, Git

Test Frameworks : JUnit,Cucumber, CppUnit

Artifact Repositories : Nexus, Artifactory

Using Jenkins:

Download the jenkins.war file from Here. And copy to Webapps folder of tomcat. Start Tomcat, and Jenkins installation will be available under http://localhost:8080/jenkins

We need to have accessible code repository in a version control tool like SVN/GIT/ClearCase. A plugin is used to connect to the code repository. A working set of build scripts in Ant/Maven. Artifact repository like Nexus(if using Maven) and application deployment destination (WebSphere). Build jobs are set up in jenkins to cater to a integration branch(ideal scenario). In the build job, the poll time is defined (say 10 mins). 

Jenkins also supports master-slave setup. In which, there will be a master and multiple slaves to distribute the build job load. Slave set up has to be done on the systems requires. Cross OS set ups are also supported. For example: Master being on Linux, slaves can be set up on windows systems. Jobs can be linked to built by a specific jenkins slave. A “master” is an installation of Jenkins. When you weren’t using the master/slave support, a master was all you had. Even in the master/slave mode, the role of a master remains the same. It will serve all HTTP requests, and it can still build projects on its own.Slaves are computers that are set up to build projects for a master. Jenkins runs a separate program called “slave agent” on slaves. In other words, there is no need to install the full Jenkins (package or compiled binaries) on a slave node. 

When slaves are registered to a master, a master starts distributing loads to slaves. The exact delegation behavior depends on configuration of each project. Some projects may choose to “stick” to a particular machine for a build, while others may choose to roam freely between slaves. For people accessing Jenkins website, things works mostly transparently. You can still browse javadoc, see test results, download build results from a master, without ever noticing that builds were done by slaves.  In other words, the master becomes a sort of “portal” to the entire build farm.

Jenkins has a built-in SSH client implementation that it can use to talk to remote sshd and start a slave agent. This is the most convenient and preferred method for Unix slaves, which normally has sshd out-of-the-box. Click Manage Jenkins, then Manage Nodes, then click “New Node.” In this set up, you’ll supply the connection information (the slave host name, user name, and ssh credential). Note that the slave will need the master’s public ssh key copied to ~/.ssh/authorized_keys. Jenkins will do the rest of the work by itself, including copying the binary needed for a slave agent, and starting/stopping slaves. If your project has external dependencies (like a special ~/.m2/settings.xml, or a special version of java), you’ll need to set that up yourself, though.

This is the most convenient set up on Unix. However, if your are on Windows and you don’t have ssh commands with cygwin for example, you can use a tool like PuTTY and PuTTYgen to generate your private and public pair of keys.For Windows slaves, Jenkins can use the remote management facility built into Windows 2000 or later. In this set up, you’ll supply the username and the password of the user who has the administrative access to the system, and Jenkins will use that remotely create a Windows service and remotely start/stop them.This is the most convenient set up on Windows, but does not allow you to run programs that require display interaction.

Another way of doing this is to start a slave agent through Java Web Start (JNLP). In this approach, you’ll interactively logon to the slave node, open a browser, and open the slave page. You’ll be then presented with the JNLP launch icon. Upon clicking it, Java Web Start will kick in, and it launches a slave agent on the computer where the browser was running.This mode is convenient when the master cannot initiate a connection to slaves, such as when it runs outside a firewall while the rest of the slaves are in the firewall.

If you need display interaction (e.g. for GUI tests) on Windows and you have a dedicated (virtual) test machine, this is a suitable option. Create a jenkins user account, enable auto-login, and put a shortcut to the JNLP file in the Startup items (after having trusted the slave agent’s certificate). This allows one to run tests as a restricted user as well.

This launch mode uses a mechanism very similar to Java Web Start, except that it runs without using GUI, making it convenient for an execution as a daemon on Unix. To do this, configure this slave to be a JNLP slave, take slave.jar as discussed above, and then from the slave, run a command like this:

$ java -jar slave.jar -jnlpUrl http://yourserver:port/computer/slave-name/slave-agent.jnlp

Some slaves are faster, while others are slow. Some slaves are closer (network wise) to a master, others are far away. So doing a good build distribution is a challenge. Currently, Jenkins employs the following strategy:

1. If a project is configured to stick to one computer, that’s always good.
2. Jenkins tries to build a project on the same computer that it was previously built.
3. Jenkins tries to move long builds to slaves, because the amount of network interaction between a master and a slave tends to be logarithmic to the duration of a build (IOW, even if project A takes twice as long to build as project B, it won’t require double network transfer.) So this strategy reduces the network overhead.

Node Monitoring:

Jenkins has a notion of a “node monitor” which can check the status of a slave for various conditions, displaying the results and optionally marking the slave offline accordingly. Jenkins bundles several, checking disk space in the workspace; disk space in the temporary partition; swap space; clock skew (compared to the master); and response time.The builds can be triggered in many ways. As we have seen, due to SCM polling, url triggers, Fixed interval trigger etc.

Access to jenkins can also be made secure by using jenkins own database – by setting up user permissions by creating logins and using matrix security or by integrating with enterprise LDAP. Alternatively security can be delegated to the servlet container or use unix user/group database.It is recommended to secure Jenkins. Manage Jenkins and then Configure Global Security. Select the Enable securityflag. The easiest way is to use Jenkins own user database. Create at least the user “Anonymous” with read access. Also create entries for the users you want to add in the next step.

Quiet Period:

Commits often come in a burst. This seems to happen mainly for two reasons — people sometimes forget to commit some files, and in the tranquility of waiting for your SCM to finish a commit, people sometimes realize the problems in the commit and they quickly make follow-up changes. The conventional wisdom is that the CI server should wait for the burst to finish before attempting a build. This is said to reduce the chance of having broken build, and it is also sometimes useful in reducing the average turn-around time for builds that take longer.

As such, Hudson is capable of waiting for a commit burst to be over before it triggers a new build, and this feature is called “quiet period.” There are two parts in Hudson that interacts with the quiet period. One is the SCM polling behavior and the other is the queue.

The queue portion of the quiet period is straight-forward. When a build is scheduled into the queue with quiet period, the build will sit in the queue until the quiet period expires. If during this period, additional attempts are made to put the same build in the queue, the quiet period resets to its initial value. For example, if the quiet period is 5 minutes, and the build is put into the queue 9:00am and 9:03am, the actual build will only happen after 9:08am.

Finger Print Tracking:

When you have interdependent projects on Jenkins, it often becomes hard to keep track of which version of this is used by which version of that. Jenkins supports “file fingerprinting” to simplify this.

For example, suppose you have the TOP project that depends on the MIDDLE project, which in turn depends on the BOTTOM project. You are working on the BOTTOM project. The TOP team reported that bottom.jar that they are using causes an NPE, which you thought you fixed in BOTTOM #32. Jenkins can tell you which MIDDLE builds and TOP builds are using (or not using) your bottom.jar #32. The fingerprint of a file is simply a MD5 checksum. Jenkins maintains a database of md5sum, and for each md5sum, Jenkins records which builds of which projects used. This database is updated every time a build runs and files are fingerprinted.

To avoid the excessive disk usage, Jenkins does not store the actual file. Instead, it just stores md5sum and their usages. These files can be seen in $JENKINS_HOME/fingerprints.

Splitting Jobs:

A build is normally a fairly sequential process, and in a big project, a full execution can easily take hours. While one could bring such a job on Jenkins, a long turn-around time to the result tends to reduce the value of continuous integration. This page discusses a technique to cope with this problem.

The idea to is to split a big build into multiple stages. Each stage is executed sequentially for a particular build run, but this works like a CPU pipeline and increase the throughput of CI, and also reduces the turn-around time by reducing the time a build sits in the build queue.

In this situation, your earlier stage needs to pass files to later stages. A general way to do this is as follows:

1. An earlier stage archives all the files into a zip/tgz file at the end of the build.
2. Tell Jenkins to archive this zip/tgz file as a post-build action, take a fingerprint of it, then trigger the next stage.
3. The first thing the next stage does in its build is to obtain this bundle through the permalink for the last successful artifact, then unzip it. Do keep this archive file around because we’ll take a fingerprint of it here, too.
4. The build proceeds by using the files obtained from the earlier stage.
5. Tell Jenkins to fingerprint the zip/tgz file. This allows you to correlate executions of these stages to track the flow.

If you have more than 2 stages, you can repeat this process. In some cases, this “zip/tgz” file would have to contain the entire workspace.

Jenkins Backup:

Jenkins stores all the settings, logs and build artifacts in its home directory, for example, in /var/lib/jenkins under the default install location of Ubuntu.To create a backup of your Jenkins setup, just copy this directory.

The jobs directory contains the individual jobs configured in the Jenkins install. You can move a job from one Jenkins installation to another by copying the corresponding job directory. You can also copy a job directory to clone a job or rename the directory.

Click reload config button in the Jenkins web user interface to force Jenkins to reload configuration from the disk.

As we have understood by now, Jenkins is the most widely used CI tool and is an industry standard. This is supported by the widely available plugins which ease many tasks. Setting up and configuring jenkins is a breeze. Being in development for quite sometime, it is very stable and has a great online community support with some companies providing paid support as well.

One interesting feature is the multi-configuration project. A multi-configuration project is useful for instances where your builds will make many similar build steps, and you would otherwise be duplicating steps. The configuration matrix allows you to specify what steps to duplicate, and create a multiple-axis graph of the type of builds to create.

ClearCase UCM Project Creation

In my previous post I have explained about UCM project and different terminologies used in ClearCase. In this post I will show how to create a UCM project in IBM Clear Case. Creation of a UCM Project is done through the ClearCase Project Explorer. The Project Explorer can be started through the Start menu in Windows or through the ClearCase Explorer on the UCM page of the Toolbox tab.

Once the Project Explorer is started, right click on PVOB where the project is to be created and select New Project…This will start the New Project Wizard. (If the PVOB is not visible in the Project Explorer, select View -> Show all Project VOBs.).

In the New Project Wizard, enter the UCM Project name. It should be descriptive but should also be concise, since the play a role in the path to the elements in ClearCase. The Integration Stream Name will automatically be populated with UCM Project name and appended with “_Integration”. It is recommended to shorten this to “_Int” to keep the path name to the elements as short as possible. A short description should be entered that gives information about the purpose of the UCM Project.

Determine if this is to be a single stream (only an integration stream) project or a multistream project (Traditional). Select the option which corresponds to the development environment you are trying to create. Click Next.

At this point, it must be decided if the UCM Project is to be based on an existing project or created as a new entity. Select No to create a new UCM Project that is not based on an existing UCM Project, Click Next.

If the new UCM Project is to be based on the recommended baselines of an existing project, select Yes and highlight the integration stream of the UCM Project the new project will use as a base. If the PVOB that is needed does not appear in the window, select “Show all UCM Project VOBs”. Click Next.

In this Wizard, the components that will be included in the UCM Project will need to be selected.

Based on previous UCM Project:

If the new UCM Project was based on an existing UCM Project, the components and their baselines will be displayed in the window. Check these to make sure they are what are expected. Components can be added deleted if necessary by highlighting the component and selecting the Remove button. If the displayed component has the incorrect baseline, highlight the component and select Change. This will bring up a dialog similar to adding a component (see below). To add additional components, follow the steps for adding a component (below). If all components are correct, click Next.

New UCM Project:

If the UCM Project is not based on an existing UCM Project, components must be added to the project. To add a component to the UCM Project, select Add… This will bring up the Add Baseline window. 

From the drop down menu, select the component to be added to the UCM Project. Once the component is selected, select Change >> All Streams from “From Stream:” This will display all the available baselines for the component that was selected.

Highlight the baseline that is to be used as the basis for the UCM Project and then select OK. This will add the component to the UCM Project.

Repeat the above steps until all the components have been added to the UCM Project. Then click Next. This will display following wizard.

At this point, the components that are to be modifiable in the UCM Project must be selected. If a component is checked, the component is modifiable. If not checked, the component will be read-only in the UCM Project. 

Based on previous UCM Project:

If the UCM Project is based on a previous project, the components will be configured according to the old UCM Project. Modify this as necessary for the new UCM Project.

New UCM Project:

By default, all components are set to read-only. Check the components that will be modifiable in the UCM Project.

Once the components have been configured, select the Policies button. This will bring up the Project Policies window.

Enable the first to policies and select OK and then select Next.

Make sure No is selected and select Finish.

Review the information in the New Project – Confirmation window. If all the information is correct, select OK. This will create the new UCM Project.

BIG DATA with HADOOP

There has been data explosion over the past few years and is continuing as you are reading this. Every day we generate a lot of data, without being aware of doing so, through  social media websites, digital pictures, Meteorological data, transaction related data, Mobil phone etc. 90% of the of the heap of data that is available today is created in the last 2-3 years. This is called big Data. In other words, Big data is data that exceeds the processing capacity of conventional database systems. The data is too big, moves too fast, or doesn’t fit the strictures of your database architectures. To gain value from this data, you must choose an alternative way to process it.

Big data is characterized by three V’s:  Volume, Velocity and Variety

Volume:

Data volume is resulted in by many factors. Like Transaction-based data stored over the years, social media(eg: 12 Terabytes of data is generated each day by Tweets alone). Call details, Sensor and machine-to-machine data. With decreasing storage costs,data is being collected every data and being analysed to predict trends and sentiments. In other words, making sense out of the vast amount of data.This volume presents the most immediate challenge to conventional IT structures. It calls for scalable storage, and a distributed approach to querying. Many companies already have large amounts of archived data, perhaps in the form of logs, but not the capacity to process it.

Velocity:

Data is being generated at great speeds, and analyzing it as soon as it gets generated is equally important to get benefit out of it in most cases. Say, data collected in an aero plane needs to be analyzed immediately to avoid possible mishaps. Other the data collected is of no use. Another example is to analyze 5 million trade events created each day to identify potential fraud. It’s not just the velocity of the incoming data that’s the issue: it’s possible to stream fast-moving data into bulk storage for later batch processing, for example. The importance lies in the speed of the feedback loop, taking data from input through to decision.

Variety:

Data today comes in all types of formats. For eg: Pictures, Music files all come in multiple formats. Structured, numeric data in traditional databases. Information created from line-of-business applications. Unstructured text documents, email, video, audio, stock ticker data and financial transactions. Rarely does data present itself in a form perfectly ordered and ready for processing. A common theme in big data systems is that the source data is diverse, and doesn’t fall into neat relational structures. It could be text from social networks, image data, a raw feed directly from a sensor source. None of these things come ready for integration into an application.

Even on the web, where computer-to-computer communication ought to bring some guarantees, the reality of data is messy. Different browsers send different data, users withhold information, they may be using differing software versions or vendors to communicate with you. And you can bet that if part of the process involves a human, there will be error and inconsistency. A common use of big data processing is to take unstructured data and extract ordered meaning, for consumption either by humans or as a structured input to an application. One such example is entity resolution, the process of determining exactly what a name refers to. Is this city London, England, or London, Texas? By the time your business logic gets to it, you don’t want to be guessing.

Traditional Data Handling:

Data Storage:

Big data ranges from several terabytes to petabytes and it’s difficult to read/write huge amount of data on a single disk. At these volumes access speed of the data devices will dominate overall analysis time. Have multiple disk drives, split your data file into small enough pieces across the drives and do parallel read and processing.

Hardware reliability (failure of any drive) is a challenge

Resolving data interdependency between drives is a notorious challenge.

Number of disk drives that can be added to a server is limited.

• Analysis on the stored data

Much of Big Data is unstructured. Traditional RDBMS/EDW cannot handle easily. Log of Big Data analysis is adhoc in nature, involves whole data scan, referencing itself, joining, combing etc. Traditional RDBMS/ EDW cannot handle these with their limited scalability options and architectural limitations. You can incorporate better servers, processors and throw in more RAM but there is a limit to it.

Need Some Drastically Different Approach

• A distributed file system with high capacity and high reliability to store large amount of data (like HDFS file system).

• A process engine that can handle structure / unstructured data.

• A computation model that can operate on distributed data and abstract data dispersion (MapReduce model)

Is Hadoop a Right Solution for Big Data?

•Hadoop is an open source Apache project for handling Big Data

•It addresses data storage issue and analysis (processing) issue through

•HDFS file system

•Implementing MapReduce computation model

•It is designed for massive scalability and reliability

•The model enables leveraging cheap commodity servers keeping the cost in check.

At a high-level, Hadoop architectural components can be classified into two categories

Distributed file management system – HDFS

• Name Node: Centrally Monitors and controls the whole file system

• Data Node: Take care of the local file segments & constantly communicates with Name Node

• Secondary Name Node: This just backs up the file system status from the Name Node periodically

Distributed computing system – MapReduce Framework

• Job Tracker:Centrally Monitors the submitted Job and controls all processes funning on the nodes (computers) of the cluster.

This communicated with Name Node for file system access

• Task Tracker:

Takes care of the local job execution on the local file segment.

Talks to Data Node for file information.

This constantly communicates with job Tracker daemon to report the task progress

In other words, Hadoop MapReduce is a software framework for easily writing applications which process vast amounts of data (multi-terabyte data-sets) in-parallel on large clusters (thousands of nodes) of commodity hardware in a reliable, fault-tolerant manner.

A MapReduce job usually splits the input data-set into independent chunks which are processed by the map tasks in a completely parallel manner. The framework sorts the outputs of the maps, which are then input to the reduce tasks. Typically both the input and the output of the job are stored in a file-system. The framework takes care of scheduling tasks, monitoring them and re-executes the failed tasks.

The key aspect of the MapReduce algorithm is that if every Map and Reduce is independent of all other ongoing Maps and Reduces, then the operation can be run in parallel on different keys and lists of data. On a large cluster of machines, you can go one step further, and run the Map operations on servers where the data lives. Rather than copy the data over the network to the program, you push out the program to the machines. The output list can then be saved to the distributed filesystem, and the reducers run to merge the results. Again, it may be possible to run these in parallel, each reducing different keys.

Typically the compute nodes and the storage nodes are the same, that is, the MapReduce framework and the Hadoop Distributed File System are running on the same set of nodes. This configuration allows the framework to effectively schedule tasks on the nodes where data is already present, resulting in very high aggregate bandwidth across the cluster.

The MapReduce framework consists of a single master JobTracker and one slave TaskTracker per cluster-node. The master is responsible for scheduling the jobs’ component tasks on the slaves, monitoring them and re-executing the failed tasks. The slaves execute the tasks as directed by the master.

Minimally, applications specify the input/output locations and supply map and reduce functions via implementations of appropriate interfaces and/or abstract-classes. These, and other job parameters, comprise the job configuration. The Hadoop job client then submits the job (jar/executable etc.) and configuration to the JobTrackerwhich then assumes the responsibility of distributing the software/configuration to the slaves, scheduling tasks and monitoring them, providing status and diagnostic information to the job-client.

The value of big data to an organization falls into two categories: analytical use, and enabling new products. Big data analytics can reveal insights hidden previously by data too costly to process, such as peer influence among customers, revealed by analyzing shoppers’ transactions, social and geographical data. Being able to process every item of data in reasonable time removes the troublesome need for sampling and promotes an investigative approach to data, in contrast to the somewhat static nature of running predetermined reports.

Starting with PERL

         PERL is an acronym for Practical Extraction and Report Language. Perl is a general purpose programming language designed to be used for text processing but is also used for system administration, web development, network programming, GUI development etc. It supports both procedural and object oriented programming and a wide collection of third party modules are available.

          Perl is a programming language which can be used for a large variety of tasks. A typical simple use of Perl would be for extracting information from a text file and printing out a report or for converting a text file into another form. But Perl provides a large number of tools for quite complicated problems, including systems programming.

         Perl was introduced by Larry Wall circa 1987 as a general purpose Unix scripting language to make his programming work simpler.Perl is implemented as an interpreted language. Thus, the execution of a Perl script tends to use more CPU time than a corresponding C program, for instance. As computing became faster now a days writing something in Perl instead of C tends to save time.

To check whether PERL installed on your system already,go to command prompt and type perl -v. If Perl is installed correctly, it will display the version information.

Comments in PERL -

      The symbol # indicates a comment in PERL. Comments help in understanding and debugging code, so it's always a best practice to put useful comment in any programming language while coding/scripting.Good comments are short, but instructive. They tell you things that aren't clear from reading the code. 

A code without proper comment is considered as bad comment, check the following code : Simply by looking at this you wont get anything from this. 

for $i (@q)

 { 

my ($j) = fix($i);

Similarly a code with improper comment also considered as bad code, check this one -

for $i (@q) 

{ 

# @q is list from last sub my 

($j) = fix($i);  # Gotta fix $j... 

 transmit($j); # ...and then it goes over the wire 

 }

Following is considered as good code snippet as it has proper comment. Simply by looking at this we can understand what exactly this part of script doing.

# Now that we've got prices from database, let's send them to the buyer

 for $i (@q) { 

my ($j) = fix($i); # Add local taxes, perform currency exchange 

 transmit($j); 

 }

3 variable types: Scalars, Arrays, and Hashes.

Scalars -- > A scalar represents a single value. Scalar values can be strings, integers or floating point numbers, and Perl will automatically convert between them as required. There is no need to pre-declare variable types.

my $animal = “camel”;

my $answer = 42;

Arrays -- > An array represents a list of values. Arrays are zero-indexed. Here’s how you get at elements in an array. The special variable $#array tells you the index of the last element of an array.

my @animals = (“camel”, “llama”, “owl”);

my @mixed = (“camel”, 42, 1.23);

print $mixed[$#mixed]; # last element, prints 1.23

Hashes -- > A hash represents a set of key/value pairs. You can use whitespace and the => operator to lay them out more nicely. You can get at lists of keys and values with keys() and values().

my %fruit_color = (“apple”, “red”, “banana”, “yellow”);

Conditionals and looping :

The following looping constructs are available in Perl.

• IF
• WHILE
• FOR
• FOREACH

FOREACH is widely used construct to loop through lists without explicitly defining multiple variables.

Built in Operators:

Arithmetic

+ addition, - subtraction, * multiplication, / division

Numeric comparison

== equality, != inequality, < less than, > greater than, <= less than or equal, >= greater than or equal

String comparison

eq equality, ne inequality, lt less than, gt greater than, le less than or equal, ge greater than or equal

Boolean Logic

&& and, || or, ! not

Miscellaneous

= assignment, . string concatenation, x string multiplication, .. range operator (creates a list of numbers)

Apart from these basic features, simple solutions can be arrived at for complex problems using data structures in Perl.In this post, we've only scratched the surface of what Perl can do. PERL can do much more than this, you can get more information on PERL from here.

Regular Expressions in PERL

Regualr Expressions :

           A regular expression (regex or regexp for short) is a special text string for describing a search pattern. You can think of regular expressions as wildcards on steroids. Regular expressions are used when you want to search for specify lines of text containing a particular pattern.

            It is easy to search for a word or a string of characters. You can use ctrl+F in any programming editor or a word processor. Regular expressions are more powerful and flexible. You can specify insane set of conditions to match and regular expressions will do it for you like a breeze. You can search for a word with four or more vowels that end with an “s”. Numbers, punctuation characters, you name it, a regular expression can find it. Even editors support regex search, so it is beneficial to learn these and put to use.

Regex in Perl: 

Match Operator(m//) -  The m// operator is used for pattern matching. In between the forward slashes // the pattern to match is placed. Additional options, if any, are placed at the end after the last slash. If an expression is explicitly bound to the operator using the = or ! binding operators, that expression is searched for the pattern specified. If the binding operator is missing, as you will see in some later examples, = is assumed and $ is taken as the expression to be searched. In scalar context, the binding operator = returns a true value if the expression matches the pattern, an empty string (and hence a false value) if otherwise. ! simply inverts the logic so that if the expression matches the pattern a false value is returned, a true value otherwise.

Metacharacters serve specific purposes in a pattern. If any of these metacharacters are to be embedded in the pattern literally, you should quote them by pre xing it by n, similar to the idea of escaping in double-quoted string. In fact, the pattern in between the forward slashes are treated as a double-quoted string. For example, ~ m/\$/

Meta Characters: \ ,ˆ , . , $ , | , () , []

Matching Only Once -

There is also a simpler version of the match operator - the PATTERN operator. This is basically identical to the m// operator except that it only matches once within the string you are searching between each call to reset.

For example, you can use this to get the first and last elements within a list:

#!/usr/bin/perl @list = qw/food foosball subeo footnote terfoot canic footbrdige/; foreach (@list) {    $first = $1 if ?(foo.*)?;    $last = $1 if /(foo.*)/; } print "First: $first, Last: $last\n"; This will produce following result First: food, Last: footbrdige

Quantifiers:

Quantifiers are used to specify how many times a certain pattern can be matched consecutively.  A quanti er can be speci ed by putting the range expression inside a pair of curly brackets. The format of which is {m[, [n]]}.

Examples:

{m} Match exactly m times

{m,} Match m or more times

{m,n} Match at least m times but not more than n times

Character Classes:

A character class includes a list of characters where matching of any of these characters result in a match of the character class. A character class is constructed by placing the characters inside a pair of square brackets.

\w        Alphanumeric characters and _ ([a-zA-Z0-9_ ])

\W        Neither alphanumeric characters nor _([ˆa-zA-Z0-9_ ])

\s         Whitespace characters ([ \t\n\r\f])

\S         Non whitespace characters ([ˆ \t\n\r\f])

\d         Numeric digits ([0-9])

\D         Non numeric digits ([ˆ0-9])

Backtracking:

Parenthesized patterns have a useful property. When pattern matching is successful, the matching substrings corresponding to the parenthesized parts are saved, which allow you to save them for further operations. For example,

$string = ’Telephone: 1234-5678’;

if ($string =˜ m/ˆTelephone:\s*(\d{4}-\d{4})$/) {

print “The telephone number extracted is ’$1’.\n”;

}

In this example, the telephone number extracted is saved as $1. There can be multiple bracketed patterns in a given pattern. The matched substrings are numbered in ascending order of position of the opening parentheses.

Substitution Operator

The substitution operator, s///, is really just an extension of the match operator that allows you to replace the text matched with some new text. The basic form of the operator is:

s/PATTERN/REPLACEMENT/;

The PATTERN is the regular expression for the text that we are looking for. The REPLACEMENT is a specification for the text or regular expression that we want to use to replace the found text with.

For example, we can replace all occurrences of .dog. with .cat. using

$string =~ s/dog/cat/;

Translation 

Translation is similar, but not identical, to the principles of substitution, but unlike substitution, translation (or transliteration) does not use regular expressions for its search on replacement values. The translation operators are:

tr/SEARCHLIST/REPLACEMENTLIST/cds

y/SEARCHLIST/REPLACEMENTLIST/cds

The translation replaces all occurrences of the characters in SEARCHLIST with the corresponding characters in REPLACEMENTLIST. For example, using the "The cat sat on the mat." string we have been using in this chapter:

#/user/bin/perl

$string = 'The cat sat on the mat';

$string =~ tr/a/o/;

print "$string\n";

This will produce following result:

The cot sot on the mot.

Regular Expression Operators:

m//—PatternMatching - the m// operator performs pattern matching. It supports a number of options. The m option allows matching of individual lines in a multi-line string. The i option matches in a case-insensitive manner. By default, pattern matching is case sensitive. The g option attempts to carry out a global pattern matching on the string.

s///—       Search and Replace - This operator is a powerful search-and-replace engine that you can use to exibly search for certain patterns and replace it with a replacement string. The rst argument is the search pattern, just as the case of m//. The second argument is the replacement string. As you will soon see, backtracking is immensely useful in this regard.

tr///—       Global Character Transliteration - tr/// is a convenient and ef cient operator that changes a set of characters into another. The rst argument is the character list to search for. The second argument is the character replacement list. It builds a character translation map at compile time. At run-time, it changes any characters that can be found in the string into the corresponding character in the replacement list.

SHELL Scripting

A shell script is a script written for the shell, or command line interpreter, of an operating system. The shell is often considered a simple domain-specific programming language.There are hundreds of UNIX commands that you can execute at the shell prompt. 

Shells have their own built-in syntax that helps you to work more effectively with existing commands by allowing you to perform functions like plugging commands into each other and controlling the flow of execution.

Conditional operators

|| . use the double pipe operator in the form, command1 || command2

In the above syntax, the second command executes only if the first command fails.

&& . use the double ampersand operator in the form, command1 && command2

In the above syntax, the second command executes only if the first command executes successfully.

Command grouping operators

{ } , enclose multiple statements in braces ({}) to create a code block. The shell returns one exit status value for the entire group, rather than for each command in the block.

( ) , enclose multiple statements in round brackets to create a code block. This code block functions in the same way as a code block enclosed in braces, but runs in a subshell.

I/O redirection operators

> , use this operator to redirect command output to a file. If the specified file doesn’t exist, the shell creates the file. If the file does exist, the shell overwrites it with the command output unless the noclobber environment variable is set.

>| , use this operator to redirect command output to a file. If the specified file doesn’t exist, the shell creates the file. If the file does exist, the shell overwrites it with the command output even if the noclobber environment variable is set.

>> , use this operator to redirect command output to a file. If the file doesn’t exist, the shell creates the file. If it does exist, the shell appends the new data to the end of it.

< , use this operator to redirect command input from a file.

File descriptor redirection operators

<&n , use this operator to redirect standard input from file descriptor n.

>&n , use this operator to redirect standard input to file descriptor n.

n< filename , use this operator with a filename to redirect descriptor n from the specified file.

n> filename , use this operator with a filename to redirect descriptor n to the specified file. Unlike ordinary redirection, this will not overwrite an existing file.

n>| filename , use this operator with a filename to redirect descriptor n to the specified file, overriding the noclobber environment variable if it is set.

n>> filename , use this operator with a filename to redirect a descriptor to the specified file. This will redirect to a file but, unlike ordinary redirection, this will append to an existing file.

Filename substitution

* , use the * wildcard to match a string of any length.

? , use the ? wildcard to match a single character.

[abc] , [a-c] , [a-c1-3]

use square brackets to match only characters that appear inside the specified set. For increased convenience, you can specify multiple ranges.

!pattern , use the ! operator with a pattern to perform a reverse match. The shell returns only filenames that don’t match the pattern.

Command substitution

$(command) , use this form of command substitution to resolve a command and pass its output to another command as an argument.

$(< filename) , use this form of command substitution to pass the entire contents of a file to a command as an argument.

Tilde substitution

~ , use the ~ operator to instruct the shell to return the value of the $HOME variable.

~username , use the ~ operator with a username to instruct the shell to return the full path of a specific user’s home directory.

~+ , use the ~+ operator to instruct the shell to return the full path of the current working directory.

~- , use the ~- operator to instruct the shell to return the full path of the previous working directory you used.

New Line is New Command

Every new line should be considered a new command, or a component of a larger system. If/then/else statements, for example, will take over multiple lines, but each component of that system is in a new line. Don’t let a command into the next line, as this can truncate the previous command and give you an error on the next line. If your text editor is doing that, you should turn off text-wrapping to be on the safe side. 

Comment(#=comment)

If you start a line with #, the line is ignored. This turns it into a comment line, where you can remind yourself of what the output of the previous command was, or what the next command will do. Again, turn off text wrapping, or break you comment into multiple lines that all begin with a hash. Using lots of comments is a good practice to keep, as it lets you and other people tweak your scripts more easily.

Case Statement

If there is a first parameter, we need to figure out which it is. For this, we use a case statement:

case "$1" in

   init)

    

   ;;

   create)

    

   ;;

   run)

    

   ;;

   *)

      help

   ;;

esac

We pass the first parameter to the case statement; then, it should match one of four things: "init", "create", "run", or our wildcard, default case. Notice that we don't have an explicit "help" case: that's just our default case. This works, because anything other than "init", "create", and "run" aren't commands we recognize, so it should get the help text.

Miscellaneous syntax

; , If you enter several commands on the same line, you need to separate the commands with semicolons. The shell will execute each command successively once you press Enter.

\ , use a backslash to allow you to press Enter and continue typing commands on a new line. The shell will only begin executing your commands when you press Enter on a line that doesn’t end in a backslash. Using a backlash in this way is known as backslash escaping.

& , add a single ampersand at the end of a command to run that command as a background process. This is useful for tasks that are likely to take a long time to complete.

Shell programs can execute a wide range of UNIX commands, but they also have built-in functions to help you use shells more effectively. Most shells support standard operators for conditional execution, input/output (I/O) redirection, file descriptor redirection, and command grouping. They also allow you to perform filename, tilde, and command substitution. Different shells have shell specific features. So, shell should be chosen based on useful shell features. If a shell script has to be created, which should be run on different shells, it is suggested to develop on a basic shell like Bourne.

Simple example :

Open a text editor (but not a word processor), such as gedit or vi, and type the following three lines exactly as shown on a new, blank page:

#!/bin/bash

Clear

echo "Hello world."

Alternatively, the above code could be copied from this page and pasted to a blank page opened by the text editor page using the standard keyboard or mouse copy and paste functions.

After saving this plain text file, with a file name such as test (or anything else desired), the script is complete and almost ready to run. Scripts are typically run by typing a dot, a forward slash and the file name (with no spaces in between) and then pressing the ENTER key.

./ test

However, the script might give Permission denied error. This is because the permissions for the file first have to be set to executable. (By default, the permissions for new files are set to read and write only.) The problem can easily be solved by using the chmod command with its 755 option (which will allow the file creator to read, write and execute the file) while in the same directory as that in which the file is located as follows:

chmod 755 test

Now the script is ready to run again:

./ test