Showing posts with label maven. Show all posts
Showing posts with label maven. Show all posts

Tuesday, May 21, 2013

DevOps: Making Fast Deployments of Java Servers using Maven and Nexus

A Warning: this post is theoretical. I have never tried something like this yet. Maybe I will try it in the future. But currently it's just a nice idea.
In addition, if you know about somebody who works in a similar way, I would really like to know. So please comment!

If you provide a SAAS service you probably have multiple Java servers running in some sort of a cluster. If your SAAS solution is complicated, and if your solution is multi-tier, you should have multiple servers types. And now comes a question: How to make quick deployments to the production?

The common solution suggests that you build a package and release it. It might be a war, or a zip, or a rpm if you are running on Linux.
Once released, you upload the package to the server, unzip/copy it to the relevant folder and restart the server.

The problem with this solution might be if your packages are large. (And if you are using OSGi, your packages are usually very large!) So the upload itself takes time. It also uses traffic which might become expensive if you perform a lot of deployments. And the really funny thing is that most of the upload is redundant: most of the jars in your package are third parties that do not change between the deployments at all!

The common solution suggests pre-uploading the third party jars to the server and exclude them from the package. I've seen such a solutions and in my opinion they are the exact opposite of a good solution: in this way you split the package, the third parties become manages in two (sometimes more) places and each deployment involves at least additional (probably manual!) step of checking if the third parties were changed and if additional deployment if third parties is required.

But if you use Maven. And if you upload your released packages to Nexus (or actually any other Maven repository). This Nexus repository contains all the third parties, all the released packages and the most important: The pom file that was used to build your project!
If you download this pom file, you will be able to build the package on the production server! Pay attention that you don't need to do the full build that includes the compilation, testing and so on. You just need your package, so considering that you deploy a war, you only need to run the "mvn war:war" (Once again: I never tried it myself and the actual execution might be more complex, but I think that the idea is clear).
Sometimes, if you a running a java application with a main class (pure old java and not some kind of JEE inside the application server or a servlet container), you don't even need a package, you just need a correct classpath and Maven will be happy to assist you: mvn dependency:build-classpath.

So I guess that the idea is clear now. Each time Maven will download only the relevant jars and save them to the local repository. The dependencies are managed in the same pom file that is used to release the application, so when making a package, or creating a classpath on a production machine, the exact same dependencies will be used.
And the deployment process will become much faster!

I know that this idea is somewhat different from the usual process. Instead of doing some like "build, deploy, run", we do something that might look even more complicated: "build, deploy descriptor only, package, run". But this should be much faster. So I definitely think that this idea is worth trying.


P.S. The idea described in this post relates only to the package itself: building, packaging and running. The deployment may contains additional steps like changing the local configuration files and so on. These steps are not covered here as they are usually not covered in a build process, but part of release notes. The possible solution can be deploying the relevant scripts to Nexus repository and somehow describe them in a pom file. When downloading the pom, the relevant scripts will be also downloaded and executed.

P.P.S. The idea also doesn't cover the tool that makes the whole process. Although it describes that the tool is using Maven, it says nothing about the actual implementation. It might be a java process. Or a shell script. Or even Ant.

P.P.P.S. Notice that downloading files from Nexus using Maven makes important checks for you, for example it makes an integrity check, which is very important in case of a bad network between the Nexus with releases and a production site.
In addition, you can make some optimizations on Nexus. For example, if you have several production sites all over the world, each site may have a Nexus pointing to the main release repository and caching it. This will make the deployments even faster.


Recommended Reading



Thursday, December 8, 2011

Automated Integration Tests Using with Jetty, Maven and Other Neat Freameworks - 2

In the previous post, I have started a Jetty server in the beginning of a unit test with a configured external data-source.

Let's talk about it a little bit more. The assumption here was that the application on a regular basis uses an external data-source that is accessed via JNDI. In general it's a good practice to keep a data-source external to the application:
1. It's always possible to change the data-source without touching the application - let's say a bug was found in a data-source you are using. Or it should be configured differently. If a data-source embedded into an application and such a change is required, you will probably need to release patch. If the datasource is external, it would be enough just to change/reconfigure it.
2. In some deployments several applications can use the same datasource. Consider a Tomcat running with several wars: quite a common case, right? If a datasoure is embedded, each war has its own data-source. Usually this would be a waste of resources and duplication of a configuration.
3. And finally it would be possible to change a data-source for unit tests, which is exactly our use case.

For unit tests it's really convenient to use in-memory database. There are few databases implemented in java that provides this capability. My favorite are H2 and HSQL. In this example I have used HSQL, so here comes the jetty-ds-test.xml:

<Configure id="Server" class="org.eclipse.jetty.server.Server">
    <New id="DSTest" class="org.eclipse.jetty.plus.jndi.Resource">
        <Arg></Arg>
        <Arg>jdbc/my_ds</Arg>
        <Arg>
            <New class="org.hsqldb.jdbc.JDBCDataSource">
                <Set name="Url">jdbc:hsqldb:mem:test;sql.syntax_ora=true</Set>
                <Set name="User">sa</Set>
                <Set name="Password">sa</Set>
            </New>
        </Arg>
    </New>
</Configure>

This is configuration of HSQL in memory. Notice sql.syntax_ora=true, which makes HSQL to use Oracle syntax. This is useful if you are using Oracle in production, but you want to use HSQL for unit testing or development.

So now you have at the beginning of a unit-test you have a Jetty server running with your application connected to in-memory HSQL database via JNDI. But something is still missing. This something is a database schema: to remind you, we have just started a new database in memory and it's empty. You probably already have a script that generates schema, tables, indexes and may be some data. Now you need to run it.

Actually there are several options how do it. One of the easiest is to use Springs's SimpleJdbcTestUtils. But first we'll need to add a dependency on spring-test in our pom.xml:

<dependency>
         <groupId>org.springframework</groupId>
         <artifactId>spring-test</artifactId>
         <scope>test</scope>
         <version>${spring-version}</version>
</dependency>

And here comes the code that runs the sql script:
import javax.naming.InitialContext;
import javax.naming.NamingException;
import javax.sql.DataSource;
import org.springframework.core.io.ClassPathResource;
import org.springframework.core.io.Resource;
import org.springframework.jdbc.core.simple.SimpleJdbcTemplate;
import org.springframework.test.jdbc.SimpleJdbcTestUtils;
...
    @BeforeClass(dependsOnMethods = { "startJetty" })
    public void initiateDatabase() throws Exception {       
        InitialContext initialContext = new InitialContext();
        DataSource ds = (DataSource) initialContext.lookup("jdbc/my_ds");
        SimpleJdbcTemplate simpleJdbcTemplate = new SimpleJdbcTemplate(ds);
        Resource resource = new ClassPathResource(sqlScriptFileName);
        SimpleJdbcTestUtils.executeSqlScript(simpleJdbcTemplate, resource, false);
    }
In this snippet I load the sqlScriptFileName from the classpath. Usually it's convenient to place the script in src/test/resources, but if you don't like it, you can always load it from a different place by using other Resource implementations (e.g. URLResource is quite convenient).

As I have already said in this snippet I used Spring. If you are familiar with Spring, it is probably natural to you. If you don't - don't be afraid. Only the unit tests become dependent on Spring, but the actual application did not.

And now you are ready to start the testing.


Recommended Reading

1. Next Generation Java Testing: TestNG and Advanced Concepts
2. Apache Maven 3 Cookbook
3. Spring Recipes: A Problem-Solution Approach

Automated Integration Tests Using with Jetty, Maven and Other Neat Freameworks

Let's say that you have a web application (aka war) that exposes RESTful API and connects to a database. Now you want to do some automation tests (and I'm not going to describe here why actually you must have automation tests that run regularly on your API).

One of the best solutions to do it, in my opinion, is running your application on a Jetty server that is embedded in your unit test with some in-memory database.

But let's do it step-by-step:

Step1: Start Jetty from a Unit Test

(In this guide I have used TestNG, but I see no reason why the same functionality cannot be achieved in JUnit)

So first we need to start Jetty in our test. But before we actually do that, we need to make sure that our Maven project contains the relevant dependencies. So first we need Jetty:
<dependency>
            <groupId>org.eclipse.jetty</groupId>
            <artifactId>jetty-server</artifactId>
            <version>${jetty-version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.eclipse.jetty</groupId>
            <artifactId>jetty-webapp</artifactId>
            <version>${jetty-version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.eclipse.jetty</groupId>
            <artifactId>jetty-jndi</artifactId>
            <version>${jetty-version}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.eclipse.jetty</groupId>
            <artifactId>jetty-plus</artifactId>
            <version>${jetty-version}</version>
            <scope>test</scope>
        </dependency>

The Jetty dependencies in this guide already contains the JNDI support. It will be needed later. But if JNDI support is not required, they can be omitted.

And then let's start it before the tests start:

import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.webapp.WebAppContext;
import org.eclipse.jetty.xml.XmlConfiguration;
import org.testng.annotations.BeforeClass;

public class MyTest {

    private static final String RESOURCES_URL = "/rs";
    private static final String CONTEXT       = "/app_context";
    private static final String DS_CONFIG     = "/jetty-ds-test.xml";
    private String              baseResourceUrl;

    @BeforeClass
    public void startJetty() throws Exception {
        Server server = new Server(0);   // see notice 1
        server.setHandler(new WebAppContext("src/main/webapp", CONTEXT)); // see notice 2

        // see notice 3
        InputStream jettyConfFile = InboxTest.class.getResourceAsStream(DS_CONFIG);
        if (jettyConfFile == null) {
            throw new FileNotFoundException(DS_CONFIG);
        }
        XmlConfiguration config = new XmlConfiguration(jettyConfFile);
        config.configure(server);

        server.start();
        
        // see notice 1
        int actualPort = server.getConnectors()[0].getLocalPort();
        baseResourceUrl = "http://localhost:" + actualPort + CONTEXT + RESOURCES_URL;
    }
Please notice that:
1. Jetty is started on a random port. The actual url with the actual port is saved to baseResourceUrl to be used later by tests.
2. Web application context points to maven's src/main/webapp.
3. Jetty is started with a data source configuration. (See Runnig Jetty from Maven using JNDI Data Source)

Part 2


Recommended Reading

1. Next Generation Java Testing: TestNG and Advanced Concepts
2. Apache Maven 3 Cookbook
3. Spring Recipes: A Problem-Solution Approach

Monday, December 5, 2011

Runnig Jetty from Maven with JNDI Data Source

During the development of a java web component (aka war) it can be very useful to run the application as quick as possible. Jetty provides a Maven plugin that allows running it directly from maven build or explicitly using "mvn jetty:run" from the command line.

But what happens if the war uses external database? Especially when the datasource is defined externally to the war and accessed vie the JNDI?

Actually the solution if quite simple:

Step 1 - Define the Data-source

Create the file defining the data-source:
(The example below is for Oracle. See this page for examples of the other databases.
<Configure id="Server" class="org.eclipse.jetty.server.Server">
    <New id="DSTest" class="org.eclipse.jetty.plus.jndi.Resource">
        <Arg></Arg>
        <Arg>jdbc/my_ds</Arg>
        <Arg>
            <New class="oracle.jdbc.pool.OracleDataSource">
                <Set name="DriverType">thin</Set>
                <Set name="URL">jdbc:oracle:thin:@(DESCRIPTION=(ENABLE=BROKEN)(ADDRESS=(PROTOCOL=TCP)(HOST=127.0.0.1)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=something_to_replace)))
                </Set>
                <Set name="User">my_user</Set>
                <Set name="Password">my_password</Set>
                <Set name="connectionCachingEnabled">true</Set>
                <Set name="connectionCacheProperties">
                    <New class="java.util.Properties">
                        <Call name="setProperty">
                            <Arg>MinLimit</Arg>
                            <Arg>10</Arg>
                        </Call>
                        <Call name="setProperty">
                            <Arg>InactivityTimeout</Arg>
                            <Arg>600</Arg>
                        </Call>
                    </New>
                </Set>
            </New>
        </Arg>
    </New>
</Configure>

Pay attention to "jdbc/my_ds". It's the datasource JNDI name. Make sure to put there the actual JNDI name used by your application.

Place this file under your maven project. In my opinion the best is to place this file under src/dev/resources, but basically it can be anywhere.

To continue the example, I'll name the file: jetty-ds-dev.xml

Step 2 - Define the Jetty Maven Plugin

<build>
...
        <plugins>
...
            <plugin>
                <groupId>org.mortbay.jetty</groupId>
                <artifactId>jetty-maven-plugin</artifactId>
                <version>${jetty-version}</version>
                <configuration>
                    <jettyConfig>src/dev/resources/jetty-ds-dev.xml</jettyConfig>
                </configuration>
                <dependencies>
                    <dependency>
                        <groupId>com.oracle</groupId>
                        <artifactId>ojdbc14</artifactId>
                        <version>${oracle-ojdbc-version}</version>
                    </dependency>
                </dependencies>
            </plugin>
        </plugins>
    </build>

Pay attention to the configuration of jetty-ds-dev.xml.
Also pay attention that Jetty must be able to find the relevant JDBC driver in its classpath!

And basically that's all. Run "mvn jetty:run" and your application should work with the provided database.


Recommended Reading

1. Apache Maven 3 Cookbook
2. Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation

Monday, March 28, 2011

Maven - Hotswap Plugin

For many years I was missing the following feature of Maven: In a multi-module project, when I build a single module in a jar, I want it to be installed into the running environment (app server) seamlessly. During these years I created various scripts for this tasks, why none of them lasted too long: it's very hard to create a generic script, while nongeneric scripts need to be maintained and actually each module must have had its own script. Duh.

Thinking about it, I decided that it should be a Maven plugin, which is very simple - during the install phase, search some predefined location and hotswap the found jars. That's it. So I thought that somebody probably already thought about it and asked a question at Stackoverflow. I received few interesting answers, but not a simple plugin I needed.

So I decided to implement it myself. It took me few hours, since it was my first Maven plugin, so I had some "doing it for the first time" troubles. But finally I did it.

You are welcome to use it, and as always any feedback is appreciated.

Sunday, March 7, 2010

Maven: Copy Dependencies to a folder

So if you need to copy all project dependencies to a folder using Maven, just type
mvn dependency:copy-dependencies -DoutputDirectory=<folder name>

Can it be more simple?
So why I'm googleing for it over and over again?


Recommended Reading

1. Maven: The Definitive Guide
2. Pro Git
3. Smart and Gets Things Done: Joel Spolsky's Concise Guide to Finding the Best Technical TalentProgramming Language & Tool Books)

Wednesday, June 18, 2008

Copy dependencies to a folder using Ant that runs in Maven

The following plugin copies dependencies of the specific pom to a folder.
Ant is used for this task.


<plugin>
<artifactid>maven-antrun-plugin</artifactid>
<executions>
<execution>
<id>copy-tree</id>
<phase>initialize</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<tasks>
<typedef resource="org/apache/maven/artifact/ant/antlib.xml">
<classpath refid="maven.dependency.classpath">
</typedef>
<pom id="maven.project" file="pom.xml">
<dependencies filesetid="dependency.fileset" usescope="runtime">
<pom refid="maven.project">
</dependencies>
<mkdir dir="target/deps">
<copy todir="target/deps">
<fileset refid="dependency.fileset">
</copy>
</tasks>
</configuration>
</execution>
</executions>
<dependencies>
<dependency>
<groupid>org.apache.maven</groupid>
<artifactid>maven-artifact-ant</artifactid>
<version>2.0.4</version>
</dependency>
</dependencies>
</plugin>


via http://www.nabble.com/Using-ant-tasks-inside-antrun-to6994761.html#a6994761

P.S. It's true that it's possible to use the assembly plugin for this specific task. But sometimes things become complex and ANT is essential.

Tuesday, June 17, 2008

Maven vs. Ant

- What do you do when you need to do something complex in Maven?
- You write an ANT script.

©

Thursday, May 22, 2008

Maven commands and plugins

As I already did with JVM Options and Utilities, I'm sharing the spreadsheet with useful commands and plugins of maven.

Wednesday, May 21, 2008

Generation of Eclipse projects using maven

I believe that everyone who uses maven and eclipse is familiar with eclipse:eclipse goal. For these who are not familiar: get familiar with it as quick as you can! It does a very simple but powerful thing: generates eclipse projects for you with all the dependencies.

And here are some tips:

1. To generate new eclipse project from pom.xml, type "mvn eclipse:eclipse".

2. To generate new eclipse project with source attachments type: "mvn eclipse:eclipse -DdownloadSources=true"

3. To regenerate eclipse project close the project in eclipse and type "mvn eclipse:clean eclipse:eclipse". Add "-DdownloadSources=true", if you like to.

4. Now suppose you have hierarchy of projects. You may like to generate all projects with the proper dependencies between them or you may like to generate a project or two, which are dependent on the repository.
In the first case run "mvn eclipse:eclipse" using root pom.xml. In the second case run "mvn eclipse:eclipse" from the project's directory.

5. To use the projects in eclipse, you need to configure variable called M2_REPO which must point to your local repository. Maven can do it for you by running "mvn -Declipse.workspace= eclipse:add-maven-repo". Although I don't find this feature very useful, since you need to do it only once per workspace and it can be shortly done manually.