Blog from February, 2015

 

 

 

 


Neo4J is a great Graph Database and OSGI is currently the most serious modularisation framework for Java. Unfortunately Neo4J is not OSGI friendly out of the box and consequently we need to repackage it. 

Have a clean integration of Neo4J on OSGI runtime is a major point for the Ariane project as in the long term Ariane will be composed by plenty of OSGI modules and µ-services. For the OSGI value proposition let's have a look to this and this

There is a lot of OSGI runtime and most of them are working well. We decided to go with EclipseVirgo because we failed to make work Primefaces on http://karaf.apache.org/ at the time we have to choose our OSGI container. Since then Apache Karaf seems have provided a correction on that subject but I never get time to test it again. So I'm not saying Eclipse Virgo is better than Apache Karaf. Eclipse Virgo is just working well for our current needs on the Ariane project. But finally the work done here with Eclipse Virgo should be reusable on Apache Karaf. Some little work will be necessary but not so much. Let me know if you're planning to do that work and I'll be happy to help you.

Target of this blog post is show how to integrate the orginal Neo4J community distribution on the Eclipse Virgo Kernel. Main idea here is to make work Neo4J in Eclipse Virgo with a minimal work and the minimal overhead : that's why we choose Eclipse Virgo Kernel and not Eclipse Virgo Tomcat/Jetty. Finally at the end of this blog post you'll be able to run Neo4J on an OSGI environment as if you run it from the official distribution.

 

 

 


Prepare Virgo Kernel to embed Neo4J

We first need to changes some Virgo Kernel configurations. So if not done already you have to download Virgo Kernel (last version is 3.6.3) and unzip it in some working directory. Let's call now $VIRGO_HOME the installed Virgo directory. You also have to get and install some JRE 1.7 to run Virgo.
 

Setup Virgo Java 1.7 environment

First of all you need to enable JavaSE 1.7 for Virgo: edit the $VIRGO_HOME/configuration/java6-server.profile and add JavaSE-1,7 to the org.osgi.framework.executionenvironment list :

diff configuration/java6-server.profile

Setup Virgo SSH

To deploy OSGI bundles or Virgo plans|pars, we will use the Virgo SSH console. You need to enable the Virgo SSH consoles by editing following files :

  • $VIRGO_HOME/configuration/osgi.console.properties for the virgo kernel region
  • $VIRGO_HOME/repository/ext/osgi.console.properties for the virgo user region

Virgo Regions

Icon

To know more about Virgo regions let's have a look to the Virgo documentation.

diff configuration/osgi.console.properties
diff repository/ext/osgi.console.properties

Virgo users configuration

Icon

You can connect on Virgo through SSH by using the default user/password : admin/springsource.
You can configure the Virgo users through following file : $VIRGO_HOME/configuration/org.eclipse.virgo.kernel.users.properties
To know more about authentication in Virgo let's have a look to Virgo documentation

 

Setup Virgo neo4j-community repository

Finally we setup a dedicated neo4j-community repository into Virgo by :

  • creating the $VIRGO_HOME/repository/neo4j-community directory
  • editing the $VIRGO_HOME/configuration/org.eclipse.virgo.repository.properties file


diff configuration/org.eclipse.virgo.repository.properties

External vs Watched Virgo repository

Icon

You may want to configure a watched repository instead of an external one (Virgo will then load changes on the repository at runtime). To know more about Virgo provisioning repository look at the Virgo documentation.

 

 


Install Neo4j into Eclipse Virgo Kernel

Neo4J community classpath

When you unzip the Neo4j community distribution you have two directories where you can find Neo4J JARs which are composing the Neo4J runtime classpath :

  • $NEO4J_HOME/system/lib
  • $NEO4J_HOME/lib

There is a lot of non ready OSGI JARs in the Neo4J 2.1.2 community distribution and most of them are coming from Neo4J JARs. We have different ways to manage these non OSGI friendly JARs we will choose depending on the situations. Our strategy is resumed at the end of this blog post.

 

Install OSGI ready Neo4j dependencies into Virgo

Neo4J community dependencies will be installed in $VIRGO_HOME/repository/neo4j-community. All these bundles are public bundle you can find on maven repos (eg : http://nexus.echinopsii.net).

 

List of primary Neo4J community dependencies to install

 

To makes some of these bundles happy in OSGI environment you also need to add others bundles to fit the dependencies.

  • org.apache.commons.configuration 1.6.0 needs javax.mail_1.4.0.v201005080615.jar.
    I installed this one in $VIRGO_HOME/repository/ext as this is the target repository for this bundle in the Virgo Tomcat distribution.
     
  • org.apache.commons.configuration 1.6.0 needs commons-codec_1.5.jar.
    I installed this one in $VIRGO_HOME/repository/ext as this is the target repository for this bundle in the Virgo Tomcat distribution.
     
  • org.apache.commons.configuration 1.6.0 needs commons-jxpath-1.3.jar.
    I installed this one in $VIRGO_HOME/repository/neo4j-community.
     
  • org.apache.commons.jxpath 1.3.0 needs javax.servlet.jsp_2.2.0.v201112011158.jar
    I installed this one in $VIRGO_HOME/repository/ext as this is the target repository for this bundle in the Virgo Tomcat distribution.
     
  • javax.servlet.jsp 2.2.0.v201112011158 needs javax.el_2.2.0.v201108011116.jar
    I installed this one in $VIRGO_HOME/repository/ext as this is the target repository for this bundle in the Virgo Tomcat distribution.
     
  • org.apache.commons.jxpath 1.3.0 needs com.springsource.org.jdom-1.1.0.jar
    I installed this one in $VIRGO_HOME/repository/neo4j-community.

 

Finally after these first dependencies install repositories look like this :

 

$VIRGO_HOME/repository/ext

 

 

$VIRGO_HOME/repository/neo4j-community

 

Transform non OSGI friendly JARs dependencies into OSGI bundles 

There is some Neo4J dependencies which are not OSGI friendly JARs and we need to transform them as OSGI bundles. That transformation implies defining an OSGI ready manifest into these JARs without touching any line of codes. 
This task could be very boring but Eclipse Virgo offers us a cool tool which produces relevant OSGI manifest from a JAR file : Eclipse Virgo Bundlor

To make our life more easier, we provide some scripting around this tool and a collection of bundlor manifests in our osgi factory. This is the way we created the following bundles from original JARs to install into Virgo Kernel neo4j-community repository.
Actually most of the bundles generated through the echinopsii osgi factory can be downloaded from the echinopsii maven repository

 

Original JAR nameBundle nameBundlor manifestComments
logback-access-1.0.7net.echinopsii.3rdparty.logback-access-1.0.7

go to manifest

This bundle has been regenerated from the original one to declare some dependencies as optional (like tomcat dependencies)
opencsv-2.0.0net.echinopsii.3rdparty.net.sf.opencsv-2.0.0

go to manifest

Neo4J original dependency as a bundle
janino-2.6.1net.echinopsii.3rdparty.org.codehaus.janino.janino-2.6.1.jar

go to manifest

Neo4J original dependency as a bundle
commons-compiler-2.6.1net.echinopsii.3rdparty.org.codehaus.janino.commons-compiler-2.6.1.jar

go to manifest

Needed dependency for janino-2.6.1
ant-1.8.2net.echinopsii.3rdparty.org.apache.ant-1.8.2.jar

go to manifest

Needed dependency for janino-2.6.1
ant-launcher-1.8.2net.echinopsii.3rdparty.org.apache.ant-launcher-1.8.2.jar

go to manifest

Needed dependency for janino-2.6.1
rhino-1.7.4net.echinopsii.3rdparty.org.mozilla.rhino-1.7.4.jar

go to manifest

Neo4J original dependency as a bundle
rrd4j-2.0.7net.echinopsii.3rdparty.org.rrd4j.rrd4j-2.0.7.jar

go to manifest

Neo4J original dependency as a bundle

 

Once installed into Virgo, the neo4j-community repository looks like this : 

$VIRGO_HOME/repository/neo4j-community

 

 

The Neo4J community super-bundle

We just saw how we can easily transform non OSGI friendly jar into OSGI bundle with Eclipse Virgo Bundlor. But some times this is not enough to make work non-osgi friendly JARs into OSGI environment :

  • because you may have import constraint which becomes spaghetti-like dependency chain which could be horrible to demystify. This is the case of the jersey dependency chain.
  • because of the heavily use of java ServicesLoader feature introduced in Java 6 - which is not really OSGI friendly. This is the case of the Neo4J libraries.

In such cases the quickest (as you decide to NOT reverse-engineer the dependencies) and safest (as you decide to NOT add any code lines) solution is to create a super-bundle which will include all the problematic JARs and therefore provides a same classpath area for all these dependencies you want to make work together.

To make this super-bundle we use the great work from Apache Servicemix team who deliver a maven engineering tool to generate OSGI bundle from severals dependencies. The Neo4J community super-bundle definition can be found here.

The final neo4j.community repository looks like this : 

$VIRGO_HOME/repository/neo4j-community

 

Install and start Neo4J super-bundle in Eclipse Virgo Kernel

Once Virgo Kernel is started ($VIRGO_HOME/bin/startup.sh) we can connect on the Kernel Region console through SSH : 

Then we can start the Neo4J super-bundle like this : 

Examine Neo4J super-bundle  Expand source

 

 


Start the Neo4J server in Eclipse Virgo Kernel

Neo4J and Jetty integration into Virgo Kernel environment

To add the Jetty configuration into Virgo Kernel environment, create the $VIRGO_HOME/org/eclipse/jetty/webapp directory. Then copy the Jetty webdefault.xml in it.

To add the neo4j configuration into Virgo Kernel environment, create the following directories :

  • $VIRGO_HOME/neo4j/conf
  • $VIRGO_HOME/neo4j/data
  • $VIRGO_HOME/neo4j/graph

Then copy the neo4j configuration files into $VIRGO_HOME/neo4j/conf : 

You need to adapt these files to fit your environment path.

Enable Neo4J RMI in Virgo Kernel environment

Neo4J use RMI and so we need to enable it in Virgo Kernel environment. 
Edit the $VIRGO_HOME/bin/dmk.sh and add the following lines : 

Add RMI to Virgo Kernel

 

Then add and edit $VIRGO_HOME/configuration/virgo.policy :

Virgo security policy

OSGI security

Icon

OSGI provides security features extending the Java ones. To know more loot at this doc and this presentation.
The above security policy is totally unsecure but very  convenient. Feel free to adapt this policy to your paranoïa...

 

Get and install the neo4j-osgi demonstration application

To start the Neo4J server in Virgo, we will use a dedicated application which bootstrap Neo4J following OSGI way. You can get this application here.

Neo4JActivator

This little class is the OSGI bootstrapping handler. When starting/stopping the bundle the start/stop method of this class are called.
At startup the only thing done is the Neo4JManagedService registration into the service registry with the pid "net.echinopsii.demo.Neo4JManagedService".

Neo4JActivator.java

Neo4JManagedService

The Neo4JManagedService handle the configuration and start the Neo4J server according to the neo4j configuration path.

Neo4JManagedService.java

 

pom.xml

When building the project, maven will setup the OSGI manifest of the bundle according to this pom. Look at the necessary import to make work the Neo4J bootstrap.

pom.xml

 

install the demo app

Once builded you can install the neo4j-osgi-1.0.0.jar in the $VIRGO_HOME/repository/usr. 
You also need to define the Neo4JManagedService configuration by adding and editing $VIRGO_HOME/repository/usr/net.echinopsii.demo.Neo4JManagedService.properties file : 

Neo4JManagedService configuration

Finally you can configure the log level of this little app in $VIRGO_HOME/configuration/serviceability.xml

Set up log level

 

Run neo4j-osgi demonstration application and test the OSGI embedded Neo4J server

Execute the following commands to make run the OSGI embedded Neo4J server.

Starting Neo4J server

Once these commands are executed you can see the following lines into the Virgo logs ($VIRGO_HOME/serviceability/logs/log.log)

And consequently you can have a look to the Neo4J webapplication (http://your-server-fqdn:7474) and begin to play with the Neo4J Movie Graph tutorial (smile)

 

 

 


Annex : Neo4J classpath bundleisation strategy

JARs in $NEO4J_HOME/system/libOSGI bundle ?Bundleisation methodology
asm-3.1.jarnouse already asm bundle by springsource
to be installed in Virgo 
bcprov-jdk16-140.jarnouse already bouncycastle bundle by springsource
to be installed in Virgo 
commons-beanutils-1.8.0.jaryesto be installed in Virgo
commons-beanutils-core-1.8.0.jarnosame as commons beanutils
commons-collections-3.2.1.jaryesto be installed in Virgo
commons-compiler-2.6.1.jarnobundleisation through bundlor
to be installed in Virgo 
commons-configuration-1.6.jaryesto be installed in Virgo
commons-digester-1.8.1.jaryesto be installed in Virgo
commons-io-1.4.jaryesprovided by Virgo :
com.springsource.org.apache.commons.io_1.4.0.jar
commons-lang-2.4.jaryesto be installed in Virgo
commons-logging-1.1.1.jarnoprovided by Virgo :
org.slf4j.jcl_1.7.2.v20121108-1250.jar
jackson-core-asl-1.9.7.jaryesto be installed in Virgo
jackson-jaxrs-1.9.7.jaryesto be installed in Virgo
jackson-mapper-asl-1.9.7.jaryesto be installed in Virgo
janino-2.6.1.jarnobundleisation through bundlor
to be installed in Virgo 
javax.servlet-3.0.0.v201112011016.jaryesprovided by Virgo:
javax.servlet_3.0.0.v201112011016.jar
jcl-over-slf4j-1.6.1.jaryesprovided by Virgo :
org.slf4j.jcl_1.7.2.v20121108-1250.jar
jersey-core-1.9.jaryesNeo4j super bundle
jersey-multipart-1.9.jaryesNeo4j super bundle
jersey-server-1.9.jaryesNeo4j super bundle
jetty-http-9.0.5.v20130815.jaryesto be installed in Virgo
jetty-io-9.0.5.v20130815.jaryesto be installed in Virgo
jetty-security-9.0.5.v20130815.jaryesto be installed in Virgo
jetty-server-9.0.5.v20130815.jaryesto be installed in Virgo
jetty-servlet-9.0.5.v20130815.jaryesto be installed in Virgo
jetty-util-9.0.5.v20130815.jaryesto be installed in Virgo
jetty-webapp-9.0.5.v20130815.jaryesto be installed in Virgo
jetty-xml-9.0.5.v20130815.jaryesto be installed in Virgo
jsr311-api-1.1.2.r612.jaryes

to be ignored
 
packages already present in jersey-core...

logback-access-1.1.2.jaryesto be installed in Virgo
(Note: we'll use logback-access-1.0.7 to fit Virgo logback versioning)
logback-classic-1.1.2.jaryesprovided by Virgo :
ch.qos.logback.classic_1.0.7.v20121108-1250 
logback-core-1.1.2.jaryesprovided by Virgo : 
ch.qos.logback.core_1.0.7.v20121108-1250
mimepull-1.6.jaryesto be installed in Virgo
neo4j-browser-2.1.2.jarnoNeo4j super bundle
neo4j-server-2.1.2.jarnoNeo4j super bundle
neo4j-server-2.1.2-static-web.jarnoNeo4j super bundle
opencsv-2.0.jarnoNeo4j super bundle
rhino-1.7R4.jarnobundleisation through bundlor
to be installed in Virgo 
rrd4j-2.0.7.jarnobundleisation through bundlor
to be installed in Virgo 
slf4j-api-1.6.2.jaryesprovided by Virgo :
org.slf4j.api_1.7.2.v20121108-1250 
JARs in $NEO4J_HOME/libOSGI bundle ?Bundleisation methodology
concurrentlinkedhashmap-lru-1.3.1.jaryesto be installed in Virgo
geronimo-jta_1.1_spec-1.1.1.jar yes

provided by Virgo :
org.eclipse.osgi_3.8.1.v20120830-144521

lucene-core-3.6.2.jarnouse servicemix bundle
to be installed in Virgo 
neo4j-cypher-2.1.2.jarnoNeo4j super bundle
neo4j-cypher-commons-2.1.2.jarnoNeo4j super bundle
neo4j-cypher-compiler-1.9-2.0.3.jarnoNeo4j super bundle
neo4j-cypher-compiler-2.0-2.0.3.jarnoNeo4j super bundle
neo4j-cypher-compiler-2.1-2.1.2.jarno Neo4j super bundle
neo4j-graph-algo-2.1.2.jarnoNeo4j super bundle
neo4j-graph-matching-2.1.2.jarnoNeo4j super bundle
neo4j-jmx-2.1.2.jarnoNeo4j super bundle
neo4j-kernel-2.1.2.jarnoNeo4j super bundle
neo4j-lucene-index-2.1.2.jarnoNeo4j super bundle
neo4j-primitive-collections-2.1.2.jar noNeo4j super bundle
neo4j-shell-2.1.2.jarnoNeo4j super bundle
neo4j-udc-2.1.2.jarnoNeo4j super bundle
opencsv-2.0.jarnobundleisation through bundlor
to be installed in Virgo 
org.apache.servicemix.bundles.jline-0.9.94_1.jaryesto be installed in Virgo
parboiled-core-1.1.6.jaryesto be installed in Virgo
parboiled-scala_2.10-1.1.6.jaryesto be installed in Virgo
scala-library-2.10.4.jaryesto be installed in Virgo
server-api-2.1.2.jarnoNeo4j super bundle