Child pages
  • Maven Guide
Skip to end of metadata
Go to start of metadata

Originally this documentation was published in Using Maven with OSGi, part 1 article.

pax-url-aether is a library that makes it much easier to perform Maven tasks inside OSGi runtime. One of the most important tasks is artifact resolution, which is roughly a process of changing maven coordinates into physical resource. This resource may be then installed in OSGi runtime (e.g., as OSGi bundle).

I personally think that learning internals of any technology is the best way to use and maintain it in the longer period. Ultimately getting the official source code and reading it in your favourite IDE is much better than relying on official (or unofficial) documentation.

Of course sometimes (usually) there's no time to dig through the internals, so I hope this article will provide an alternative.

Maven

Although there are good alternatives (like Gradle), Apache Maven is still de-facto standard tool for build and dependency management. When we decompose Maven tool into parts, we can reuse the dependency management part in our code. Dependency management is one of the most important aspect of software development and inside OSGi runtime, Maven dependencies are only one of the layers of dependency management. OSGi bundles and Karaf features may be treated as other kinds of dependencies.

We would like to fetch any artifact stored in one of external Maven repositories and use it (as bundle, feature or configuration file) inside the runtime. We'd like to do it the Maven-way, i.e., in declarative way. The best example is installation of external Maven artifact inside OSGi runtime. Like Karaf:

karaf@root()> bundle:install mvn:commons-io/commons-io/2.5
Bundle ID: 52

These commands work out of the box, but usually there's a need to change the default configuration, e.g., configure additional remote repositories, change credentials, configure HTTP proxies, etc. Configuration options are described in separate page, here we'll start with the basics - the Aether library.

Aether

Eclipse Aether is a set of libraries used internally by Maven for dependency resolution. There are various tasks that can be performed using Aether, like finding a closure of artifacts for a graph of dependencies, but even with this low-level library we'll focus on one particular task - getting artifacts from remote repositories.

Official Aether Wiki page is sufficient to get started and see how to use it in code. I'll provide more detailed information in order to describe important concepts.

Aether uses an interface-based API where actual implementations of the interfaces are configured using CDI (or rather Dependency Injection for Java). There are two most important interfaces used:

  • org.eclipse.aether.RepositorySystem - an entry to repository system that provides various resolution methods
  • org.eclipse.aether.RepositorySystemSession - provide additional information specific to operations performed on RepositorySystem

and a set of classes:

  • org.eclipse.aether.*.*Request - various request classes passed as commands to RepositorySystem. We'll focus mainly on org.eclipse.aether.resolution.ArtifactRequest.

RepositorySystem is configured in dependency-injection style - we can select concrete implementations of several SPI interfaces that alter some aspects of Aether, while RepositorySystemSession is configured using properties and directly set objects. Session alters a way in which repository deals with requests.

So let's check how these work together. First let's configure the repository system:

DefaultServiceLocator locator = MavenRepositorySystemUtils.newServiceLocator();
locator.setService(RepositoryConnectorFactory.class, BasicRepositoryConnectorFactory.class);
locator.setService(TransporterFactory.class, FileTransporterFactory.class);
locator.setService(TransporterFactory.class, HttpTransporterFactory.class);
locator.setService(org.eclipse.aether.spi.log.LoggerFactory.class, Slf4jLoggerFactory.class);
RepositorySystem system = locator.getService(RepositorySystem.class);

Nothing extraordinary: we'll have access to http:// and file:// based repositories and SLF4J API will be used for logging.

Now let's configure session. The configuration property is arbitrary and more properties will be described later.

RepositorySystemSession session = MavenRepositorySystemUtils.newSession();
((DefaultRepositorySystemSession)session).setConfigProperty("aether.connector.basic.threads", "2");
LocalRepositoryManager localRepositoryManager = system.newLocalRepositoryManager(session, new LocalRepository("/home/me/.m2/repository"));
((DefaultRepositorySystemSession)session).setLocalRepositoryManager(localRepositoryManager);

And finally let's perform some operation - artifact resolution:

ArtifactRequest req = new ArtifactRequest();
req.setArtifact(new DefaultArtifact("commons-io", "commons-io", "jar", "2.5"));
req.addRepository(new RemoteRepository.Builder("central", "default", "http://repo1.maven.org/maven2").build());
ArtifactResult res = system.resolveArtifact(session, req);

The above tells Aether to resolve artifact commons-io:commons-io:jar:2.5 using local repository inside /home/user/.m2/repository and if it's not found there, to search for the artifact inside http://repo1.maven.org/maven2 remote repository. We could (and it's usual practice) configure more remote repositories (using org.eclipse.aether.resolution.ArtifactRequest#addRepository()) to be searched if artifact isn't available locally.

The code above isn't needed to use Maven inside Karaf or JBoss Fuse, but it brings two super important concepts:

  • local repository - accessed by Aether with the help of org.eclipse.aether.repository.LocalRepositoryManager interface and org.eclipse.aether.repository.LocalRepository class. Effectively local repository is a wrapper for locally accessible filesystem directory that follows specific structure (organization of Maven artifacts).
  • remote repository - accessed by Aether with the help of org.eclipse.aether.repository.RemoteRepository interface. Effectively remote repository is a wrapper for URI, a set of policies related to snapshot/release versions plus proxy, mirroring and authentication information.

The key point is that if an artifact can't be found in local repository it is being searched for in (one of the) remote repositories. Proper code should ensure that local repositories are always searched before remote repositories.

Logs

For debugging purposes it is very helpful to see all the operations in logs. We can increase logging level for few loggers (etc/org.ops4j.pax.logging.cfg configuration file in Karaf):

log4j.logger.org.eclipse.aether = DEBUG
log4j.logger.org.apache.http.headers = DEBUG
log4j.logger.shaded.org.eclipse.aether = DEBUG
log4j.logger.shaded.org.apache.http.headers = DEBUG

The loggers with shaded. are necessary when using pax-url-aether in Karaf, because pax-url-aether uses shaded (private) versions of some 3rd party libraries - like Aether or httpclient.

Also, we'll add another remote repository to see how Aether checks them all:

req.addRepository(new RemoteRepository.Builder("jboss-public", "default", "https://repository.jboss.org/nexus/content/groups/public").build());
req.addRepository(new RemoteRepository.Builder("central", "default", "http://repo1.maven.org/maven2").build());

Here are the logs when commons-io:commons-io:2.5:jar artifact is resolved and it is not available in local repository:

11:13:47.181
 DEBUG {main} [o.e.a.i.i.DefaultLocalRepositoryProvider] : Using manager
 EnhancedLocalRepositoryManager with priority 10.0 for 
target/repo-1469178827169
11:13:47.188 INFO  {main} [g.t.m.a.AetherTest] : Request: commons-io:commons-io:jar:2.5 < [jboss-public (https://repository.jboss.org/nexus/content/groups/public, default, releases+snapshots), central (http://repo1.maven.org/maven2, default, releases+snapshots)]
11:13:47.631 DEBUG {main} [o.e.a.i.i.DefaultTransporterProvider] : Using transporter HttpTransporter with priority 5.0 for https://repository.jboss.org/nexus/content/groups/public
11:13:47.632
 DEBUG {main} [o.e.a.i.i.DefaultRepositoryConnectorProvider] : Using 
connector BasicRepositoryConnector with priority 0.0 for https://repository.jboss.org/nexus/content/groups/public
11:13:49.015
 DEBUG {main} [o.a.h.headers] : >> GET 
/nexus/content/groups/public/commons-io/commons-io/2.5/commons-io-2.5.jar
 HTTP/1.1
11:13:49.015 DEBUG {main} [o.a.h.headers] : >> Host: repository.jboss.org
...
11:13:49.385 DEBUG {main} [o.a.h.headers] : << HTTP/1.1 404 Not Found
...
11:13:49.572 DEBUG {main} [o.e.a.i.i.DefaultTransporterProvider] : Using transporter HttpTransporter with priority 5.0 for http://repo1.maven.org/maven2
11:13:49.572
 DEBUG {main} [o.e.a.i.i.DefaultRepositoryConnectorProvider] : Using 
connector BasicRepositoryConnector with priority 0.0 for http://repo1.maven.org/maven2
11:13:49.704 DEBUG {main} [o.a.h.headers] : >> GET /maven2/commons-io/commons-io/2.5/commons-io-2.5.jar HTTP/1.1
11:13:49.705 DEBUG {main} [o.a.h.headers] : >> Host: repo1.maven.org
...
11:13:49.770 DEBUG {main} [o.a.h.headers] : << HTTP/1.1 200 OK
...
11:13:50.079 DEBUG {main} [o.a.h.headers] : >> GET /maven2/commons-io/commons-io/2.5/commons-io-2.5.jar.sha1 HTTP/1.1
11:13:50.079 DEBUG {main} [o.a.h.headers] : >> Host: repo1.maven.org
...
11:13:50.145 DEBUG {main} [o.a.h.headers] : << HTTP/1.1 200 OK
...
11:13:50.156
 DEBUG {main} [o.e.a.i.i.EnhancedLocalRepositoryManager] : Writing 
tracking file 
/data/ggrzybek/sources/_testing/grgr-test-maven/target/repo-1469178827169/commons-io/commons-io/2.5/_remote.repositories
11:13:50.161 INFO  {main} [g.t.m.a.AetherTest] : Result: commons-io:commons-io:jar:2.5 < central (http://repo1.maven.org/maven2, default, releases+snapshots)

As we can see here's the sequence of events:

  1. Aether uses local repository at target/repo-1469178827169 location
  2. https://repository.jboss.org/nexus/content/groups/public is checked first and we get HTTP 404
  3. http://repo1.maven.org/maven2 is checked next and we get HTTP 200
  4. Aether fetches SHA1 checksum then for found artifact
  5. Aether writes tracking file at target/repo-1469178827169/commons-io/commons-io/2.5/_remote.repositories that looks like this:

    #NOTE: This is an Aether internal implementation file, its format can be changed without prior notice.
    #Fri Jul 22 11:13:50 CEST 2016
    commons-io-2.5.jar>central=

    This file allows us to recall where the artifact was downloaded from.

SNAPSHOTs

Let's see how Aether works when resolving SNAPSHOT versions. We'll reuse the same remote repositories as before. By default using new RemoteRepository.Builder("central", "default", "http://repo1.maven.org/maven2").build() gives us remote repository that's enabled regardless of whether we use the repository to resolve SNAPSHOT or non-SNAPSHOT artifacts. We can of course change it:

RemoteRepository.Builder b1 = new RemoteRepository.Builder("central", "default", "http://repo1.maven.org/maven2");
RemoteRepository.Builder b2 = new RemoteRepository.Builder("jboss-public", "default", "https://repository.jboss.org/nexus/content/groups/public");
RepositoryPolicy enabledPolicy = new RepositoryPolicy(true, RepositoryPolicy.UPDATE_POLICY_ALWAYS, RepositoryPolicy.CHECKSUM_POLICY_FAIL);
RepositoryPolicy disabledPolicy = new RepositoryPolicy(false, RepositoryPolicy.UPDATE_POLICY_ALWAYS, RepositoryPolicy.CHECKSUM_POLICY_FAIL);
b1.setReleasePolicy(enabledPolicy);
b1.setSnapshotPolicy(enabledPolicy);
b2.setReleasePolicy(disabledPolicy);
b2.setSnapshotPolicy(enabledPolicy);
req.addRepository(b1.build());
req.addRepository(b2.build());

In the above example, we explicitly enable resolving SNAPSHOT artifacts in central and jboss-public repositories. We won't try to resolve non-SNAPSHOT artifacts in jboss-public. Here are the logs related to resolving commons-io:commons-io:2.5-SNAPSHOT:jar:

12:11:17.195 DEBUG {main} [o.e.a.i.i.DefaultLocalRepositoryProvider] : Using manager EnhancedLocalRepositoryManager with priority 10.0 for target/repo-1469182277187
12:11:17.201 INFO  {main} [g.t.m.a.AetherTest] : Request: commons-io:commons-io:jar:2.5-SNAPSHOT < [central (http://repo1.maven.org/maven2, default, releases+snapshots), jboss-public (https://repository.jboss.org/nexus/content/groups/public, default, snapshots)]
12:11:17.851 DEBUG {DefaultMetadataResolver-0-1} [o.e.a.i.i.DefaultTransporterProvider] : Using transporter HttpTransporter with priority 5.0 for https://repository.jboss.org/nexus/content/groups/public
12:11:17.852 DEBUG {DefaultMetadataResolver-0-1} [o.e.a.i.i.DefaultRepositoryConnectorProvider] : Using connector BasicRepositoryConnector with priority 0.0 for https://repository.jboss.org/nexus/content/groups/public
12:11:17.853 DEBUG {DefaultMetadataResolver-0-0} [o.e.a.i.i.DefaultTransporterProvider] : Using transporter HttpTransporter with priority 5.0 for http://repo1.maven.org/maven2
12:11:17.854 DEBUG {DefaultMetadataResolver-0-0} [o.e.a.i.i.DefaultRepositoryConnectorProvider] : Using connector BasicRepositoryConnector with priority 0.0 for http://repo1.maven.org/maven2
12:11:18.158 DEBUG {DefaultMetadataResolver-0-0} [o.a.h.headers] : >> GET /maven2/commons-io/commons-io/2.5-SNAPSHOT/maven-metadata.xml HTTP/1.1
12:11:18.158 DEBUG {DefaultMetadataResolver-0-0} [o.a.h.headers] : >> Host: repo1.maven.org
...
12:11:18.225 DEBUG {DefaultMetadataResolver-0-0} [o.a.h.headers] : << HTTP/1.1 404 Not Found
...
12:11:18.245 DEBUG {DefaultMetadataResolver-0-0} [o.e.a.i.i.DefaultUpdateCheckManager] : Writing tracking file /data/ggrzybek/sources/_testing/grgr-test-maven/target/repo-1469182277187/commons-io/commons-io/2.5-SNAPSHOT/resolver-status.properties
12:11:19.332 DEBUG {DefaultMetadataResolver-0-1} [o.a.h.headers] : >> GET /nexus/content/groups/public/commons-io/commons-io/2.5-SNAPSHOT/maven-metadata.xml HTTP/1.1
12:11:19.332 DEBUG {DefaultMetadataResolver-0-1} [o.a.h.headers] : >> Host: repository.jboss.org
...
12:11:19.611 DEBUG {DefaultMetadataResolver-0-1} [o.a.h.headers] : << HTTP/1.1 200 OK
...
12:11:19.850 DEBUG {DefaultMetadataResolver-0-1} [o.a.h.headers] : >> GET /nexus/content/groups/public/commons-io/commons-io/2.5-SNAPSHOT/maven-metadata.xml.sha1 HTTP/1.1
12:11:19.850 DEBUG {DefaultMetadataResolver-0-1} [o.a.h.headers] : >> Host: repository.jboss.org
...
12:11:20.079 DEBUG {DefaultMetadataResolver-0-1} [o.a.h.headers] : << HTTP/1.1 200 OK
...
12:11:20.082 DEBUG {DefaultMetadataResolver-0-1} [o.e.a.i.i.DefaultUpdateCheckManager] : Writing tracking file /data/ggrzybek/sources/_testing/grgr-test-maven/target/repo-1469182277187/commons-io/commons-io/2.5-SNAPSHOT/resolver-status.properties
12:11:20.107 DEBUG {main} [o.e.a.i.i.DefaultTransporterProvider] : Using transporter HttpTransporter with priority 5.0 for https://repository.jboss.org/nexus/content/groups/public
12:11:20.107 DEBUG {main} [o.e.a.i.i.DefaultRepositoryConnectorProvider] : Using connector BasicRepositoryConnector with priority 0.0 for https://repository.jboss.org/nexus/content/groups/public
12:11:20.694 DEBUG {main} [o.a.h.headers] : >> GET /nexus/content/groups/public/commons-io/commons-io/2.5-SNAPSHOT/commons-io-2.5-20151119.212356-154.jar HTTP/1.1
12:11:20.694 DEBUG {main} [o.a.h.headers] : >> Host: repository.jboss.org
...
12:11:20.901 DEBUG {main} [o.a.h.headers] : << HTTP/1.1 200 OK
...
12:11:21.590 DEBUG {main} [o.e.a.i.i.EnhancedLocalRepositoryManager] : Writing tracking file /data/ggrzybek/sources/_testing/grgr-test-maven/target/repo-1469182277187/commons-io/commons-io/2.5-SNAPSHOT/_remote.repositories
12:11:21.591 INFO  {main} [g.t.m.a.AetherTest] : Result: commons-io:commons-io:jar:2.5-20151119.212356-154 < jboss-public (https://repository.jboss.org/nexus/content/groups/public, default, snapshots)

here's the sequence of events:

  1. Aether uses local repository at target/repo-1469182277187 location
  2. commons-io/commons-io/2.5-SNAPSHOT/maven-metadata.xml metadata artifacts are fetched in parallel from both remote repositories.
  3. Metadata is found only in jboss-public repository
  4. Aether fetches metadata SHA1 checksum
  5. target/repo-1469182277187/commons-io/commons-io/2.5-SNAPSHOT/resolver-status.properties is written to track the information about metadata
  6. target/repo-1469182277187/commons-io/commons-io/2.5-SNAPSHOT/maven-metadata-jboss-public.xml file shows that 2.5-20151119.212356-154 is the latest version of SNAPSHOT artifact
  7. Aether downloads commons-io/commons-io/2.5-SNAPSHOT/commons-io-2.5-20151119.212356-154.jar from jboss-public
  8. Aether writes tracking file at target/repo-1469182277187/commons-io/commons-io/2.5-SNAPSHOT/_remote.repositories that looks like this:

    #NOTE: This is an Aether internal implementation file, its format can be changed without prior notice.
    #Fri Jul 22 12:11:21 CEST 2016
    commons-io-2.5-20151119.212356-154.jar>jboss-public=

    This file allows us to recall where the artifact was downloaded from.

There's one more thing worth noting - this time Aether invokes some operations in separate threads (DefaultMetadataResolver-0-* threads in addition to main thread). Aether usually does that when doing more than one task at a time.

With non-SNAPSHOT artifact resolution, we checked one repository at a time, because we had one task - org.eclipse.aether.resolution.ArtifactRequest

With SNAPSHOT artifact resolution, Aether internally invokes two org.eclipse.aether.resolution.MetadataRequest tasks (one for each remote repository) to find the latest SNAPSHOT.

The number of threads used in this operation can be controlled with aether.metadataResolver.threads configuration property.

Summary

This page was meant to provide some background, low-level information and basic concepts like local repository and remote repository. Mvn Protocol provides information about pax-url-aether itself.