srcdeps logo srcdeps

Source dependencies with Maven and Gradle

 

Peter Palaga

About me

Peter Palaga

Agenda

  • Source dependencies:

    • What is it

    • What is it good for

  • srcdeps implementation for Maven and Gradle

  • srcdeps demo

What are source dependencies?

The "usual" dependencies

  • Software projects often depend on artifacts produced by other projects

  • Those artifacts

    • typically contain compiled code

    • usually available in repositories (Maven Central, jcenter, …​) on the internet

  • Build tools (Maven, Gradle, …​) responsible for finding and downloading the dependencies

Source dependencies

  • The build of every git commit deterministic enough

    ⇒ Source code of a dependency enough to use it in a dependent project

  • The build tool has to:

    • Check out the sources (e.g. from a git repo)

    • Build the artifacts required by the dependent project

NB: Dependency artifacts do not need to be available in a repository

The idea is not new

Some languages (or their build tools) have it already

Ruby

Gemfile of a Ruby project

# A binary dependency pulled from rubygems.org
gem 'rack', '2.0.1'

# A source dependency built by Blunder
# from a git commit
gem 'rack',
    :git => 'https://github.com/rack/rack.git',
    :ref => '0bd839d'

Scala

build.sbt

// A binary dependency
libraryDependencies +=
    "com.typesafe.play" %% "play-slick" % "2.0.2"

// A source dependency
myProject
  .dependsOn(
    ProjectRef(
      uri("git://github.com/freekh/play-slick.git#v1.2.3")
      "play-slick"
    )
  )

What is this good for?

Testing and Integration

  • Test each commit of the dependency inside the dependent project

  • Find issues early

Multilayer Continuous Delivery

Multiple components in multiple separate source repositories

wf component dependencies

Fixing third-party code (1/2)

Dependency project dead or not releasing fast enough

  • Fork and use as a source dependency

  • No consent from the dependency project needed

Fixing third-party code (2/2)

Dependency project doing nasty things

  • Fork and accept only some of their changes (e.g. security fixes)

  • Throw away all that can harm your stability

  • Again, no consent from the dependency project needed

How is this different from Maven SNAPSHOTs?

SNAPSHOTs built locally

  • What you build is what you get

  • Handy on a developer’s machine

  • Hard to share:

    • CI machines

    • Teammates

Remote SNAPSHOTs are evil

  • You never know what you get

    • local/remote SNAPSHOT?

    • latest today != latest tomorrow

       

  • The build of a component depending on another SNAPSHOT component is not reproducible over time

  • Remote SNAPSHOTs should be always off

Source dependencies

  • As compared SNAPSHOTs:

    • Reproducible

    • Easy to share with teammates and CI

       

  • As compared to traditional releases:

    • Releases done just for the sake of testing and integration may be avoided

    • May eliminate the need for releases and the artifact repository infra altogether

srcdeps

  • Maven implementation since ~October 2015, now robust and stable

  • General parts moved to srcdeps-core in 2016 to be able to add support for other build tools

  • Gradle PoC recently

How srcdeps work (1/3)

Three basic ideas

  1. Coin a convention for version strings to express the commit ID to build the given dependency from

    <dependency>
      <groupId>org.my-group</groupId>
      <artifactId>my-artifact</artifactId>
      <version>1.2.3-SRC-revision-deadbeef</version>
    </dependency><!--             ⬑ a git commit ID  -->
  2. Provide a configuration that maps dependency artifacts to source repository URLs

How srcdeps work (2/3)

  1. Mechanism to trigger the build of the dependency:

    • Maven: custom implementation of the Local Maven Repository

    • Gradle PoC: srcdeps plugin scans the dependencies in project.afterEvaluate {}

How srcdeps work (3/3)

When an artifact with *-SRC-revision-{commitId} version is found

  • Find a git URL for it

  • Checkout the source to ~/.m2/srcdeps directory

  • Change the versions in the pom.xml/build.gradle files to whatever *-SRC-revision-{commitId} was requested

  • Build the dependency and install the resulting artifacts locally

  • The primary build then takes the artifacts from the Maven Local Repository

Demo: srcdeps with Maven

Let’s have a simple project

public class Demo {
  public String sayHello() {
    return "Hello World!";
  }
}

public class DemoTest {
  @Test
  public void sayHelloTest() {
    Assert.assertEquals("Hello World!", new Demo().sayHello());
  }
}

We need a new assertion

public class Demo {
  public String sayHello() {
    return "Hello World!";
  }
}

public class DemoTest {
  @Test
  public void sayHelloTest() {
    Assert.assertJavaViennaHello(new Demo().sayHello());
    //      ⬑ not available in the stock jUnit :(
  }
}

Clone JUnit

public class Assert {

  // Add the new method to org.junit.Assert
  public static void assertJavaViennaHello(String actual) {
    assertEquals(
      "Not the right conference!!!",
      "Hello Java Vienna!", actual
    );
  }
...

... and commit and push to your fork

Intialize srcdeps configuration in the depeendent project

mvn org.srcdeps.mvn:srcdeps-maven-plugin:3.0.1:init

That generates .mvn/extensions.xml and .mvn/srcdeps.yaml files for you

.mvn/extensions.xml

<extensions>
  <extension>
    <groupId>org.l2x6.srcdeps</groupId>
    <artifactId>srcdeps-maven-local-repository</artifactId>
    <version>3.0.1</version>
  </extension>
</extensions>
  • Maven Core Extensions since Maven 3.3.1

  • Allows for replacing substantial parts of Maven by our own custom implementations

  • Much more powerful than the plugin API

    • Parent, BoM imports are looked up earlier than any plugin code can be invoked

srcdeps.yaml

configModelVersion: 2.0
repositories:
  junit:
    selectors:
    - junit # a groupId[:artifactId[:version]] pattern
            # may contain * wildcards
    urls:
    - git:https://github.com/ppalaga/junit4.git
  • A mapping from artifacts to git URLs

  • Plus some other options

Upgrade junit in the dependent project

...
<dependency>
  <groupId>junit</groupId>
  <artifactId>junit</artifactId>
  <scope>test</scope>
  <version>4.13-SRC-revision-fd0a1c029b99277c955417b0c</version>
</dependency>
...

Build the dependent project

cd srcdeps-demo
mvn clean test
...
Failed tests:
  sayHelloTest(org.srcdeps.DemoTest): Not the right conference!
    expected:<Hello [Java Vienna]!> but was:<Hello [World]!>
...

Fix Demo.sayHello() to return "Hello Java Vienna!"

cd srcdeps-demo
mvn clean test
...
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
...
BUILD SUCCESS

Q.E.D.

Interesting locations

  • The source dependency was built under ${local.maven.repo.dir}/../srcdeps/${groupId}, typically ~/.m2/srcdeps/${groupId}

  • The source dependency was installed in the local Maven repo, typically `~/.m2/repository

Demo: srcdeps with Gradle

  • Gradle support PoC only

srcdeps features

  • Dependencies can refer to not only commits, but also branches and tags:

    • 1.2.3-SRC-revision-{myCommitId}

    • 1.2.3-SRC-branch-{myBranchName}

    • 1.2.3-SRC-revision-{myTagName}

       

  • In Maven, source dependencies work for vitually any kind of a dependency incl. parent, managed imports and even plugins

srcdeps.yaml config. options

Limitations (1/2)

Can you think of any?

  • Tools unaware of srcdeps (IDEs, static pom.xml analysers, …​) will see the -SRC- deps as non-available

  • Only Maven and git supported well ATM, Gradle on the way

  • However immutable git commits are, they can still disappear from repos, or even the whole repo can be deleted

    • Best practice: use srcdeps only against your own repos or your own mirrors of third party repos

Limitations (2/2)

  • There is still a few things that may differ among devs: java version, mvn version

    • Projects should use mvnw with a fixed mvn version

    • Enforcer plugin for Java version

  • The -SRC- artifacts (jars, wars, …​) built by you and me will typically not be binary equal

    • Typically will make no harm

    • Where would this matter actually?

 

Thanks!