Avoid platform specific dependencies in SBT packageBin

In a Java11 project (actually Java Play-Framework) I depend directly on org.deeplearning4j:deeplearning4j-nlp:1.0.0-M2 in my build.sbt running SBT 1.5.5.

That dependency transitively depends on several other libraries (like e.g. org.bytedeco:ffmpeg:5.0-1.5.7) that exists as platform dependent libraries and seem to use the SBT “classifier” to indicate the platform .. linux-x86, linux-x86_64, android-x86_64, windows… and so on. Now with my SBT commands like sbt universal:packageBin that yields me all platform versions of all of those quite large jars in my final “fat-jar” outcome and the result is 8 times as big as needed. The other native packager variants like windows:packageBin or debian:packageBin behave the same and include those jars in all platform variants.

I dont know how to set globally, that for any transitive dependency, I want to exclude the classifier X, Y and Z (in my case I would exclude the platforms that are not relevant for me)

Even the verbose way of excluding each unwanted one doesnt work:

excludeDependencies  += "org.bytedeco" % "ffmpeg" % "5.0-1.5.7" classifier "macosx-x86_64"

.. still has the “org.bytedeco.ffmpeg-5.0-1.5.7-macosx-x86_64.jar” in my final fat-jar artifact.

---

Here the redacted build.sbt

import com.typesafe.sbt.packager.SettingsHelper._

name := """my-name"""

lazy val isSubmodule = !file("./root.txt").exists

// common properties ///////////////////////////////////////////////////////////////

lazy val commonPropsPath = if (isSubmodule) { "../../common_conf/config.properties" }
  else { "./common_conf/config.properties"}

lazy val commonProps = settingKey[java.util.Properties]("The application properties")

commonProps := {
  val prop = new java.util.Properties()
  IO.load(prop, new File(commonPropsPath))
  prop
}

organization := commonProps.value.getProperty("organization")
version := commonProps.value.getProperty("version")
scalaVersion := commonProps.value.getProperty("scalaVersion")

// ######### javacOptions #########
javacOptions ++= Seq("-source", "1.10", "-target", "1.10")
javacOptions ++= Seq("-Xlint:unchecked", "-Xlint:cast")
javacOptions ++= Seq("-Werror")
javacOptions ++= Seq("-Xlint:-options")

// ######### scalacOptions #########
scalacOptions ++=  Seq("-target:jvm-1.8")
scalacOptions in(Compile, doc) += "-no-java-comments"

// common properties ///////////////////////////////////////////////////////////////


lazy val matching = (project in file("."))
  .enablePlugins(PlayJava, PlayEbean, GitVersioning, BuildInfoPlugin, JavaAppPackaging, DebianPlugin, SbtWeb)
  .settings(
    ...... )


Compile / playEbeanModels := Seq("my.code.package.**")
playEbeanDebugLevel := 4

libraryDependencies ++= Seq(
  javaJdbc,
  filters,
  javaWs,
  guice,

  "mysql" % "mysql-connector-java" % "8.0.29",
  "org.liquibase" % "liquibase-core" % "4.11.0",

  "org.deeplearning4j" % "deeplearning4j-nlp" % "1.0.0-M2" ,  
  "org.nd4j" % "nd4j-native-platform" % "1.0.0-M2",
  "org.nd4j" % "nd4j-native" % "1.0.0-M2",
  "org.apache.lucene" % "lucene-snowball" % "3.0.3",
  "org.jsoup" % "jsoup" % "1.15.+", 

  "org.projectlombok" % "lombok" % "1.18.24", 
  "io.ebean" % "ebean-externalmapping-xml" % "12.6.1", 

  "commons-io" % "commons-io" % "2.11.+",
  "commons-codec" % "commons-codec" % "1.15",
  "org.apache.commons" % "commons-lang3" % "3.12.+",


  "javax.xml.bind" % "jaxb-api" % "2.3.1",
  "com.sun.xml.bind" % "jaxb-core" % "2.3.0.1",
  "com.sun.xml.bind" % "jaxb-impl" % "2.3.1"

)

// test here to exclude each unwanted dependency individually
excludeDependencies  += "org.bytedeco" % "ffmpeg" % "5.0-1.5.7" classifier "macosx-x86_64"
excludeDependencies  += "org.bytedeco" % "ffmpeg" % "5.0-1.5.7" classifier "windows-x86_64"
excludeDependencies  += "org.bytedeco" % "ffmpeg" % "5.0-1.5.7" classifier "windows-x86"


resolvers ++= Seq(
  Resolver.mavenLocal,
  Resolver.sonatypeRepo("snapshots")
)

Test / fork := false

updateOptions := updateOptions.value.withCachedResolution(true)

if (isSubmodule) {
  jcheckStyleConfig := "../../conf/checkstyle-config.xml"
}
else {
  jcheckStyleConfig := "./conf/checkstyle-config.xml"
}

Test / jacocoExcludes := Seq(
  "**.*"
)

// Debian package description
Linux / maintainer := "My values here"
Linux / packageSummary := "My values here"
packageDescription := "My values here"

publishTo := {
 ........
}

credentials += Credentials(.......)

publish in Debian := (publish in Debian).dependsOn(publish).value

Compile / packageBin / publishArtifact := true
Compile / packageDoc / publishArtifact := false
Compile / packageSrc / publishArtifact := false
Test / publishArtifact := false

makeDeploymentSettings(Debian, packageBin in Debian, "deb")

Answer

I solved it for my specific case now. First I opened a request on the DL4J forum (https://community.konduit.ai/t/limit-native-dependencies-to-one-platform-on-sbt/2005/2) and got similar suggestions as here.

In short – all the “SBT-JavaCV” related settings did not show any results, I dont know if it was active at all and I didnt find log output so its unclear to me, what I missed there.

So in the end I changed in my build.sbt to the following:

1. I added to my “deeplearning4j-nlp” dependency these transitive excludes in the section libraryDependencies ++= Seq(:

  "org.deeplearning4j" % "deeplearning4j-nlp" % "1.0.0-M2"  excludeAll(
    ExclusionRule(organization = "org.bytedeco", name = "ffmpeg-platform"),
    ExclusionRule(organization = "org.bytedeco", name = "hdf5-platform"),
    ExclusionRule(organization = "org.bytedeco", name = "hdf5-platform"),
    ExclusionRule(organization = "org.bytedeco", name = "javacpp-platform"),
    ExclusionRule(organization = "org.bytedeco", name = "leptonica-platform"),
    ExclusionRule(organization = "org.bytedeco", name = "openblas-platform"),
    ExclusionRule(organization = "org.bytedeco", name = "opencv-platform")
  ),

.. and that avoided the xyz-plattform jars to come into my fat-jar outcome. Then after it, I just explicitly named the platforms that Im interested in via the “classifier”:

 "org.bytedeco" % "ffmpeg" % "5.0-1.5.7"        classifier "windows-x86_64" classifier "linux-x86_64",
  "org.bytedeco" % "hdf5" % "1.12.1-1.5.7"       classifier "windows-x86_64" classifier "linux-x86_64",
  "org.bytedeco" % "javacpp" % "1.5.7"           classifier "windows-x86_64" classifier "linux-x86_64",
  "org.bytedeco" % "leptonica" % "1.82.0-1.5.7"  classifier "windows-x86_64" classifier "linux-x86_64",
  "org.bytedeco" % "openblas" % "0.3.19-1.5.7"   classifier "windows-x86_64" classifier "linux-x86_64",
  "org.bytedeco" % "opencv" % "4.5.5-1.5.7"      classifier "windows-x86_64" classifier "linux-x86_64",

And this yields me a fat-jar with just the platform libs that I specify in that “classifier”.

Anyways, @SamuelAudet & @AdamGibson thanks for you time and guidance.