In a Java11 project (actually Java Play-Framework) I depend directly on org.deeplearning4j:deeplearning4j-nlp:1.0.0-M2 in my build.sbt running SBT 1.5.5.
That dependency transitively depends on several other libraries (like e.g. org.bytedeco:ffmpeg:5.0-1.5.7) that exists as platform dependent libraries and seem to use the SBT “classifier” to indicate the platform .. linux-x86, linux-x86_64, android-x86_64, windows… and so on. Now with my SBT commands like sbt universal:packageBin that yields me all platform versions of all of those quite large jars in my final “fat-jar” outcome and the result is 8 times as big as needed.
The other native packager variants like windows:packageBin or debian:packageBin behave the same and include those jars in all platform variants.
I dont know how to set globally, that for any transitive dependency, I want to exclude the classifier X, Y and Z (in my case I would exclude the platforms that are not relevant for me)
Even the verbose way of excluding each unwanted one doesnt work:
excludeDependencies += "org.bytedeco" % "ffmpeg" % "5.0-1.5.7" classifier "macosx-x86_64"
.. still has the “org.bytedeco.ffmpeg-5.0-1.5.7-macosx-x86_64.jar” in my final fat-jar artifact.
Here the redacted build.sbt
import com.typesafe.sbt.packager.SettingsHelper._
name := """my-name"""
lazy val isSubmodule = !file("./root.txt").exists
// common properties ///////////////////////////////////////////////////////////////
lazy val commonPropsPath = if (isSubmodule) { "../../common_conf/config.properties" }
else { "./common_conf/config.properties"}
lazy val commonProps = settingKey[java.util.Properties]("The application properties")
commonProps := {
val prop = new java.util.Properties()
IO.load(prop, new File(commonPropsPath))
prop
}
organization := commonProps.value.getProperty("organization")
version := commonProps.value.getProperty("version")
scalaVersion := commonProps.value.getProperty("scalaVersion")
// ######### javacOptions #########
javacOptions ++= Seq("-source", "1.10", "-target", "1.10")
javacOptions ++= Seq("-Xlint:unchecked", "-Xlint:cast")
javacOptions ++= Seq("-Werror")
javacOptions ++= Seq("-Xlint:-options")
// ######### scalacOptions #########
scalacOptions ++= Seq("-target:jvm-1.8")
scalacOptions in(Compile, doc) += "-no-java-comments"
// common properties ///////////////////////////////////////////////////////////////
lazy val matching = (project in file("."))
.enablePlugins(PlayJava, PlayEbean, GitVersioning, BuildInfoPlugin, JavaAppPackaging, DebianPlugin, SbtWeb)
.settings(
...... )
Compile / playEbeanModels := Seq("my.code.package.**")
playEbeanDebugLevel := 4
libraryDependencies ++= Seq(
javaJdbc,
filters,
javaWs,
guice,
"mysql" % "mysql-connector-java" % "8.0.29",
"org.liquibase" % "liquibase-core" % "4.11.0",
"org.deeplearning4j" % "deeplearning4j-nlp" % "1.0.0-M2" ,
"org.nd4j" % "nd4j-native-platform" % "1.0.0-M2",
"org.nd4j" % "nd4j-native" % "1.0.0-M2",
"org.apache.lucene" % "lucene-snowball" % "3.0.3",
"org.jsoup" % "jsoup" % "1.15.+",
"org.projectlombok" % "lombok" % "1.18.24",
"io.ebean" % "ebean-externalmapping-xml" % "12.6.1",
"commons-io" % "commons-io" % "2.11.+",
"commons-codec" % "commons-codec" % "1.15",
"org.apache.commons" % "commons-lang3" % "3.12.+",
"javax.xml.bind" % "jaxb-api" % "2.3.1",
"com.sun.xml.bind" % "jaxb-core" % "2.3.0.1",
"com.sun.xml.bind" % "jaxb-impl" % "2.3.1"
)
// test here to exclude each unwanted dependency individually
excludeDependencies += "org.bytedeco" % "ffmpeg" % "5.0-1.5.7" classifier "macosx-x86_64"
excludeDependencies += "org.bytedeco" % "ffmpeg" % "5.0-1.5.7" classifier "windows-x86_64"
excludeDependencies += "org.bytedeco" % "ffmpeg" % "5.0-1.5.7" classifier "windows-x86"
resolvers ++= Seq(
Resolver.mavenLocal,
Resolver.sonatypeRepo("snapshots")
)
Test / fork := false
updateOptions := updateOptions.value.withCachedResolution(true)
if (isSubmodule) {
jcheckStyleConfig := "../../conf/checkstyle-config.xml"
}
else {
jcheckStyleConfig := "./conf/checkstyle-config.xml"
}
Test / jacocoExcludes := Seq(
"**.*"
)
// Debian package description
Linux / maintainer := "My values here"
Linux / packageSummary := "My values here"
packageDescription := "My values here"
publishTo := {
........
}
credentials += Credentials(.......)
publish in Debian := (publish in Debian).dependsOn(publish).value
Compile / packageBin / publishArtifact := true
Compile / packageDoc / publishArtifact := false
Compile / packageSrc / publishArtifact := false
Test / publishArtifact := false
makeDeploymentSettings(Debian, packageBin in Debian, "deb")
Advertisement
Answer
I solved it for my specific case now. First I opened a request on the DL4J forum (https://community.konduit.ai/t/limit-native-dependencies-to-one-platform-on-sbt/2005/2) and got similar suggestions as here.
In short – all the “SBT-JavaCV” related settings did not show any results, I dont know if it was active at all and I didnt find log output so its unclear to me, what I missed there.
So in the end I changed in my build.sbt to the following:
- I added to my “deeplearning4j-nlp” dependency these transitive excludes in the section libraryDependencies ++= Seq(:
"org.deeplearning4j" % "deeplearning4j-nlp" % "1.0.0-M2" excludeAll(
ExclusionRule(organization = "org.bytedeco", name = "ffmpeg-platform"),
ExclusionRule(organization = "org.bytedeco", name = "hdf5-platform"),
ExclusionRule(organization = "org.bytedeco", name = "hdf5-platform"),
ExclusionRule(organization = "org.bytedeco", name = "javacpp-platform"),
ExclusionRule(organization = "org.bytedeco", name = "leptonica-platform"),
ExclusionRule(organization = "org.bytedeco", name = "openblas-platform"),
ExclusionRule(organization = "org.bytedeco", name = "opencv-platform")
),
.. and that avoided the xyz-plattform jars to come into my fat-jar outcome. Then after it, I just explicitly named the platforms that Im interested in via the “classifier”:
"org.bytedeco" % "ffmpeg" % "5.0-1.5.7" classifier "windows-x86_64" classifier "linux-x86_64", "org.bytedeco" % "hdf5" % "1.12.1-1.5.7" classifier "windows-x86_64" classifier "linux-x86_64", "org.bytedeco" % "javacpp" % "1.5.7" classifier "windows-x86_64" classifier "linux-x86_64", "org.bytedeco" % "leptonica" % "1.82.0-1.5.7" classifier "windows-x86_64" classifier "linux-x86_64", "org.bytedeco" % "openblas" % "0.3.19-1.5.7" classifier "windows-x86_64" classifier "linux-x86_64", "org.bytedeco" % "opencv" % "4.5.5-1.5.7" classifier "windows-x86_64" classifier "linux-x86_64",
And this yields me a fat-jar with just the platform libs that I specify in that “classifier”.
Anyways, @SamuelAudet & @AdamGibson thanks for you time and guidance.