[TOREE-557] Bump Spark 3.5

[pan3793]

Spark 3.4.x is EOL. Under the updated Spark versioning policy, 3.5.x has an extended LTS window until November 2027 (security fixes only), so it remains the right target for Toree. This PR moves Toree to Spark 3.5.9, the latest 3.5.x release.

https://spark.apache.org/versioning-policy.html

The PR also:

• Aligns bundled dependencies with what Spark 3.5 provides at runtime: jackson-databind 2.15.2, coursier 2.0.16, py4j 0.10.9.7, and a scala-collection-compat 2.7.0 override. etc/legal records the resolved versions.
• Scans plugin classes with ASM 9 (asm, asm-commons, asm-util) instead of the ASM 7.1 that arrives with classutil.

[pan3793]

some modern version libs, like Jackson, use multi release jars, which main classes are built against Java 8, but ship optional classes compiled with higher JDK version, we must use newer ASM lib to make it work.

see also bmc/classutil#45

[pan3793]

Upgrade ASM libs used by org.clapper:classutil to address

25/10/07 03:10:47 WARN Main$$anon$1: No external magics provided to PluginManager!
Exception in thread "main" java.lang.IllegalArgumentException: Unsupported class file major version 61
	at shadeasm.org.objectweb.asm.ClassReader.<init>(ClassReader.java:195)
	at shadeasm.org.objectweb.asm.ClassReader.<init>(ClassReader.java:176)
	at shadeasm.org.objectweb.asm.ClassReader.<init>(ClassReader.java:162)
	at shadeasm.org.objectweb.asm.ClassReader.<init>(ClassReader.java:283)
	at shadeclapper.org.clapper.classutil.asm.ClassFile$.load(ClassFinderImpl.scala:222)
	at shadeclapper.org.clapper.classutil.ClassFinder.classData(ClassFinder.scala:404)
	at shadeclapper.org.clapper.classutil.ClassFinder.$anonfun$processOpenZip$2(ClassFinder.scala:359)
	at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
	at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492)
	at scala.collection.Iterator.toStream(Iterator.scala:1417)
	at scala.collection.Iterator.toStream$(Iterator.scala:1416)
	at scala.collection.AbstractIterator.toStream(Iterator.scala:1431)
	at scala.collection.Iterator.$anonfun$toStream$1(Iterator.scala:1417)
	at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1173)
	at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1163)
	at scala.collection.immutable.Stream.$anonfun$$plus$plus$1(Stream.scala:372)
	at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1173)
	at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1163)
	at scala.collection.immutable.StreamIterator.$anonfun$next$1(Stream.scala:1061)
	at scala.collection.immutable.StreamIterator$LazyCell.v$lzycompute(Stream.scala:1050)
	at scala.collection.immutable.StreamIterator$LazyCell.v(Stream.scala:1050)
	at scala.collection.immutable.StreamIterator.hasNext(Stream.scala:1055)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
	at scala.collection.Iterator.foreach(Iterator.scala:943)
	at scala.collection.Iterator.foreach$(Iterator.scala:943)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
	at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:62)
	at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:53)
	at scala.collection.immutable.Map$MapBuilderImpl.$plus$plus$eq(Map.scala:648)
	at scala.collection.immutable.Map$MapBuilderImpl.$plus$plus$eq(Map.scala:595)
	at scala.collection.TraversableOnce.toMap(TraversableOnce.scala:372)
	at scala.collection.TraversableOnce.toMap$(TraversableOnce.scala:370)
	at scala.collection.AbstractIterator.toMap(Iterator.scala:1431)
	at shadeclapper.org.clapper.classutil.ClassFinder$.classInfoMap(ClassFinder.scala:445)
	at org.apache.toree.plugins.PluginSearcher.loadClassMap(PluginSearcher.scala:80)
	at org.apache.toree.plugins.PluginSearcher.internalClassInfo$lzycompute(PluginSearcher.scala:36)
	at org.apache.toree.plugins.PluginSearcher.internalClassInfo(PluginSearcher.scala:35)
	at org.apache.toree.plugins.PluginSearcher.internal$lzycompute(PluginSearcher.scala:39)
	at org.apache.toree.plugins.PluginSearcher.internal(PluginSearcher.scala:39)
	at org.apache.toree.plugins.PluginManager.internalPlugins$lzycompute(PluginManager.scala:45)
	at org.apache.toree.plugins.PluginManager.internalPlugins(PluginManager.scala:44)
	at org.apache.toree.plugins.PluginManager.initialize(PluginManager.scala:80)
	at org.apache.toree.boot.layer.StandardComponentInitialization.initializePlugins(ComponentInitialization.scala:219)
	at org.apache.toree.boot.layer.StandardComponentInitialization.initializeComponents(ComponentInitialization.scala:83)
	at org.apache.toree.boot.layer.StandardComponentInitialization.initializeComponents$(ComponentInitialization.scala:69)
	at org.apache.toree.Main$$anon$1.initializeComponents(Main.scala:35)
	at org.apache.toree.boot.KernelBootstrap.initialize(KernelBootstrap.scala:102)
	at org.apache.toree.Main$.delayedEndpoint$org$apache$toree$Main$1(Main.scala:35)
	at org.apache.toree.Main$delayedInit$body.apply(Main.scala:24)
	at scala.Function0.apply$mcV$sp(Function0.scala:39)
	at scala.Function0.apply$mcV$sp$(Function0.scala:39)
	at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:17)
	at scala.App.$anonfun$main$1$adapted(App.scala:80)
	at scala.collection.immutable.List.foreach(List.scala:431)
	at scala.App.main(App.scala:80)
	at scala.App.main$(App.scala:78)
	at org.apache.toree.Main$.main(Main.scala:24)
	at org.apache.toree.Main.main(Main.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1034)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:199)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:222)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1125)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1134)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

[pjfanning]

@lresende can we get this merged? Spark 3.4 is well past its end of life date.
https://endoflife.date/apache-spark