Native library lz4 not available for Spark

Question

How do I add the lz4 native libraries for use by Spark workers?

I have tried to add them via both LD_LIBRARY_PATH and ( as shown - but no accepted or even upvoted answer - in Apache Spark Native Libraries ) - in SPARK_LIBRARY_PATH. They are not working: we get:

java.lang.RuntimeException: native lz4 library not available
  at org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
  at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)
  at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:165)
  at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1201)
  at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1094)
  at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.<init>(SequenceFile.java:1444)
  at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:277)
  at BIDMat.HDFSIO.writeThing(HDFSIO.scala:96)

Here is the LD_LIBRARY_PATH

$echo $LD_LIBRARY_PATH
/usr/local/Cellar/lz4/r131/lib:/usr/local/Cellar/hadoop/2.7.2/libexec/lib:
12:15:35/BIDMach_Spark $ll /usr/local/Cellar/lz4/r131/lib

and the contents of the lz4 related entry:

$ll /usr/local/Cellar/lz4/r131/lib
total 528
-r--r--r--  1 macuser  admin  71144 Sep 21  2015 liblz4.a
drwxr-xr-x  7 macuser  admin    238 Sep 21  2015 .
drwxr-xr-x  3 macuser  admin    102 Jun 13 10:41 pkgconfig
-r--r--r--  1 macuser  admin  64120 Jun 13 10:41 liblz4.dylib
-r--r--r--  1 macuser  admin  64120 Jun 13 10:41 liblz4.1.dylib
-r--r--r--  1 macuser  admin  64120 Jun 13 10:41 liblz4.1.7.1.dylib

So I don't use OSX but you have no `.so` (`.bundle`?) files which is pretty much what you need here. — zero323, Jun 13 '16 at 19:33
@zero323 Good question - I did not know the answer myself. Apparently .dylib is the mac equiv. http://stackoverflow.com/questions/2339679/what-are-the-differences-between-so-and-dylib-on-osx — WestCoastProjects, Jun 13 '16 at 19:35
You need the `MH_BUNDLE` not `MH_DYLIB` (now I am smart ;) ) — zero323, Jun 13 '16 at 19:39
@zero323 This is well deeper than where I have gone so far. I simply installed the `lz4` via `brew` - and am attempting to use those results. If they (apparently) used MH_DYLIB on their gc++ options that is outside of my "reach". Maybe I have to figure out how to compile native hadoop libraries on the mac. Sounding ominous. — WestCoastProjects, Jun 13 '16 at 19:48
Don't ask me. While it is good UNIX all these brews and stuff sound like a black magic ;) Still, I am fairly certain this is the issue. On a side note I answered the other question but it won't be useful here because the warning is about `libhadoop`. — zero323, Jun 13 '16 at 19:57
Thanks. Couldn't you just adjust the recipe to generate shared libraries? — zero323, Jun 13 '16 at 20:27
@zero323 I went closer to the source and downloaded `lz4` source code. Unfortunately their `Makefile` only has `-shared` and not `-dynamiclib` in it: still working through how to translate that to `-module`. This is bit more adventurous than I would prefer. — WestCoastProjects, Jun 13 '16 at 20:28
At the first glance `make lib` / `make install` should give you all you need. — zero323, Jun 13 '16 at 20:36
That will just get us the same ol' .dynlib outputs. I am looking into full blown native hadoop build now. — WestCoastProjects, Jun 13 '16 at 20:50
Note that spark is a parallel processing framework, designed with servers in mind. There are many linux servers, some windows servers, but not many Mac servers. So thought it is not impossible to do most things on Mac OS. It may be simpler to work on (a vm of) a different OS. — Dennis Jaheruddin, Jul 31 '20 at 09:25

score -2 · Answer 1 · answered Oct 26 '20 at 18:51

-2

Update your hadoop jars and should work perfectly fine.

answered Oct 26 '20 at 18:51

Harshit

560
1
5
15

Native library lz4 not available for Spark

1 Answers1