Including and Reading Resources with Maven/Java/Scala

  In last few days, I tended to make a Maven dependency to read resources from its own jar. And I suddenly found that I have no experience of that before. After some googling and experiments, I managed to read the resources, and here is the notes.

  Let's say, you want to publish a jar library which associated a few default configuration files, of course, you never want to write the configuration into your code.(don't do that if you had ever thought about it before.) Then how do you keep the file in the jar? Experienced Java developer should know that jar is a compress file, or more literally, a zip file, so being able to keep a file inside it is reasonable, isn't it? If you are using Maven as your publishing tool, thing should be easy.

  Setting up your pom.xml, assign the resource setup like a boss, uh, I mean, like the following.

<project>
  ...
  <name>My Resources Plugin Practice Project</name>
  ...
  <build>
    ...
    <resources>
      <resource>
        <directory>src/resources</directory>
        <includes>
          <include>**/*</include>
        </includes>
      </resource>
      ...
    </resources>
    ...
  </build>
  ...
</project>

Of course you can assign wherever you want, but in practical, I would recommend to put the resources directory under src/, or src/main if you want to separate resources between the main and the test program.

  Another tip, for some files that are included for a specific package or class, you may want to put the files into the directory same as the package name. For example, I got a picture chick.jpg for net.sunshire.farm.Chick, I would put the picture at the path src/main/resources/net/sunshire/farm/chick.jpg. By doing so, Maven packages the chick.jpg in the package net/sunshire/farm, which allows you keep the independency of each jar.(You won't mass resources between different jars when you are importing many different dependencies.)

  Now you are done with including resources, if you package the jar with Maven, you can see the resource that you just included inside your jar file.(To inspect a jar file, simply change the extension to .zip and decompress it.)

  And how do we read the resource that we included with Scala? It is actually pretty simple, there is a method getResourceAsStream of Class. Here is the usage in Scala:

package net.sunshire.farm;

class Chick {
  val picPath: String = "/net/sunshire/farm/chick.jpg";
  // relative one: "chick.jpg", do NOT recommend
  val chickPic: InputStream = getClass.getResourceAsStream(picPath);
  // do whatever you want with the stream.
}

The above demonstration is pretty straightforward, I guess most most problem is that we do not know the method. The worth mentioning part is that getResourceAsStream takes either relative path or absolute path, same as UNIX/POSIX system ,/ represents root path. To clarify, the absolute path did not mean from the root of the file system, it is from the root of the jar. And the relative path is start from the package declared on the top(/net/sunshire/farm/ in the example).
  Another thing, if you specify the relative path, it is going to read the relative path of the run time package.(net.sunshire.house.Chicken extends Chick, and Chick specified the path with the relative one, then the path is going to be /net/sunshire/house/chick.jpg)

  If you do not understand Scala and need a Java example, the pom.xml part is the same, for the code part please refer to the 2nd reference. This is all about how to including and reading resources, hope it is helpful. :D

References:
[1] Resources Packaging - Maven Official
[2] Read a Resource in Java - Stack Overflow

Spark: Read/Write Sequence files

    Recently, I got some extraordinary demands for Spark RDD. I need an RDD which supports multi-key value pair, followed by IO operations. As usual, I started from python and I found that applying saveAsSequenceFile does not always work. After some searches, I assume it is because of the writing type of sequence file. Which is not demanded when writing, however, is demanded when reading(sequenceFile). After all, if I did not specify the output type, how should I know what type to read?

    After that, I think I had to quit my obsession on leveraging Spark with Python. I switched the language to Scala, which is fully supported to Spark. Nonetheless, story did not end here perfectly like the fairy tale of prince charming and snow white. I tried to write the key with classes implemented Writable, which would cause an implicit conversion on RDD to SequenceFileRDDFunctions. Unfortunately, I noticed that both key and value has to implemented Serializable, which is reasonable. If the key and value are not serializable, how do we pass them between workers?

    At last, to simplify all problems, I decided to transform my keys to a String with concatenation. Since the beginning, all of my keys are String for sure.

TL;DR - The key and value of an RDD have to implement Serializable. If you want to save the RDD, the kay and value have to implement Writable.

    Well I guess you are here for how to read/write a sequence file. If you meet the above requirement. You can just do
val path: String = "your/path/for/rdd/here";
val rdd = sc.parallelize(


List(("Raccoon", 1), ("Squirrel", 2), ("Ferret", 3))

);

rdd.saveAsSequenceFile(path);

val readRdd = sc.sequenceFile(path, classOf[Text], classOf[IntWritable]);
// here, the type you have read is Tuple2[Text, IntWritable].
// Do not forget to transform them into Tuple[String, Int].
val usableRdd = readRdd.map{ case (key, value) =>

(key.toString, value.get)

};