soulmachine

Machine has soul.

Calling C Code From Java Using JNI

| Comments

This blog is a fork of http://stuf.ro/calling-c-code-from-java-using-jni with some errors fixed and C++ support added.

In this tutorial we’ll be creating a Java application calling code from a native library. We’ll have a Java application called HelloWorld which will call the function helloFromC from a shared library named ctest, using Java Native Interface.

First off, we’ll create a file named HelloWorld.java to contain the HelloWorld class.

1
2
3
4
5
6
7
8
9
10
/* HelloWorld.java */

public class HelloWorld {
    native void helloFromC(); /* (1) */
    static public void main(String argv[]) {
        System.loadLibrary("ctest"); /* (2) */
        HelloWorld helloWorld = new HelloWorld();
        helloWorld.helloFromC(); /* (3) */
    }
}
  1. Make the JVM aware of a function defined externally, named helloFromC
  2. Load an external library called ctest (which will need to define this function)
  3. Call the function we talked about

Even though we didn’t write any library yet, we can still compile the Java application, because this is a dependency that will be resolved at runtime. So, let’s compile the application:

javac HelloWorld.java

Compile and Run Java Source Code in Memory

| Comments

In this blog I will show you how to compile and run Java source code on the fly in memory. In other words, use Java as a script language.

Here are the three source code files.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
package me.soulmachine.compiler;

import java.io.IOException;
import java.io.PrintWriter;
import java.io.Writer;
import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;

import javax.tools.*;

/**
 * Simple interface to Java compiler using JSR 199 Compiler API.
 */
public class MemoryJavaCompiler {
    private javax.tools.JavaCompiler tool;
    private StandardJavaFileManager stdManager;

    public MemoryJavaCompiler() {
        tool = ToolProvider.getSystemJavaCompiler();
        if (tool == null) {
            throw new RuntimeException("Could not get Java compiler. Please, ensure that JDK is used instead of JRE.");
        }
        stdManager = tool.getStandardFileManager(null, null, null);
    }

    /**
     * Compile a single static method.
     */
    public Method compileStaticMethod(final String methodName, final String className,
        final String source)
        throws ClassNotFoundException {
        final Map<String, byte[]> classBytes = compile(className + ".java", source);
        final MemoryClassLoader classLoader = new MemoryClassLoader(classBytes);
        final Class clazz = classLoader.loadClass(className);
        final Method[] methods = clazz.getDeclaredMethods();
        for (final Method method : methods) {
            if (method.getName().equals(methodName)) {
                if (!method.isAccessible()) method.setAccessible(true);
                return method;
            }
        }
        throw new NoSuchMethodError(methodName);
    }


    public Map<String, byte[]> compile(String fileName, String source) {
        return compile(fileName, source, new PrintWriter(System.err), null, null);
    }


    /**
     * compile given String source and return bytecodes as a Map.
     *
     * @param fileName source fileName to be used for error messages etc.
     * @param source Java source as String
     * @param err error writer where diagnostic messages are written
     * @param sourcePath location of additional .java source files
     * @param classPath location of additional .class files
     */
    private Map<String, byte[]> compile(String fileName, String source,
        Writer err, String sourcePath, String classPath) {
        // to collect errors, warnings etc.
        DiagnosticCollector<JavaFileObject> diagnostics =
            new DiagnosticCollector<JavaFileObject>();

        // create a new memory JavaFileManager
        MemoryJavaFileManager fileManager = new MemoryJavaFileManager(stdManager);

        // prepare the compilation unit
        List<JavaFileObject> compUnits = new ArrayList<JavaFileObject>(1);
        compUnits.add(fileManager.makeStringSource(fileName, source));

        return compile(compUnits, fileManager, err, sourcePath, classPath);
    }

    private Map<String, byte[]> compile(final List<JavaFileObject> compUnits,
        final MemoryJavaFileManager fileManager,
        Writer err, String sourcePath, String classPath) {
        // to collect errors, warnings etc.
        DiagnosticCollector<JavaFileObject> diagnostics =
            new DiagnosticCollector<JavaFileObject>();

        // javac options
        List<String> options = new ArrayList<String>();
        options.add("-Xlint:all");
        //       options.add("-g:none");
        options.add("-deprecation");
        if (sourcePath != null) {
            options.add("-sourcepath");
            options.add(sourcePath);
        }

        if (classPath != null) {
            options.add("-classpath");
            options.add(classPath);
        }

        // create a compilation task
        javax.tools.JavaCompiler.CompilationTask task =
            tool.getTask(err, fileManager, diagnostics,
                options, null, compUnits);

        if (task.call() == false) {
            PrintWriter perr = new PrintWriter(err);
            for (Diagnostic diagnostic : diagnostics.getDiagnostics()) {
                perr.println(diagnostic);
            }
            perr.flush();
            return null;
        }

        Map<String, byte[]> classBytes = fileManager.getClassBytes();
        try {
            fileManager.close();
        } catch (IOException exp) {
        }

        return classBytes;
    }
}

Deserialize a JSON String to a Binary Tree

| Comments

A simple solution

We know that Jackson is very convenient to deserialize a JSON string into a ArrayList, HashMap or POJO object.

But how to deserialize a JSON String to a binary tree?

The definition of binary tree node is as follows:

1
2
3
4
5
public class BinaryTreeNode<E>
{
    public E value;
    public BinaryTreeNode<E> left;
    public BinaryTreeNode<E> right;

The JSON String is as follows:

{
   "value": 2,
   "left": {
    "value": 1,
    "left": null,
    "right": null
  },
  "right": {
    "value": 10,
    "left": {
      "value": 5,
      "left": null,
      "right": null
    },
    "right": null
  }
}

The above JSON string represents a binary tree as the following:

  2
 / \
1   10
   /
  5

The solution is quite simpler, since JSON can express tree naturally, Jackson can deal with recursive tree directly. Just annotate a constructor with JsonCreator:

1
2
3
4
5
6
7
8
9
10
11
public static class BinaryTreeNode<E>
{
    public E value;
    public BinaryTreeNode left;
    public BinaryTreeNode right;

    @JsonCreator
    public BinaryTreeNode(@JsonProperty("value") final E value) {
        this.value = value;
    }
}

Let’s write a unite test to try it:

Deserialize a JSON Array to a Singly Linked List

| Comments

We know that Jackson is very convenient to deserialize a JSON string into a ArrayList, HashMap or POJO object.

But how to deserialize a JSON array, such as [1,2,3,4,5] to a singly linked list?

The definition of singly linked list is as follows:

1
2
3
4
5
public class SinglyLinkedListNode<E>
{
    public E value;
    public SinglyLinkedListNode<E> next;
}

Well, the solution is quite simpler than I expected: Just make SinglyLinkedListNode implement java.util.List or java.util.Collection , Jackson will automatically deserialize it!

This idea comes from Tatu Saloranta, the discussion post is here, special thanks to him!

Here is the complete code of SinglyLinkedListNode:

Debug Hadoop Applications With IntelliJ

| Comments

In my previous blog, I explained how to Create a Hadoop Compilation and Development Environment so that you can build the Hadoop source code you get from the Apache Hadoop git repository.

However, not everyone need to dig into the source code, normally more people will just call the Hadoop APIs to write a MapReduce program. When you write a MapReduce program, you rarely get it right for one time, then you need to debug your code.

This time I’ll show you how to debug your Hadoop applications using the IntelliJ IDE.

Environment: CentOS 6.6, Oracle JDK 1.7.0_75, Maven 3.2.5, IntelliJ Idea 14.0.3

1. Create a MapReduce Maven project

First we need to create a MapReduce program to debug. Let’s use the simplest example, WordCount, for demonstration. The source code is here WordCount – Hadoop Wiki.

1.1 Generate the Maven project

Use the mvn command to generate the scaffolding for the project.

mvn archetype:generate -DgroupId=me.soulmachine -DartifactId=wordcount -DarchetypeArtifactId=maven-archetype-quickstart -DinteractiveMode=false

Delete the src/test/java/me/soulmachine/AppTest.java file, as it will not be used in this example, and rename the src/main/java/me/soulmachine/App.java file to WordCount.java.

Create a Hadoop Compilation and Development Environment

| Comments

If you want to read the source code of Hadoop and dig into the internals of Hadoop, then you need to know how to compile the source code and use IDEs (such as Eclipse or IntelliJ Idea) to open the projects. This blog will introduce you how to create a Hadoop build and development environment.

Environment: CentOS 6.6, Oracle JDK 1.7.0_75

1. Install Oracle JDK 7

Download jdk-7u75-linux-x64.rpm
sudo yum localinstall -y ./jdk-7u75-linux-x64.rpm

For now if you use JDK 8 to compile the source code of Hadoop, it will fail because the javadoc in Java 8 is considerably more strict than the one in earlier version, see more detailes here

2. Install Maven

wget http://mirrors.gigenet.com/apache/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.tar.gz
sudo tar -zxf apache-maven-3.2.5-bin.tar.gz -C /opt
sudo vim /etc/profile
export M2_HOME=/opt/apache-maven-3.2.5
export PATH=$M2_HOME/bin:$PATH

3. Install FindBugs(Optional)

wget http://iweb.dl.sourceforge.net/project/findbugs/findbugs/3.0.0/findbugs-3.0.0.tar.gz
sudo tar -zxf findbugs-3.0.0.tar.gz -C /opt
sudo vim /etc/profile
export FB_HOME=/opt/findbugs-3.0.0
export PATH=$FB_HOME/bin:$PATH

If you want to run mvn compile findbugs:findbugs then you need to install FindBugs.

Java Code Style and Static Analysis

| Comments

1. Checkstyle

CheckStyle is a development tool to help programmers write Java code that adheres to a coding standard. It automates the process of checking Java code to spare humans of this boring (but important) task. This makes it ideal for projects that want to enforce a coding standard.

Which Java code style to choose? Google has published a few coding standards, include Google Java Style. Aditionally, there are xml configuration files for Eclipse and IntelliJ in the SVN repository, https://code.google.com/p/google-styleguide/source/browse/trunk.

Checkstyle Eclipse plugin has already used Google Java style by default, checkt it by clicking “Window->Preferences->Checkstyle->Global Check Configurations”, or at https://github.com/checkstyle/checkstyle/blob/master/google_checks.xml.

1.1 Checkstyle Eclipse Plugin

To check your coding style automatically in Eclipse, just install the Checkstyle Eclpise plugin. Launch Eclipse, click Menu “Help –> Install New Software”, input http://eclipse-cs.sf.net/update/, then click “Next” to install the plugin.

1.2 Checkstyle Maven Plugin

Add the following lines to pom.xml to enable Checkstyle:

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-checkstyle-plugin</artifactId>
            <version>2.13</version>
            <dependencies>
                <dependency>
                    <groupId>com.puppycrawl.tools</groupId>
                    <artifactId>checkstyle</artifactId>
                    <version>6.1.1</version>
                </dependency>
            </dependencies>
            <executions>
                <execution>
                    <id>checkstyle</id>
                    <phase>validate</phase>
                    <configuration>
                        <configLocation>google_checks.xml</configLocation>
                        <encoding>UTF-8</encoding>
                        <consoleOutput>true</consoleOutput>
                        <failsOnError>true</failsOnError>
                        <includeTestSourceDirectory>true</includeTestSourceDirectory>
                    </configuration>
                    <goals>
                        <goal>checkstyle</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>
<reporting>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-checkstyle-plugin</artifactId>
            <version>2.13</version>
            <configuration>
                <configLocation>https://raw.githubusercontent.com/checkstyle/checkstyle/master/google_checks.xml</configLocation>
            </configuration>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-jxr-plugin</artifactId>
            <version>2.5</version>
        </plugin>
    </plugins>
</reporting>

Restore Octopress at a New Computer

| Comments

OS: Ubuntu 12.04 64-bit

1. Install ruby

1.1 Install ruby via RVM

$ \curl -sSL https://get.rvm.io | bash -s stable --ruby

1.2 Integrating RVM with gnome-terminal

/etc/profile, ~/.bash_profile are for login shell, and ~/.bashrc is for interactive shell, and RVM’s path is added to ~/.bash_profile, so you need to set the shell as a login shell.

1.3 Give it a try

Exit current shell, and open a new shell,

ruby -v

You have successfully installed ruby.

2. Install Python

$ sudo apt-get install -y python

Because Pygments syntax highlighting needs Python.

3. Clone your blog to the new machine

First you need to clone the source branch to the local octopress folder.

$ git clone -b source git@github.com:username/username.github.com.git octopress