Saturday, March 12, 2016

Interface to Google APIs with Python

    I have demonstrated how to obtain a Google App credential of a user in the last post. And of course, you read that post because you need to create an application. This post is talking about how to use Google-API-Python-Client and its interface concept idea(of course, under my own interpretation). I will mainly use Drive API in this demonstration.



    To call Google services you need to import the following packages:
import apiclient
from oauth2client.file import Storage
import httplib2


    Before you calling a service API, you need to retrieve corresponding service with the user credential obtained by previous post. Like the following:
SERVICE_NAME = 'drive'
SERVICE_VERSION = 'v3'
storage = Storage(CREDENTIAL_NAME)
credential = storage.get()
http = credential.authorize(httplib2.Http())
service = apiclient.discovery.build(SERVICE_NAME, SERVICE_VERSION, http=http)

    The SERVICE_NAME is which service you are going to retrieve, and SERVICE_VERSION is the service, obviously. The services names and versions can be seen on APIs-Explorer. You can easily click the service you want to interface, and check the prefix before first .(dot) of each API. The same service suppose to have the same prefix, that is your service name. And the version just depends on your need, it shall be in the form v[1-9].

    Now you have retrieved the service instance, let's see which API you are going to call. At the previous section, you have decide the service name with first term of API name. You are going to check what the remaining terms are, that is the path to make the API call. For example, when you are calling drive.files.list it maps to Python code service.files().list(...); or analytics.data.ga.get corresponds to service.data().ga().get(...)(Of course, the service are retrieve by different build() call.)

    And you are giving the parameters of an API with the Pythonic named arguments. The following is a call of drive.files.list:
PAGE_SIZE = 10
QUERY = "name contains 'Hello'"
FIELDS = 'nextPageToken,files(id, name)'
service.files().list(

pageSize=PAGE_SIZE,
q=QUERY,
fields=FIELDS
)

    According the document of drive.files.list, we can simply assign the parameters with named arguments.

    However, it is not always as simple as the above. Like the API drive.files.create, you cannot specify the upload content in parameters shown in reference. In this case, I recommend you to check your API with Interactive Help in IPython(or pure Python interpreter). Like the following:
help(service.files().create)
    With help(...), you are able to check all information about this function, including arguments and returned value in a very detailed way.

    After checked the drive.files.create with help(...), now we are able to make file upload(creation). Like the following:
FILE_NAME = "test_text"
mediaBody = apiclient.http.MediaFileUpload(FILE_PATH, mimetype='text/plain')
body = {
"name": FILE_NAME
}
service.files().create(media_body=mediaBody, body=body)
# or the following
# service.files().create(media_body=mediaBody, name=FILE_NAME)




    This is it. When I was interfacing Google APIs, I have encountered a lot problems and did a lot of googling. The above all was my notes about the package. Help it is helpful for you.

Monday, February 29, 2016

Generate Google Application User Credentials with Python/Flask

    Recently, I got a demand of building Google Application for our company. After some trials, I understood how things work. And I want to document it, hope it is going to help people who met a similar need.

    First of all, you need to create a project for your application on Google Develops Console. And then you need to enable the service you are going to interface, by click the link 'Enable and manage APIs' or enter the 'API Manager.' Now you are suppose to see a few lists of Google APIs, choose the one that you need, and click the enable button.

    The following step is to generate application credentials. Go to the 'API Manager > Credentials,' and click Create Credentials > API key. You are going to select between types 'Server key', 'Browser key', 'iOS key' or 'Android key', this depends on which platform your application is(Here I choose the server key, because my application is going to call APIs on a server instead of browsers or mobiles.) All information you need to provide for this key is server name(If you can provide your server IP address is better, of course.)  Now you can see you API key in the credentials page.

    After you generated an API key, there are another credential to generate. Which is OAuth 2.0 Client ID. Before you click the 'create credentials' button, you need to switch to the tab 'OAuth consent screen' which is just above the 'create credentials' button and fill the field 'Product name shown to users'. Now you are fine to create the 'OAuth client ID' by clicking 'create credentials'. You need to choose which type of your application type(I guess this doesn't really matter, I choose the 'Other' type), and fill your application name. After you created the client ID, you are able to download your client key in the credentials page, the right side of your client ID. And your client secret are able to be seen by click your client ID name.

** Remember, anyone of API keys, client IDs and client secret are suppose to be kept secret. DO NOT share with anyone which is not in your project. **

    Now you are done with all the paperworks, let's get hands dirty. You need to get user credentials to perform actions for this user. I deploy an web application to acquire user credentials with Python/Flask. You need to build two pages, one is for requesting authentications, the other one is for retrieve authentications. The application may like this
from oauth2client.client import OAuth2WebServerFlow
from oauth2client.file import Storage

from flask import Flask, url_for, redirect, request
app = Flask(__name__)

CLIENT_ID_FILE = "your_client_id_file_name"
CLIENT_SECRET = "your_client_secret"
SCHEME = "http://"
DOMAIN = "localhost:5000"
AUTH_RETURN_PATH = "/auth_return"
REDIRECT_URI = SCHEME + DOMAIN + AUTH_RETURN_PATH
CREDENTIAL_NAME = "credential_name"
SCOPES = [
  "https://www.googleapis.com/auth/drive.file"
]

@app.route("/auth")
def auth():
  storage = Storage(CREDENTIAL_NAME)
  credentials = storage.get()
  if not credentials or credentials.invalid:
    print("!!! Cannot find this credential !!!")
    return request_credential(storage)
  else:
    return "Your have authorized your credential."

def request_credential(storage):
  flow = OAuth2WebServerFlow(client_id=CLIENT_ID_FILE,
    client_secret=CLIENT_SECRET,
    scope=SCOPES,
    redirect_uri=REDIRECT_URI)
  auth_uri = flow.step1_get_authorize_url()
  return redirect(auth_uri)

@app.route("/auth_return")
def auth_return():
  flow = OAuth2WebServerFlow(client_id=CLIENT_ID_FILE,
    client_secret=CLIENT_SECRET,
    scope=SCOPES,
    redirect_uri=REDIRECT_URI)
  credentials = flow.step2_exchange(request.args.get("code", ""))
  storage = Storage(CREDENTIAL_NAME)
  storage.put(credentials)
  credentials = storage.get()
  if not credentials or credentials.invalid:
    return "Authorization failed."
  else:
  return "Authorization succeed."


    In the above codes, the entry point is 'http://localhost:5000/auth', which would lead user to auth(). It check if the credential exists, it call the request_credential if not. The authentication is following OAuth 2.0 two-step authentication, so it will lead user to Google to authenticate his/her authentication and then redirect to the return path that you have given in redirect_uri.

    After you got the credential and call storage.put(credential), the credential will be stored at the location CREDENTIAL_NAME(Absolute or relative path depends on it.)

    There is a constant named CLIENT_ID_FILE, which is the path to your client ID in your file system, however, without the extension(.json). And obviously, CLIENT_SECRET is what have been mentioned in the above article.

    And the SCOPES in the code, can be found in the API document. In this example, I requested the https://www.googleapis.com/auth/drive.file scope, which is shown in the Drive document.



    After you read this article, you should be able to get a Google service credential from a user. If there is any part is unclear, please ask without hesitation, I am glad to answer it. I may write another post for how to operate some Google services that I have interfaced with.

Saturday, February 20, 2016

Spark: Read/Write Sequence files

    Recently, I got some extraordinary demands for Spark RDD. I need an RDD which supports multi-key value pair, followed by IO operations. As usual, I started from python and I found that applying saveAsSequenceFile does not always work. After some searches, I assume it is because of the writing type of sequence file. Which is not demanded when writing, however, is demanded when reading(sequenceFile). After all, if I did not specify the output type, how should I know what type to read?

    After that, I think I had to quit my obsession on leveraging Spark with Python. I switched the language to Scala, which is fully supported to Spark. Nonetheless, story did not end here perfectly like the fairy tale of prince charming and snow white. I tried to write the key with classes implemented Writable, which would cause an implicit conversion on RDD to SequenceFileRDDFunctions. Unfortunately, I noticed that both key and value has to implemented Serializable, which is reasonable. If the key and value are not serializable, how do we pass them between workers?

    At last, to simplify all problems, I decided to transform my keys to a String with concatenation. Since the beginning, all of my keys are String for sure.

TL;DR - The key and value of an RDD have to implement Serializable. If you want to save the RDD, the kay and value have to implement Writable.

    Well I guess you are here for how to read/write a sequence file. If you meet the above requirement. You can just do
val path: String = "your/path/for/rdd/here";
val rdd = sc.parallelize(

List(("Raccoon", 1), ("Squirrel", 2), ("Ferret", 3))
);
rdd.saveAsSequenceFile(path);

val readRdd = sc.sequenceFile(path, classOf[Text], classOf[IntWritable]);
// here, the type you have read is Tuple2[Text, IntWritable].
// Do not forget to transform them into Tuple[String, Int].
val usableRdd = readRdd.map{ case (key, value) =>
(key.toString, value.get)
};

Monday, January 18, 2016

Setup WebHDFS on Existed Hadoop and Operate with Python

This article is based on how to enable the WebHDFS on Hadoop and then read/write with Python. If you haven't setup a Hadoop environment, I recommend you follow this tutorial.

Recently, I need to setup and test Hadoop on work. And I find the instruction on Hadoop is not detailed enough. However, after some googling, I finally done the setup and test. Here are some experiences, hope it would be helpful. :)



1. Enable the WebHDFS in the configuration file,
"HADOOP_HOME/etc/hadoop/hdfs-site.xml"
By inserting:
<property>
   <name>dfs.webhdfs.enabled</name>
   <value>True</value>
</property>
And restart the Hadoop server. After that, the WebHDFS shall be ready for Hadoop web API. In this article, I will not cover the authenticated WebHDFS operation.

2. Test the service with curl.
i. Create
For create a file via WebHDFS, you can execute the command
curl -i -X PUT "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=CREATE"
And you will receive a HTTP 307 which bring you a location to where to create file.
Then execute the following
curl -i -X PUT -T <LOCAL_FILE> <LOCATION_FROM_PREVIOUS>
The above one only allow you to upload a file. If you want to send string to the file, just replace the "-T <LOCAL_FILE>" to "-d <DATA>"
ii. Read
then you can read file with
curl -i -L "http://<HOST>:<PORT>/webhdfs/v1/<PATH>?op=OPEN"
Reading API also apply the two-step operation strategy. In this part, you just follow the returned location then you can get the data/file you are indexing.
Now, if you have successfully setup the WebHDFS, you will see that it create and return your file correctly. If you want to learn more about operations to WebHDFS with curl. Please refer the official instruction.

3. Operations with Python/hdfs [doc]
i. Install this package
pip install hdfs
ii. Write a configuration file
[global]
default.alias = dev

[dev.alias]
url = http://hdfs.hashfarm.cc:50070
user = sunshire
This config file is telling mtth/hdfs to connect to http://hdfs.hashfarm.cc:50070 as sunshire when the alias is dev, also set 'dev' as the default alias.
And the file shall be saved at ~/.hdfscli.cfg by default. Or you can set the environment variable HDFSCLI_CONFIG to specify your very own config file location.
iii. Write yourself a script
from hdfs import Config
import sys
fileName = "/helloWord"
message = "Hello :)"

client = Config().get_client()

with client.write(fileName, overwrite=True) as writer:
  writer.write(message)

ls = client.list("/")
if fileName not in ls:
  print("file not found")
  sys.exit()

readMessage = ""
with client.read(fileName) as reader:
  readMessage = reader.read()

print("wrote: " + message + ", read: " + readMessage)
In the above code, we generate a client via config. Here it apply the default alias, which is 'dev'. To claim a explicit alias just pass the alias to the first parameter to get_client, like the following:
client = Config().get_client("dev")
With mtth/hdfs, you can read/write hdfs like ordinary python file operations, just using the with syntax.
By using client.list, you can get the sub-directory list of a directory.
Notice, while indexing a file or a directory. Remember to add "/" at the beginning. Otherwise the package will not find a corresponded path, and do not throw an error, either.

Saturday, December 5, 2015

Let's Talk about I Origins and Central Limited Theorem

[Spoiler Alert] This article spoil the I Origins straight down to the end. If you want to watch this film but haven't, I would recommend you to finish the film first. 

----


  In the I Origins, there is a scene that Grey visited India and met Priya. He made Priya do a sequence of tests, which is picking a prefer one from three pictures for 25 sets. As a result, Priya made a 44% accuracy compare to the result that Grey thought Sofi would choose. Obviously, this result is above the average. But could we say that it is convincible? And how convinsing it is? Here comes the Central Limited Theorem to rescue.


  For those who has no prior knowledge about Central Limited Theorem. The theorem is a easier way to estimate how likely for a lot of I.I.D. events occurred less than a given times. (For those who don't really know statistics, you can treat I.I.D. as a lot of repeated experiments which, of course, have the same distribution.)


  In mathematical representation, it is $$P(S_n \leq k) \approx \phi(z); z= \frac{k - n}{\sigma \sqrt{n}}$$

which means the probability of the sum of $n$ times I.I.D. outcomes are lesser than or equal to $k$ is close to $\phi(z)$, where $\phi(.)$ is the CDF(cumulative distribution function) of normal distribution, $\sigma$ and $\mu$ are the standard deviation and mean of the distribution of the given event respectively.

  And now, we are ready for some applications. In the film, Priya have done 25 tests and got a 44% accuracy, which simply indicates Priya got 11 true answers. We assume that Priya had guessed randomly under uniform distributed. Therefore Priya had a 33.3% probability of true, and 66.7% for false. This is a Bernoulli distribution, so we got a $\mu=\frac{1}{3}$ and $\sigma^2=\frac{2}{9}$. Priya did 11, so let us estimate how likely a person can achieve 11 or more in totally random selection.

  That is $P(S_{25} \geq 11) = 1 - P(S_{25} \leq 10)$, and we knew the following by Central Limited Theorem.
$$P(S_{25} \leq 10) \approx \phi(\frac{10-25*\frac{1}{3}}{\sqrt{25*\frac{2}{9}}})=\phi(\sqrt{2}) \approx \phi(0.707) \approx 0.7794$$
To interpret, $P(S_{25} \leq 10)$ is the probability of a guy answered 10 or less correctly in 25 tests, and as we can see, about 78% people are able to do so. On the other hand, $P(S_{25} \geq 11)$ is the probability of a person reach 11 or more in the 25 tests, 22% people can do so. That is about 1 out of 4~5 people would reach such an accuracy.

  I guess that is not promising enough to show that Priya inherited Sofi's soul, is it? Not to mention that some of those pictures are beauty and naturally attractive, which increases the probability of Priya made a choice that same as Sofi does.

----

  I Origins elaborated that a scientist(represented as ration) still a human, who is emotional more or less. Grey knew that Sofi may not be a good mate, but he enjoys her love. That is the difference between lover and mate. Grey's passion about iris and the misrecognized result from the recognition system are the last straws, which push him to find out if extremely similar irises connect to the same door of a soul. And the irrational of Grey made a fortune for both him and Priya. Although this is not part of the story, I hope Grey adopt Priya after that and bring her a better life.

  It is not an amusement film, but a film can make us think more after watched it.

Sunday, November 2, 2014

Upgrades AppServ to Apache2.4 and php 5.6.2

   Since most php learners start under AppServ (well, at least in my social circle), they might build their projects under Appserv until someday they found AppServ are way too old version on php and mysql. Recently, as I tried to import some sdks, I found really frequently that my php version is old, and I can't import sdks under such an old version of php.

    As I upgraded my AppServ, I found it's kinda tricky and I want to write it as a guide. Hope other people who find this article can upgrade their php version painlessly.

    Suppose that the original AppServ located at D:\AppServ\ ; Apache is at D:\AppServ\Apache\

    1. Pre-works, Export all your databases and user information from phpmyadmin.
        Stop your Apache server and MySQL Service. You can find it in the startup>AppServ>Control Server By Sevice, just execute stop script as the administrator.


    2. Download the Apache 2.4, php 5.6.2 "thread safe", MySQL 5.6.21 and PhpMyAdmin 4.2. Notice that you need to download the corresponding version of your system structure. If your server is 32-bit then download x86 version, else if your server is 64-bit then download x64 version. In this article, I do the x64 version.
        PHP: http://windows.php.net/download/
        Apache: http://www.apachelounge.com/download/
        MySQL: https://dev.mysql.com/downloads/installer/
        PhpMyAdmin: http://www.phpmyadmin.net/home_page/downloads.php


    3. Change the name of the original Apache folder, so you can recover your server when the installation failed. And then extract the folder "Apache24" from "httpd-2.4.10-win64-VC11.zip" to the directory where the original Apache located. In this case, extracted Apache24 to D:\AppServ\.

         Find startup scripts at startup>AppServ>Control Server By Sevice, there are three scripts for original Apache, named "Apache Restart", "Apache Start" and "Apache Stop". Change the target to the new Apache directory and the arguments from Apache2.2 to Apache2.4 in the properties option.

        Now execute the script "Apache Start" as the administrator. After the console disappear, try to connect to http://127.0.0.1/; the browser shows "It works!", if the Apache works fine. Then you can stop your Apache server temporary.


    4. Change the name of the original php folder, so you can recover your server when the installation failed. And extract all stuff from "php-5.6.2-Win32-VC11-x64.zip" to "D:\AppServ\php5".

        Then you can find "php.ini-development" in the directory, D:\AppServ\php5\, which you just made. Change the file name "php.ini-development" to "php.ini". And edit that file, find the line "extension_dir", make it as extension_dir = "D:/AppServ\php5\ext". Remember to remove the semi-colon in the beginning of the line.


    5. Edit the file, D:\AppServ\Apache24\conf\httpd.conf. Search "LoadModule", append following lines:
    LoadFile D:/AppServ\php5\php5ts.dll
    LoadModule php5_module D:/AppServ\php5\php5apache2_4.dll
    PhpIniDir D:/AppServ\php5\php.ini
    addtype application/x-httpd-php .php .html


       Search DocumentRoot, change DocumentRoot "D:\AppServ\Apache24\htdocs" to DocumentRoot "D:\AppServ\www".

        Search DocumentIndex, change DocumentIndex index.html to DocumentIndex index.php index.html.

        And start your Apache again, browse http://127.0.0.1/phpinfo.php; the new version phpinfo shows if your php works fine.


    6. Change the original MySQL folder name, so you can recover your server when the installation failed.

        Execute "mysql-installer-community-5.5.40.1.msi" which you have downloaded. After Installed it, you will have MySQL Installer Community in your server which can be found in startup list.

        Execute MySQL Installer Community. Choose add, and install MySQL Server only. After the Next, customize your installation, select the path to the original MySQL directory. In my case, it is D:\AppServ\MySQL.


    7. After MySQL installation finished, change the original phpmyadmin folder name, so you can recover your server when the installation failed.

        And extract the folder "phpMyAdmin-4.2.10.1-all-languages" from "phpMyAdmin-4.2.10.1-all-languages.zip" to the directory where the original phpmyadmin located. In my case, it is D:\AppServ\www\.

        The extracted folder named "phpMyAdmin-4.2.10.1-all-languages", so don't forget to change it as "phpMyAdmin".


    8. Edit php.ini, search "extension=". As you can see, there are a lot of php extensions. You can give them a glance, de-comment those extensions you need indeed. If you are not sure which of them needs to be de-comment, just refer to the php.ini in C:\Windows\php.ini.

        In here, you will need to de-comment the lines
extension=php_mbstring.dll
extension=php_exif.dll      ; Must be after mbstring as it depends on it
extension=php_mysql.dll
extension=php_mysqli.dll

        So your php and mysql can cooperate well.


    9. After step 6, step 7 and step 8, execute the script "Apache Restart". And browse http://127.0.0.1/phpMyAdmin/, the new phpmyadmin interface shows if your mysql and phpmyadmin works fine.

        If your phpmyadmin works fine, the last step you need to do is import the databases and user information with phpmyadmin.

    After all these efforts, your AppServ is successfully upgraded. You are no more bounded with the old version php!


Cheers!


*Note: You got to upgrade php accompany with Apache, since the php later than 5.4 needs Apache 2.4. And the newer php version doesn't compatible with the old phpmyadmin, so the phpmyadmin upgrade are need indeed. Nevertheless, newer phpmyadmin needs newer MySQL, that's why we got to upgrade all components of the server.

[Reference]
http://www.cc.net.tw/2014/02/2622/
http://www.apachelounge.com/viewtopic.php?t=724

Monday, September 22, 2014

project "Shoot" -- library to visualize your tree structure

    Hi there, it has been a long while since last blog I post. Well, forgive me, it was heat break. But I really did something, I will show them. Just be patient :)

----


    Shoot was a small project I wrote to help me visualize some assignments from my data structure class. It have been stay in my disk for a while. Recently, I think maybe it's a good idea to publish it so other students can easily visualize their works. Cuz I really did a search to find if there a easy way to draw my tree structure when I was in course. Unfortunately, I didn't find one, so I made one.



    Originally this project was developed in pure C language, not C++. But I think object-oriented is much better for this project. Because I can define some virtual function and call them directly.

    This project doesn't really takes much my time, most CSS/Javascript are done when I was in course. Interestingly, my internet breaks during developing. And I can't find any reference for some bugs, Fortunately, At the day after, the internet was fixed. So I can finish it.

----

    Here is the link to "Shoot".

    To use this project, you have to inherit your own node class from shoot::TreeNode. And then you need to implements your own queue/heap/tree structure, and some virtual methods in shoot::TreeNode. Including sibling(), child(), hasSibling(), hasChild(), getData().
    Here is some detail I wrote on github before:
            YourTreeNode *sibling() : no argument, return the pointer of this node's next sibling. If there is no next sibling you can return NULL instead.
            YourTreeNode *child() : no argument, return the pointer of this node's first child. If it don't have any child you can return NULL instead.
            bool hasSibling() : no argument, return a boolean that if this node has next sibling. If it is the last sibling in this generation, return false. Otherwise, return true.
            bool hasChild() : no argument, return a boolean that if this node has any child. If it is the leaf node, return false. Else cases, return true.

    Once you finish these methods, the visualize feature is done. All you have to do is call YourTreeNode->generate(filename) .
    But you gonna notice that "shoot.hd" shall be in the same directory as your .exe is. And it needs the internet to access jquery, if you want to make it works offline,  just download jquery from jquery.com/download to the same directory as "shoot.hd" is. And modify the link source at the first line in "shoot.hd".

----

    If you don't know how to import and compile this project correctly, here are the instructions.
        1. Don't forget to include "shoot.h" in your cpp.
        2. After you finished your code, compile the program with "shoot.o"
            ( g++ -o example.out example.cpp shoot.o)
        3. Check if "shoot.hd" is in the same directory as exe is.

----

    Sincerely hope this project and guidance really help you. :)