Download Sentry Events via RESTful API

For those users who use Sentry as a error recording service. When you got an error issue, and wanting to know all events detail. But on Sentry web interface, you can read an event each time. And, of course, you wouldn't like the idea of copying each event til it ends. Fortunately, Sentry provided a web API for users who analysis data on their Sentry server. You may refer to Sentry API Docs for more detail.

After some time to explore, I understand that you need to create an access token on your sentry server http://your-host/api/. And after you are done, you are ready to access the APIs listed on the above document. And here is a simple example I used to retrieve all events of an issue and make some analysis.

import json, re, requests, sys

def main(host, issueId, token):
  endpoint = "http://%s/api/0/issues/%s/events/" % (host, issueId)
  headers = {"Authorization": "Bearer " + token}
  with open("data/%s.json" % issueId, "w") as f:
    f.write("[")
    nextLink = endpoint
    while (nextLink != None):
      if nextLink != endpoint :
        f.write(",")
      response = requests.get(nextLink, headers=headers)
      text = response.text[1:-1]
      f.write(text)
      headerLink = response.headers.get("link", "")
      regSearch = re.search("<([^<]*)>; rel=\"next\"; results=\"(true|false)\";", headerLink)
      if len(regSearch.groups()) >= 2 and regSearch.group(2) == "true":
        nextLink = regSearch.group(1)
      else:
        nextLink = None
    f.write("]")

In the above code, I uses Python requests to send requests to the sentry server. And I apply the regular expression to extract the next link of this page. I guess this shall be a simple one, and now you are ready to get your hands on the massive error logs.

Retrieve Access(Bearer) Token via oauth2client

I am doing a project interfacing with Google Cloud Platform and service account. But this time, I am not going to access the service with personal account, I do not want user experience the authentication window, and the personal profile doesn't matter. Then I found there was a kind of credential called service account, that I can make all users access the service with this account.

Besides that, I don't want create a credential file on users device. So I need a lite-weight authentication. And fortunately, there is a kind of stuff called access token, which generated by the service account key file(which suppose to be a secret). With the access token, you can access the service via a simple request associated with a bearer token.

After all these survey, I am going to find a way to generate the access token automatically. Initially, I knew that gcloud could generate token via gcloud auth activate-service-account then gcloud auth print-access-token, but I don't want to make the application make system call if not necessary. And it took me quite some time and effort to learn how to do that. I find myself got poor knowledge about OAuth2, so I downloaded the gcloud source code via from google-cloud-sdk. (I just read the gcloud installation bash and found this link.)

And I found the concept of OAuth2 service account authentication pretty intuitive. All you need to do is read the key file which you obtained from Google Developer Console, and refresh the service account credential. The following is an illustration:

from oauth2client.service_account import ServiceAccountCredentials
import httplib2

fileName = "/path/to/your/service-account-secret.json"
creds = ServiceAccountCredentials.from_json_keyfile_name(
  fileName,
  scopes=['https://www.googleapis.com/auth/cloud-platform'])
creds.user_agent = creds._user_agent = "google-cloud-sdk"
# User agent may not be necessary.
# This suffered from oauth2client bug, so we have to assign it manually.
# See https://github.com/google/oauth2client/issues/445
creds.refresh(httplib.Http())
print(creds.access_token)

Since the snippet is from gcloud, it inherits some knowledge from Google. The Credential object suffering a bug about _user_agent. so we need to assign the user_agent manually.

And with the access token generating script, I could deploy a micro-service for authenticating usage. If users needed an access token, I just refresh the credential and respond the token to users. And don't forget, the token is valid for one hour, so you may apply some micro-cache mechanism(like nginx provided) to your application and reduce the server loading.

Enable nmred/kafka-php and Zookeeper on php and CentOS

  I was working on a project that needs to send messages to Kafka. After some survey, I chose the library nmred/kafka-php. Since the setup might be difficult to people who are new to zookeeper and Kakfa, I would like to leave some record.

  The first step, import nmred/kafka-php with composer.json.

  "require": {
    ...,
    "nmred/kafka-php": "0.1.*"
  },

  And you need to install the zookeeper environment on your machine. By executing the following

yum --nogpgcheck localinstall -y https://archive.cloudera.com/cdh5/one-click-install/redhat/7/x86_64/cloudera-cdh-5-0.x86_64.rpm
yum install -y zookeeper

This allows you to access cdh5 rpm and install zookeeper client on the machine.

But your php still not have the zookeeper.so to make php communicate with zookeeper. So now we need to generate the zookeeper.so. By the following

yum install -y php-devel # this enables phpize
cd /tmp
wget http://apache.stu.edu.tw/zookeeper/zookeeper-3.4.9/zookeeper-3.4.9.tar.gz # the zookeeper source
tar zxvf zookeeper-3.4.9.tar.gz
cd /tmp/zookeeper-3.4.9/src/c # source of c lang
./configure --prefix=/usr/local/zookeeperlib # configure the environment of libzookeeper
make && make install
cd /tmp
wget https://pecl.php.net/get/zookeeper-0.3.1.tgz # download the php-zookeeper source
tar zxvf zookeeper-0.3.1.tgz
cd zookeeper-0.3.1
phpize
./configure --with-php-config=/usr/bin/php-config --with-libzookeeper-dir=/usr/local/zookeeperlib/ # setup the environment
make && make install # install zookeeper.so
echo '; zookeeper' >> /etc/php.ini
echo 'extension=zookeeper.so' >> /etc/php.ini # attach the zookeeper.so information
cd /tmp
rm -rf zookeeper-3.4.9
rm -rf zookeeper-0.3.1

Notice that, choose the php-zookeeper wisely. Make sure the version of library you use matches your php version. For example, I use 0.3.1 because the project is a php70 project.