手机需要root过，不然没有权限拷贝apk到 /system/app 目录下面。

【步骤】

通过手机的【设置】--【关于手机】 查看当前系统的Android版本，例如我的手机是4.2.2。
在 http://wiki.rootzwiki.com/Google_Apps 中下载对应的服务包。

里面包含了很多的服务(system\app)app，我们只需要安装服务框架，登录服务，日历同步，通讯录同步，邮件同步即可。

拷贝到 \system\app 系统目录中。

【账户同步】

手机安装账户同步服务，首发于润物无声。

版本：Kubuntu 12.04
下载地址：http://www.kubuntu.org/getkubuntu/download-lts

【更新源(主要是下载速度的因素)】
deb http://mirrors.163.com/ubuntu/ precise main restricted universe multiverse
deb http://mirrors.163.com/ubuntu/ precise-security main restricted universe multiverse
deb http://mirrors.163.com/ubuntu/ precise-updates main restricted universe multiverse
deb http://mirrors.163.com/ubuntu/ precise-proposed main restricted universe multiverse
deb http://mirrors.163.com/ubuntu/ precise-backports main restricted universe multiverse
deb-src http://mirrors.163.com/ubuntu/ precise main restricted universe multiverse
deb-src http://mirrors.163.com/ubuntu/ precise-security main restricted universe multiverse
deb-src http://mirrors.163.com/ubuntu/ precise-updates main restricted universe multiverse
deb-src http://mirrors.163.com/ubuntu/ precise-proposed main restricted universe multiverse
deb-src http://mirrors.163.com/ubuntu/ precise-backports main restricted universe multiverse

# Remastersys Precise
deb http://www.remastersys.com/ubuntu precise main

【添加公钥】
wget -O - http://www.remastersys.com/ubuntu/remastersys.gpg.key | sudo apt-key add -

【安装 remastersys】
sudo apt-get update; sudo apt-get install remastersys

【测试文件 (careytest/test.test)】

• /home 目录，存放用户相关配置信息
• /etc 目录，存放全局配置信息
• /usr/bin/ 目录，存放安装文件
• /usr/lib/ 目录，存放库文件
• /var/spool/ 目录，存放运行相关文件

【采用 backup 方式备份全盘资料】
sudo remastersys backup

【ISO 镜像文件测试】

• live 模式测试，文件全部存在，内容一致
• install 安装测试，文件全部都在，内容一致，

注：在安装过程中自己创建的用户被丢弃

【ISC* 镜像文件测试】

• 测试通过

【总结】

• 如果是GNOME的主题，采用原生Ubuntu系统
• 如果是KDE的主题，采用原生KUbuntu系统

【参考资料】

• Creating Custom Kubuntu Using Remastersys:http://www.youtube.com/watch?v=wAvZr5z6Y-Q
• http://www.psychocats.net/ubuntu/remastersys
• 《 Remaster your system with Remastersys – tobrunet.ch Techblog》
• 《 Free Imaging software - CloneZilla & PartImage - Tutorial》

Ubuntu 制作一键安装盘（四），首发于润物无声。

注意：经过测试 remastersys 不能很好的支持GNOME发行版安装的KDE的桌面环境。

【安装KDE桌面环境】

1. sudo apt-get install kubuntu-desktop kde-standard
2. 简体中文支持：sudo apt-get install language-pack-kde-zh-hans language-pack-kde-zh-hant

【步骤】

• 制作发布安装盘 sudo remastersys dist
• live 模式测试, 版本 Ubuntu 12.04 KDE 32位

结果：进入KDE登录界面，因为custom账户没有密码，无法登录进去系统。

• live 模式测试, 版本 Ubuntu 12.04 KDE 64位

结果：可以进入GNOME主题系统，但是因为缺少/var/spool/isc* 工作目录，后台测试程序不完整。

考虑解决方案(isc* 为测试程序，工作目录在/var/spool/isc*)：
1. 修改 /usr/bin/remastersys 脚本文件，加入自己的处理，例如拷贝必要的文件

log_msg "Copying iscxxx files"
service iscxxx stop &> /dev/null
cp -rf /var/spool/isc* $WORKDIR/dummysys/var/spool/ &> /dev/null


2. sudo remastersys dist

结果2：可以进入GNOME主题系统，前后台程序运行正常。

• install 模式测试，无法进行系统的安装，没有安装过程，直接进入live模式

Ubuntu 制作一键安装盘（三），首发于润物无声。

【步骤】

• 修改源 /etc/apt/sources.list ，加入

# Remastersys Precise
deb http://www.remastersys.com/ubuntu precise main

• 安装RemasterSys，sudo apt-get update; sudo apt-get install remastersys，

如果出现GPG错误，请参照文章最后[错误解决]。
错误解决的另一种方法, Add repository key:
wget -O - http://www.remastersys.com/ubuntu/remastersys.gpg.key | sudo apt-key add -

• 进行备份
• • sudo remastersys dist cdfs

• • sudo remastersys dist iso custom.iso，生成 iso文件，大约 1.2GB

• • 拷贝出来 sudo cp /home/remastersys/remastersys/custom.iso /media/sf_vmshare/
• 新建虚拟机进行测试，运行 live 模式，guest账户登录，测试程序可以正常运行，测试通过

• 系统安装测试，测试通过

• 可选："sudo remastersys dist" 方式 live 光盘测试,测试通过

注：
更简便的方法可以采用 sudo remastersys backup，它可以同时备份用户账号信息。
采用 sudo remastersys dist，上面两步的合集。

【使用帮助】

【参考资料】

• 官网：http://www.remastersys.com/
• 百科Remastersys: http://baike.baidu.com/view/3051607.htm
• http://wiki.ubuntu.org.cn/Remastersys

【错误解决】

在安装更新时，运行命令行sudo apt-get update 或者运行更新管理器的时候，出现:

W: GPG 错误：http://ppa.launchpad.net precise Release: 由于没有公钥，无法验证下列签名： NO_PUBKEY EAA903E3A2F4C039错误，

解决方法：

在终端中运行：sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EAA903E3A2F4C039 即可。

示例：

Ubuntu 制作一键安装盘（二），首发于润物无声。

注意：利用如下方法可以正常做出安装盘，但是安装过程中会出现错误，另外运行live cd的时候也会出现如下low graphic的问题，暂时无解，第一次尝试失败。

【步骤】

• 下载软件 relinux，https://launchpad.net/relinux，例如 relinux_0.4a1-5_i386.deb
• 安装 sudo dpkg -i relinux_0.4a1-5_i386.deb
• 依赖错误修复 sudo apt-get install -f
• 语言环境（简体中文）设定 echo $LANG; export LANG='zh_CN.UTF-8'; echo $LANG
• 制作镜像 sudo relinux, 暂时保留默认设定

• 查看 iso 文件，大小 2.0GB

zhours@zhours-VirtualBox:~/relinux$ ll /home/relinux/
总用量 1923372
drwxr-xr-x  4 root root       4096  5月 12 15:09 ./
drwxr-xr-x  4 root root       4096  5月 12 14:31 ../
-rw-r--r--  1 root root 1969508352  5月 12 15:09 custom.iso
-rw-r--r--  1 root root         47  5月 12 15:11 custom.iso.md5
drwxr-xr-x  6 root root       4096  5月 12 15:05 .ISO_STRUCTURE/
drwxr-xr-x 12 root root       4096  5月 12 14:31 .TMPSYS/
zhours@zhours-VirtualBox:~/relinux$
【参考资料】

• 官方：https://launchpad.net/relinux
• 视频：http://www.youtube.com/watch?v=3q6mibDs4Fo
• 制作发布包《 Creating Your Own Distributable Ubuntu DVD (Relinux) | HowtoForge - Linux Howtos and Tutorials》

Ubuntu 制作一键安装盘（一），首发于润物无声。

1. C++ 调用 C 语言

头文件包含：

extern "C"
{
#include "iec61850_server.h"
#include "iso_server.h"
#include "acse.h"
#include "thread.h"
#include "model.h"
}

函数调用：

IedServer_observeDataAttribute(iedServer, (DataAttribute*)mmsi.dmm, observerCallback);

回调函数observerCallback的声明和实现如下：

extern "C"
{
  void observerCallback(DataAttribute* dataAttribute)
  {
    call_Lib_daObserverCallback(mod, dataAttribute);
  }
}

2. C 调用 C++ 语言

在C语言的函数中是无法直接调用C++代码的，如果要调用，需要做一个wrapper，例如call_Lib_daObserverCallback，它的声明和实现如下：

extern "C" void call_Lib_daObserverCallback(Lib* p, DataAttribute* dataAttribute) // wrapper function
{
   p->daObserverCallback(dataAttribute);
}

daObserverCallback 才是我们C++代码的实现：

void Lib::daObserverCallback(DataAttribute* dataAttribute)
{
  map<string, MMSINFO>::iterator it;
  // ...
}

备忘：C与C++之间的互相调用，首发于润物无声。

While it’s not always the case that every Python program you write will require a rigorous performance analysis, it is reassuring to know that there are a wide variety of tools in Python’s ecosystem that one can turn to when the time arises.

Analyzing a program’s performance boils down to answering 4 basic questions:

1. How fast is it running?
2. Where are the speed bottlenecks?
3. How much memory is it using?
4. Where is memory leaking?

Below, we’ll dive into the details of answering these questions using some awesome tools.

Coarse grain timing with time（代码执行时间）

Let’s begin by using a quick and dirty method of timing our code: the good old unix utilitytime.

$ time python yourprogram.py

real    0m1.028s
user    0m0.001s
sys     0m0.003s

The meaning between the three output measurements are detailed in this stackoverflow article, but in short

• real - refers to the actual elasped time
• user - refers to the amount of cpu time spent outside of kernel
• sys - refers to the amount of cpu time spent inside kernel specific functions

You can get a sense of how many cpu cycles your program used up regardless of other programs running on the system by adding together the sys and user times.

If the sum of sys and user times is much less than real time, then you can guess that most your program’s performance issues are most likely related to IO waits.

Fine grain timing with a timing context manager

Our next technique involves direct instrumentation of the code to get access to finer grain timing information. Here’s a small snippet I’ve found invaluable for making ad-hoc timing measurements:

timer.py

import time

class Timer(object):
    def __init__(self, verbose=False):
        self.verbose = verbose

    def __enter__(self):
        self.start = time.time()
        return self

    def __exit__(self, *args):
        self.end = time.time()
        self.secs = self.end - self.start
        self.msecs = self.secs * 1000  # millisecs
        if self.verbose:
            print 'elapsed time: %f ms' % self.msecs

In order to use it, wrap blocks of code that you want to time with Python’s with keyword and this Timer context manager. It will take care of starting the timer when your code block begins execution and stopping the timer when your code block ends.

Here’s an example use of the snippet:

from timer import Timer
from redis import Redis
rdb = Redis()

with Timer() as t:
    rdb.lpush("foo", "bar")
print "=> elasped lpush: %s s" % t.secs

with Timer as t:
    rdb.lpop("foo")
print "=> elasped lpop: %s s" % t.secs

I’ll often log the outputs of these timers to a file in order to see how my program’s performance evolves over time.

Line-by-line timing and execution frequency with a profiler（代码行运行时间分析）

Robert Kern has a nice project called line_profiler which I often use to see how fast and how often each line of code is running in my scripts.

To use it, you’ll need to install the python package via pip:

$ pip install line_profiler

Once installed you’ll have access to a new module called “line_profiler” as well as an executable script “kernprof.py”.

To use this tool, first modify your source code by decorating the function you want to measure with the @profile decorator. Don’t worry, you don’t have to import anyting in order to use this decorator. The kernprof.py script automatically injects it into your script’s runtime during execution.

primes.py

@profile
def primes(n): 
    if n==2:
        return [2]
    elif n<2:
        return []
    s=range(3,n+1,2)
    mroot = n ** 0.5
    half=(n+1)/2-1
    i=0
    m=3
    while m <= mroot:
        if s[i]:
            j=(m*m-3)/2
            s[j]=0
            while j<half:
                s[j]=0
                j+=m
        i=i+1
        m=2*i+3
    return [2]+[x for x in s if x]
primes(100)

Once you’ve gotten your code setup with the @profile decorator, use kernprof.py to run your script.

$ kernprof.py -l -v fib.py

The -l option tells kernprof to inject the @profile decorator into your script’s builtins, and -v tells kernprof to display timing information once you’re script finishes. Here’s one the output should look like for the above script:

Wrote profile results to primes.py.lprof
Timer unit: 1e-06 s

File: primes.py
Function: primes at line 2
Total time: 0.00019 s

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     2                                           @profile
     3                                           def primes(n): 
     4         1            2      2.0      1.1      if n==2:
     5                                                   return [2]
     6         1            1      1.0      0.5      elif n<2:
     7                                                   return []
     8         1            4      4.0      2.1      s=range(3,n+1,2)
     9         1           10     10.0      5.3      mroot = n ** 0.5
    10         1            2      2.0      1.1      half=(n+1)/2-1
    11         1            1      1.0      0.5      i=0
    12         1            1      1.0      0.5      m=3
    13         5            7      1.4      3.7      while m <= mroot:
    14         4            4      1.0      2.1          if s[i]:
    15         3            4      1.3      2.1              j=(m*m-3)/2
    16         3            4      1.3      2.1              s[j]=0
    17        31           31      1.0     16.3              while j<half:
    18        28           28      1.0     14.7                  s[j]=0
    19        28           29      1.0     15.3                  j+=m
    20         4            4      1.0      2.1          i=i+1
    21         4            4      1.0      2.1          m=2*i+3
    22        50           54      1.1     28.4      return [2]+[x for x in s if x]

Look for lines with a high amount of hits or a high time interval. These are the areas where optimizations can yield the greatest improvements.

How much memory does it use?（内存占用）

Now that we have a good grasp on timing our code, let’s move on to figuring out how much memory our programs are using. Fortunately for us, Fabian Pedregosa has implemented a nicememory profiler modeled after Robert Kern’s line_profiler.

First install it via pip:

$ pip install -U memory_profiler
$ pip install psutil

(Installing the psutil package here is recommended because it greatly improves the performance of the memory_profiler).

Like line_profiler, memory_profiler requires that you decorate your function of interest with an@profile decorator like so:

@profile
def primes(n): 
    ...
    ...

To see how much memory your function uses run the following:

$ python -m memory_profiler primes.py

You should see output that looks like this once your program exits:

Filename: primes.py

Line #    Mem usage  Increment   Line Contents
==============================================
     2                           @profile
     3    7.9219 MB  0.0000 MB   def primes(n): 
     4    7.9219 MB  0.0000 MB       if n==2:
     5                                   return [2]
     6    7.9219 MB  0.0000 MB       elif n<2:
     7                                   return []
     8    7.9219 MB  0.0000 MB       s=range(3,n+1,2)
     9    7.9258 MB  0.0039 MB       mroot = n ** 0.5
    10    7.9258 MB  0.0000 MB       half=(n+1)/2-1
    11    7.9258 MB  0.0000 MB       i=0
    12    7.9258 MB  0.0000 MB       m=3
    13    7.9297 MB  0.0039 MB       while m <= mroot:
    14    7.9297 MB  0.0000 MB           if s[i]:
    15    7.9297 MB  0.0000 MB               j=(m*m-3)/2
    16    7.9258 MB -0.0039 MB               s[j]=0
    17    7.9297 MB  0.0039 MB               while j<half:
    18    7.9297 MB  0.0000 MB                   s[j]=0
    19    7.9297 MB  0.0000 MB                   j+=m
    20    7.9297 MB  0.0000 MB           i=i+1
    21    7.9297 MB  0.0000 MB           m=2*i+3
    22    7.9297 MB  0.0000 MB       return [2]+[x for x in s if x]

Where’s the memory leak?（内存泄露）

The cPython interpreter uses reference counting as it’s main method of keeping track of memory. This means that every object contains a counter, which is incremented when a reference to the object is stored somewhere, and decremented when a reference to it is deleted. When the counter reaches zero, the cPython interpreter knows that the object is no longer in use so it deletes the object and deallocates the occupied memory.

A memory leak can often occur in your program if references to objects are held even though the object is no longer in use.

The quickest way to find these “memory leaks” is to use an awesome tool called objgraphwritten by Marius Gedminas. This tool allows you to see the number of objects in memory and also locate all the different places in your code that hold references to these objects.

To get started, first install objgraph:

pip install objgraph

Once you have this tool installed, insert into your code a statement to invoke the debugger:

import pdb; pdb.set_trace()

Which objects are the most common?

At run time, you can inspect the top 20 most prevalent objects in your program by running:

(pdb) import objgraph
(pdb) objgraph.show_most_common_types()

MyBigFatObject             20000
tuple                      16938
function                   4310
dict                       2790
wrapper_descriptor         1181
builtin_function_or_method 934
weakref                    764
list                       634
method_descriptor          507
getset_descriptor          451
type                       439

Which objects have been added or deleted?

We can also see which objects have been added or deleted between two points in time:

(pdb) import objgraph
(pdb) objgraph.show_growth()
.
.
.
(pdb) objgraph.show_growth()   # this only shows objects that has been added or deleted since last show_growth() call

traceback                4        +2
KeyboardInterrupt        1        +1
frame                   24        +1
list                   667        +1
tuple                16969        +1

What is referencing this leaky object?（资源引用计数）

Continuing down this route, we can also see where references to any given object is being held. Let’s take as an example the simple program below:

x = [1]
y = [x, [x], {"a":x}]
import pdb; pdb.set_trace()

To see what is holding a reference to the variable x, run the objgraph.show_backref()function:

(pdb) import objgraph
(pdb) objgraph.show_backref([x], filename="/tmp/backrefs.png")

The output of that command should be a PNG image stored at /tmp/backrefs.png and it should look something like this:

The box at the bottom with red lettering is our object of interest. We can see that it’s referenced by the symbol x once and by the list y three times. If x is the object causing a memory leak, we can use this method to see why it’s not automatically being deallocated by tracking down all of its references.

So to review, objgraph allows us to:

• show the top N objects occupying our python program’s memory
• show what objects have been deleted or added over a period of time
• show all references to a given object in our script

Effort vs precision

In this post, I’ve shown you how to use several tools to analyze a python program’s performance. Armed with these tools and techniques you should have all the information required to track down most memory leaks as well as identify speed bottlenecks in a Python program.

As with many other topics, running a performance analysis means balancing the tradeoffs between effort and precision. When in doubt, implement the simplest solution that will suit your current needs.

Refrences

• stack overflow - time explained
• line_profiler
• memory_profiler
• objgraph

文章参考：http://www.huyng.com/posts/python-performance-analysis/

Python开发之性能优化，首发于润物无声。

查看当前系统的版本信息

chs@chs-VirtualBox:~/project/openscada$ sudo lsb_release -a
[sudo] password for chs:
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 11.04
Release: 11.04
Codename: natty

可以发现当前的系统版本是11.04，代号为natty。

old-releases源信息如下

deb http://old-releases.ubuntu.com/ubuntu/ natty main restricted universe multiverse
deb http://old-releases.ubuntu.com/ubuntu/ natty-security main restricted universe multiverse
deb http://old-releases.ubuntu.com/ubuntu/ natty-updates main restricted universe multiverse
deb http://old-releases.ubuntu.com/ubuntu/ natty-proposed main restricted universe multiverse
deb http://old-releases.ubuntu.com/ubuntu/ natty-backports main restricted universe multiverse
deb-src http://old-releases.ubuntu.com/ubuntu/ natty main restricted universe multiverse
deb-src http://old-releases.ubuntu.com/ubuntu/ natty-security main restricted universe multiverse
deb-src http://old-releases.ubuntu.com/ubuntu/ natty-updates main restricted universe multiverse
deb-src http://old-releases.ubuntu.com/ubuntu/ natty-proposed main restricted universe multiverse
deb-src http://old-releases.ubuntu.com/ubuntu/ natty-backports main restricted universe multiverse

修改更新源，替换成上面的源信息

chs@chs-VirtualBox:~/project/openscada$sudo vim /etc/apt/sources.list

源更新

chs@chs-VirtualBox:~/project/openscada$sudo apt-get update

其他过期版本的old-releases源信息只需替换如上代号（natty）部分即可。

Ubuntu升级源备忘，首发于润物无声。

英文：svn: OPTIONS of 'https://server.domain.local/svn/repo': SSL handshake failed: SSL error:
Key usage violation in certificate has been detected. (https://server.domain.local)

中文：SSL handshake failed: SSL 错误：在证书中检测到违规的密钥用法。

解决方案

Symptoms

Subversion clients receive the following error message when attempting to connect to VisualSVN Server:

You may experience the issue if both of the following conditions are met:

• VisualSVN Server has a self-signed certificate applied and
• Subversion client is built against the GnuTLS library.

NoteGnuTLS library is an alternative to OpenSSL. Most Subversion clients for Windows are built against OpenSSL and are not affected by this issue. While some Subversion packages (available mostly on Linux-based operating systems such as Ubuntu and Debian) are built against GnuTLS and are affected.

Technical background

During the initial setup VisualSVN Server 2.5 generates a self-signed certificate and adds it to the Trusted Root Certification Authorities store on the local machine. To avoid possible security issues, VisualSVN Server makes this self-signed certificate to be valid for server authentication only (by specifying the 'Key Usage' extension). Subversion clients built against GnuTLS don't recognize such certificate and the error occurs.

Workaround

It's not recommended to use a self-signed certificate in a production environment. We advise to use a certificate issued by your domain or a third-party certificate authority instead of a self-signed one.

If you have to use a self-signed certificate please follow the instruction to generate a cerificate without specifying 'Key Usage' extension:

1. Add the following registry value to the Windows registry（修改注册表，添加一条dword类型的键值）:
   • for 32-bit system:

     [HKEY_LOCAL_MACHINE\SOFTWARE\VisualSVN\VisualSVN Server]
     "CreateGnuTLSCompatibleCertificate"=dword:00000001

   • for 64-bit system:

     [HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\VisualSVN\VisualSVN Server]
     "CreateGnuTLSCompatibleCertificate"=dword:00000001

2. Start VisualSVN Server Manager.
3. Go to Action | Properties | Certificate.
4. Click Change certificate...（重新生成新证书） and follow the wizard instructions to generate a new self-signed certificate.

The certificate will be generated without the 'Key Usage' extension and will be compatible both with GnuTLS and OpenSSL.

参考资料

1. http://www.visualsvn.com/support/topic/00056/

Ubuntu SVN客户端连接windows VisualSVN SSL证书问题，首发于润物无声。

Stickyworld's consultation web app has supported video for a long time but it's been hosted via a YouTube embed. When we started building the new version of the web app we wanted to take control of the video content and also free our users from YouTube's terms of service.

I personally had worked on projects with clients in the past which did video transcoding and it was never something easy to achieve. It takes a lot to accept every video, audio and container format under the sun and output them into various video formats that the web knows and loves.

With that in mind we decided the conversion process would be handled byEncoding.com. They let you encode your first Gigabyte of video with them for free and then have a tiered pricing system there after.

Throughout the development of the code below I would upload a two-second, 178KB video to test that everything was working. When the exceptions stopped being raised, I tested larger and more exotic files.

Stage 1: The User uploads a video

At the moment the new codebase just has a quick-and-dirty HTML5-based uploading mechanism. This is the CoffeeScript for uploading from the client to the server:

$scope.upload_slide = (upload_slide_form) ->
    file = document.getElementById("slide_file").files[0]
    reader = new FileReader()
    reader.readAsDataURL file
    reader.onload = (event) ->
      result = event.target.result
      fileName = document.getElementById("slide_file").files[0].name
      $.post "/world/upload_slide",
        data: result
        name: fileName
        room_id: $scope.room.id
        (response_data) ->
          if response_data.success? is not yes
            console.error "There was an error uploading the file", response_data
          else
            console.log "Upload successful", response_data
    reader.onloadstart = ->
      console.log "onloadstart"
    reader.onprogress = (event) ->
      console.log "onprogress", event.total, event.loaded, (event.loaded / event.total) * 100
    reader.onabort = ->
      console.log "onabort"
    reader.onerror = ->
      console.log "onerror"
    reader.onloadend = (event) ->
      console.log "onloadend", event

It would be nice to loop through ("slide_file").files and upload each file individually instead of just the first file. This is somewhere we'll be addressing soon.

Stage 2: Validate and upload to S3

On the backend we're running Django and RabbitMQ. The key modules we're using are:

$ pip install 'Django>=1.5.2' 'django-celery>=3.0.21' \
    'django-storages>=1.1.8' 'lxml>=3.2.3' 'python-magic>=0.4.3'

I created two models: SlideUploadQueue to store references to all the uploads and SlideVideoMedia to store all the references to the processed videos.

class SlideUploadQueue(models.Model):
    created_by = models.ForeignKey(User)
    created_time = models.DateTimeField(db_index=True)
    original_file = models.FileField(
        upload_to=filename_sanitiser, blank=True, default='')
    media_type = models.ForeignKey(MediaType)
    encoding_com_tracking_code = models.CharField(
        default='', max_length=24, blank=True)

    STATUS_AWAITING_DATA = 0
    STATUS_AWAITING_PROCESSING = 1
    STATUS_PROCESSING = 2
    STATUS_AWAITING_3RD_PARTY_PROCESSING = 5
    STATUS_FINISHED = 3
    STATUS_FAILED = 4

    STATUS_LIST = (
        (STATUS_AWAITING_DATA, 'Awaiting Data'),
        (STATUS_AWAITING_PROCESSING, 'Awaiting processing'),
        (STATUS_PROCESSING, 'Processing'),
        (STATUS_AWAITING_3RD_PARTY_PROCESSING,
            'Awaiting 3rd-party processing'),
        (STATUS_FINISHED, 'Finished'),
        (STATUS_FAILED, 'Failed'),
    )

    status = models.PositiveSmallIntegerField(
        default=STATUS_AWAITING_DATA, choices=STATUS_LIST)

    class Meta:
        verbose_name = 'Slide'
        verbose_name_plural = 'Slide upload queue'

    def save(self, *args, **kwargs):
        if not self.created_time:
            self.created_time = \
                datetime.utcnow().replace(tzinfo=pytz.utc)

        return super(SlideUploadQueue, self).save(*args, **kwargs)

    def __unicode__(self):
        if self.id is None:
            return 'new <SlideUploadQueue>'
        return '<SlideUploadQueue> %d' % self.id

class SlideVideoMedia(models.Model):
    converted_file = models.FileField(
        upload_to=filename_sanitiser, blank=True, default='')

    FORMAT_MP4 = 0
    FORMAT_WEBM = 1
    FORMAT_OGG = 2
    FORMAT_FL9 = 3
    FORMAT_THUMB = 4

    supported_formats = (
        (FORMAT_MP4, 'MPEG 4'),
        (FORMAT_WEBM, 'WebM'),
        (FORMAT_OGG, 'OGG'),
        (FORMAT_FL9, 'Flash 9 Video'),
        (FORMAT_THUMB, 'Thumbnail'),
    )

    mime_types = (
        (FORMAT_MP4, 'video/mp4'),
        (FORMAT_WEBM, 'video/webm'),
        (FORMAT_OGG, 'video/ogg'),
        (FORMAT_FL9, 'video/mp4'),
        (FORMAT_THUMB, 'image/jpeg'),
    )

    format = models.PositiveSmallIntegerField(
        default=FORMAT_MP4, choices=supported_formats)

    class Meta:
        verbose_name = 'Slide video'
        verbose_name_plural = 'Slide videos'

    def __unicode__(self):
        if self.id is None:
            return 'new <SlideVideoMedia>'
        return '<SlideVideoMedia> %d' % self.id

Our models use a filename_sanitiser method in each models.FileField field to automatically adjust filenames into a <model>/<uuid4>.<extention>format. This sanitises each filename and makes sure they're unique. To add to that, we used signed URLs that expire so we can control who is served our content and for how long.

def filename_sanitiser(instance, filename):
    folder = instance.__class__.__name__.lower()
    ext = 'jpg'

    if '.' in filename:
        t_ext = filename.split('.')[-1].strip().lower()

        if t_ext != '':
            ext = t_ext

    return '%s/%s.%s' % (folder, str(uuid.uuid4()), ext)

A file uploaded as testing.mov would turn into https://our-bucket.s3.amazonaws.com/slideuploadqueue/3fe27193-e87f-4244-9aa2-66409f70ebd3.mov and would be uploaded by Django Storages.

In our backend endpoint where video is uploaded to from the browser we validate the uploaded content using Magic. This detects what kind of file it is based on it's contents:

@verify_auth_token
@return_json
def upload_slide(request):
    file_data = request.POST.get('data', '')
    file_data = base64.b64decode(file_data.split(';base64,')[1])
    description = magic.from_buffer(file_data)

So if description matches something like MPEG v4 system or Apple QuickTime movie then we know it's suitable for transcoding. If it isn't something like the above, we can flag it up with the user.

Next, we'll save the video into a our SlideUploadQueue model and send a job off to RabbitMQ. Because we're using Django Storages, it'll be uploaded to Amazon S3 automatically.

slide_upload = SlideUploadQueue()
...
slide_upload.status = SlideUploadQueue.STATUS_AWAITING_PROCESSING
slide_upload.save()
slide_upload.original_file.\
    save('anything.%s' % file_ext, ContentFile(file_data))
slide_upload.save()

task = ConvertRawSlideToSlide()
task.delay(slide_upload)

Stage 3: Send the video to a 3rd party

RabbitMQ will take over and handle the task.delay(slide_upload) call.

Here all we're doing is sending Encoding.com a URL of our video and instructions on what output formats we want. They'll give us a job number in return which we'll use to check up on the progress on the transcoding at a later point.

class ConvertRawSlideToSlide(Task):
    queue = 'backend_convert_raw_slides'
    ...
    def _handle_video(self, slide_upload):
        mp4 = {
            'output': 'mp4',
            'size': '320x240',
            'bitrate': '256k',
            'audio_bitrate': '64k',
            'audio_channels_number': '2',
            'keep_aspect_ratio': 'yes',
            'video_codec': 'mpeg4',
            'profile': 'main',
            'vcodecparameters': 'no',
            'audio_codec': 'libfaac',
            'two_pass': 'no',
            'cbr': 'no',
            'deinterlacing': 'no',
            'keyframe': '300',
            'audio_volume': '100',
            'file_extension': 'mp4',
            'hint': 'no',
        }

        webm = {
            'output': 'webm',
            'size': '320x240',
            'bitrate': '256k',
            'audio_bitrate': '64k',
            'audio_sample_rate': '44100',
            'audio_channels_number': '2',
            'keep_aspect_ratio': 'yes',
            'video_codec': 'libvpx',
            'profile': 'baseline',
            'vcodecparameters': 'no',
            'audio_codec': 'libvorbis',
            'two_pass': 'no',
            'cbr': 'no',
            'deinterlacing': 'no',
            'keyframe': '300',
            'audio_volume': '100',
            'preset': '6',
            'file_extension': 'webm',
            'acbr': 'no',
        }

        ogg = {
            'output': 'ogg',
            'size': '320x240',
            'bitrate': '256k',
            'audio_bitrate': '64k',
            'audio_sample_rate': '44100',
            'audio_channels_number': '2',
            'keep_aspect_ratio': 'yes',
            'video_codec': 'libtheora',
            'profile': 'baseline',
            'vcodecparameters': 'no',
            'audio_codec': 'libvorbis',
            'two_pass': 'no',
            'cbr': 'no',
            'deinterlacing': 'no',
            'keyframe': '300',
            'audio_volume': '100',
            'file_extension': 'ogg',
            'acbr': 'no',
        }

        flv = {
            'output': 'fl9',
            'size': '320x240',
            'bitrate': '256k',
            'audio_bitrate': '64k',
            'audio_channels_number': '2',
            'keep_aspect_ratio': 'yes',
            'video_codec': 'libx264',
            'profile': 'high',
            'vcodecparameters': 'no',
            'audio_codec': 'libfaac',
            'two_pass': 'no',
            'cbr': 'no',
            'deinterlacing': 'no',
            'keyframe': '300',
            'audio_volume': '100',
            'file_extension': 'mp4',
        }

        thumbnail = {
            'output': 'thumbnail',
            'time': '5',
            'video_codec': 'mjpeg',
            'keep_aspect_ratio': 'yes',
            'file_extension': 'jpg',
        }

        encoder = Encoding(settings.ENCODING_API_USER_ID,
            settings.ENCODING_API_USER_KEY)
        resp = encoder.add_media(source=[slide_upload.original_file.url],
            formats=[mp4, webm, ogg, flv, thumbnail])

        media_id = None

        if resp is not None and resp.get('response') is not None:
            media_id = resp.get('response').get('MediaID')

        if media_id is None:
            slide_upload.status = SlideUploadQueue.STATUS_FAILED
            slide_upload.save()
            log.error('Unable to communicate with encoding.com')
            return False

        slide_upload.encoding_com_tracking_code = media_id
        slide_upload.status = \
            SlideUploadQueue.STATUS_AWAITING_3RD_PARTY_PROCESSING
        slide_upload.save()
        return True

Encoding.com recommended some less-than-ideal python wrappers for communicating with their service. I added a few fixes into the module but there is still work to be done to get it to a state I'm happy with. Below is what this wrapper currently looks like in our codebase:

import httplib
from lxml import etree
import urllib
from xml.parsers.expat import ExpatError
import xmltodict

ENCODING_API_URL = 'manage.encoding.com:80'

class Encoding(object):

    def __init__(self, userid, userkey, url=ENCODING_API_URL):
        self.url = url
        self.userid = userid
        self.userkey = userkey

    def get_media_info(self, action='GetMediaInfo', ids=[],
        headers={'Content-Type': 'application/x-www-form-urlencoded'}):
        query = etree.Element('query')

        nodes = {
            'userid': self.userid,
            'userkey': self.userkey,
            'action': action,
            'mediaid': ','.join(ids),
        }

        query = self._build_tree(etree.Element('query'), nodes)
        results = self._execute_request(query, headers)

        return self._parse_results(results)

    def get_status(self, action='GetStatus', ids=[], extended='no',
        headers={'Content-Type': 'application/x-www-form-urlencoded'}):
        query = etree.Element('query')

        nodes = {
            'userid': self.userid,
            'userkey': self.userkey,
            'action': action,
            'extended': extended,
            'mediaid': ','.join(ids),
        }

        query = self._build_tree(etree.Element('query'), nodes)
        results = self._execute_request(query, headers)

        return self._parse_results(results)

    def add_media(self, action='AddMedia', source=[], notify='', formats=[],
        instant='no',
        headers={'Content-Type': 'application/x-www-form-urlencoded'}):
        query = etree.Element('query')

        nodes = {
            'userid': self.userid,
            'userkey': self.userkey,
            'action': action,
            'source': source,
            'notify': notify,
            'instant': instant,
        }

        query = self._build_tree(etree.Element('query'), nodes)

        for format in formats:
            format_node = self._build_tree(etree.Element('format'), format)
            query.append(format_node)

        results = self._execute_request(query, headers)
        return self._parse_results(results)

    def _build_tree(self, node, data):
        for k, v in data.items():
            if isinstance(v, list):
                for item in v:
                    element = etree.Element(k)
                    element.text = item
                    node.append(element)
            else:
                element = etree.Element(k)
                element.text = v
                node.append(element)

        return node

    def _execute_request(self, xml, headers, path='', method='POST'):
        params = urllib.urlencode({'xml': etree.tostring(xml)})

        conn = httplib.HTTPConnection(self.url)
        conn.request(method, path, params, headers)
        response = conn.getresponse()
        data = response.read()
        conn.close()
        return data

    def _parse_results(self, results):
        try:
            return xmltodict.parse(results)
        except ExpatError, e:
            print 'Error parsing encoding.com response'
            print e
            return None

Left on the todo list include HTTPS-only transmission with strict validation of Encoding.com's SSL certificate and to write some unit tests (tickets, tickets and more tickets).

Stage 4: Download all new video formats

We have a periodic job running every 15 seconds via RabbitMQ which is checking up on the progress of the video transcoding:

class CheckUpOnThirdParties(PeriodicTask):
    run_every = timedelta(seconds=settings.THIRD_PARTY_CHECK_UP_INTERVAL)
    ...
    def _handle_encoding_com(self, slides):
        format_lookup = {
            'mp4': SlideVideoMedia.FORMAT_MP4,
            'webm': SlideVideoMedia.FORMAT_WEBM,
            'ogg': SlideVideoMedia.FORMAT_OGG,
            'fl9': SlideVideoMedia.FORMAT_FL9,
            'thumbnail': SlideVideoMedia.FORMAT_THUMB,
        }

        encoder = Encoding(settings.ENCODING_API_USER_ID,
            settings.ENCODING_API_USER_KEY)

        job_ids = [item.encoding_com_tracking_code for item in slides]
        resp = encoder.get_status(ids=job_ids)

        if resp is None:
            log.error('Unable to check up on encoding.com')
            return False

We'll go through the response from Encoding.com validating each piece of the response as we go along:

if resp.get('response') is None:
    log.error('Unable to get response node from encoding.com')
    return False

resp_id = resp.get('response').get('id')

if resp_id is None:
    log.error('Unable to get media id from encoding.com')
    return False

slide = SlideUploadQueue.objects.filter(
    status=SlideUploadQueue.STATUS_AWAITING_3RD_PARTY_PROCESSING,
    encoding_com_tracking_code=resp_id)

if len(slide) != 1:
    log.error('Unable to find a single record for %s' % resp_id)
    return False

resp_status = resp.get('response').get('status')

if resp_status is None:
    log.error('Unable to get status from encoding.com')
    return False

if resp_status != u'Finished':
    log.debug("%s isn't finished, will check back later" % resp_id)
    return True

formats = resp.get('response').get('format')

if formats is None:
    log.error("No output formats were found. Something's wrong.")
    return False

for format in formats:
    try:
        assert format.get('status') == u'Finished', \
        "%s is not finished. Something's wrong." % format.get('id')

        output = format.get('output')
        assert output in ('mp4', 'webm', 'ogg', 'fl9',
            'thumbnail'), 'Unknown output format %s' % output

        s3_dest = format.get('s3_destination')
        assert 'http://encoding.com.result.s3.amazonaws.com/'\
            in s3_dest, 'Suspicious S3 url: %s' % s3_dest

        https_link = \
            'https://s3.amazonaws.com/encoding.com.result/%s' %\
            s3_dest.split('/')[-1]
        file_ext = https_link.split('.')[-1].strip()

        assert len(file_ext) > 0,\
            'Unable to get file extension from %s' % https_link

        count = SlideVideoMedia.objects.filter(slide_upload=slide,
            format=format_lookup[output]).count()

        if count != 0:
            print 'There is already a %s file for this slide' % output
            continue

        content = self.download_content(https_link)

        assert content is not None,\
            'There is no content for %s' % format.get('id')
    except AssertionError, e:
        log.error('A format did not pass all assertions: %s' % e)
        continue

At this point we've asserted everything is as it should be a we can save each of the videos:

media = SlideVideoMedia()
media.format = format_lookup[output]
media.converted_file.save('blah.%s' % file_ext, ContentFile(content))
media.save()

Stage 5: Video via HTML5

On our frontend we've created a page with an HTML5 video element. We're using video.js to display the video in the best-supported format for each browser.

➫ bower install video.js
bower caching git://github.com/videojs/video.js-component.git
bower cloning git://github.com/videojs/video.js-component.git
bower fetching video.js
bower checking out video.js#v4.0.3
bower copying /home/mark/.bower/cache/video.js/5ab058cd60c5615aa38e8e706cd0f307
bower installing video.js#4.0.3

In our index.jade file we include it's dependencies:

!!! 5
html(lang="en", class="no-js")
  head
    meta(http-equiv='Content-Type', content='text/html; charset=UTF-8')
    ...
    link(rel='stylesheet', type='text/css', href='/components/video-js-4.1.0/video-js.css')
    script(type='text/javascript', src='/components/video-js-4.1.0/video.js')

In a Angular.js/JADE-based template we've included a <video> tag and it's <source> children tags. There is also a poster element that will show a static image of the video we've transcoded from the first few moments of the video.

#main.span12
    video#example_video_1.video-js.vjs-default-skin(controls, preload="auto", width="640", height="264", poster="{{video_thumbnail}}", data-setup='{"example_option":true}', ng-show="videos")
        source(ng-repeat="video in videos", src="{{video.src}}", type="{{video.type}}")

This will print out every video format we've converted into, each as it's own <source> tag. Video.js will decide which of them to play based on the browser the user is using.

We still have a lot of work to do around fallback support, building unit tests and improving the robustness of our Encoding.com service wrapper. If this sort of work interests you please do get in touch.

文章节选：http://techblog.stickyworld.com/video-with-python.html

搭建基于Python的视频服务器，首发于润物无声。