其中比較有趣的是
1.
Mail::SpamAssassin::Spamd::Apache2 -- mod_perl2 module, implementing spamd as a mod_perl module, contributed as a Google Summer of Code project by Radoslaw Zielinski.
沒錯apache2 已不像過去,只是一個web server,他可以在上面發展其他的protocol,除了之前看過的smtp/ftp現在又多了spamd,但之前在apache2 上開發 lprd protocol 卻發現他的效率似乎沒有比自己寫伺服器來的好,至於穩不穩定,可能要有實際的數據支持
2.
sa-compile: compilation of SpamAssassin rules into a fast parallel-matching DFA, implemented in native code
這個技術我還沒時間仔細去trace,不過他應該是將spamassassin 使用的rules, 透過這個工具轉成c , 然後透過 xs 讓perl 來使用,這應該很有研究價值
Saturday, May 19, 2007
Monday, April 30, 2007
xmpp 中文翻譯計畫 網站
http://wiki.jabbercn.org/space/start
另一個不錯的相關網站
http://hi.baidu.com/jabber/blog/category/Jep
iq:roster 的交易過程真是複雜的可以
http://64.233.179.104/translate_c?hl=zh-TW&sl=zh-CN&u=http://www.linuxboy.net/jabber/004.html&prev=/search%3Fq%3Diq:roster%26complete%3D1%26hl%3Dzh-TW%26pwst%3D1
另一個不錯的相關網站
http://hi.baidu.com/jabber/blog/category/Jep
iq:roster 的交易過程真是複雜的可以
http://64.233.179.104/translate_c?hl=zh-TW&sl=zh-CN&u=http://www.linuxboy.net/jabber/004.html&prev=/search%3Fq%3Diq:roster%26complete%3D1%26hl%3Dzh-TW%26pwst%3D1
Saturday, April 28, 2007
Concurrent Programming 相關報告
一. 我會接觸Erlang的緣由
1.RFID Middleware
2.jabber (xml::stream http://zh.wikipedia.org/wiki/Jabber)
3.ejabber (http://www.process-one.net/en/ )
二. 現在的商業環境(web server)所面臨的問題
1.連線的數量不斷的攀升
2.連線的時間很長
傳統上httpd 使用Prefork的方式來解決,短時間時密集連線的問題,在現在的環境愈到了嚴重的挑戰,比如: HTTP_Streaming、Server Push、COMET 這些需要長時間連線的架構,使得httpd 能夠服務的連線變少了,而fork process 最大的問題是,他所需要佔用記憶體的空間過於龐大,於是其他的伺服器架構崛起(lighthttpd ghttpd …)
The C10K problem( http://www.kegel.com/c10k.html )
It's time for web servers to handle ten thousand clients simultaneously, don't you think? After all, the web is a big place now.
And computers are big, too. You can buy a 1000MHz machine with 2 gigabytes of RAM and an 1000Mbit/sec Ethernet card for $1200 or so. Let's see - at 20000 clients, that's 50KHz, 100Kbytes, and 50Kbits/sec per client. It shouldn't take any more horsepower than that to take four kilobytes from the disk and send them to the network once a second for each of twenty thousand clients. (That works out to $0.08 per client, by the way. Those $100/client licensing fees some operating systems charge are starting to look a little heavy!) So hardware is no longer the bottleneck???
三. Concurrent Programming
1. fork
原始的程式
(程式+資料) --fork(複製一份)(程式+資料)
當程式fork 後,child 繼承原來的資料,此後彼此不相關,如果要傳遞資訊,需要使用pipe sharememory 或是 unix socket 來做資料交換
2. thread
事實上在Linux 系統下，執行緒只是一個light weight process：Linux 核心是以fork() system call 來產生一個新的行程（process），而執行緒是以clone() system call 產生的。fork()和clone()的差別只是在clone()可以指定和父行程共用的資源有哪些，當所有資源都和父行程共用時就相當於一個執行緒了。因為Thread 的使用會讓子父行程共用資源,因此非常容易引發dead lock / race condition …這類的問題
3. lightweight Threads ( http://www.defmacro.org/ramblings/concurrency.html)
Erlang process 是一個輕量級的Thread,因此他可以非常輕易的去開啟或是結束且快速在彼此做切換,因為掀開他的底層,他只是一個簡單的function罷了,process節省了大量的context switching浪費僅在一些function上做切換的動作(Erlang 的Thread 是 vm level thread)
這份文件簡單的提到了Erlang的概觀
http://mirror.linux.org.au/pub/linux.conf.au/2007/video/talks
/252.pdf
四. Erlang ( http://www.erlang.org/ )
1.以下是 about Erlang 對他自己的簡述
Erlang is a programming language which has many features more commonly associated with an operating system than with a programming language: concurrent processes, scheduling, memory management, distribution, networking, etc.
The initial open-source Erlang release contains the implementation of Erlang, as well as a large part of Ericsson's middleware for building distributed high-availability systems.
Erlang is characterized by the following features:
Concurrency - Erlang has extremely lightweight processes whose memory requirements can vary dynamically. Processes have no shared memory and communicate by asynchronous message passing. Erlang supports applications with very large numbers of concurrent processes. No requirements for concurrency are placed on the host operating system.
Distribution - Erlang is designed to be run in a distributed environment. An Erlang virtual machine is called an Erlang node. A distributed Erlang system is a network of Erlang nodes (typically one per processor). An Erlang node can create parallel processes running on other nodes, which perhaps use other operating systems. Processes residing on different nodes communicate in exactly the same was as processes residing on the same node.
Soft real-time - Erlang supports programming "soft" real-time systems, which require response times in the order of milliseconds. Long garbage collection delays in such systems are unacceptable, so Erlang uses incremental garbage collection techniques.
Hot code upgrade - Many systems cannot be stopped for software maintenance. Erlang allows program code to be changed in a running system. Old code can be phased out and replaced by new code. During the transition, both old code and new code can coexist. It is thus possible to install bug fixes and upgrades in a running system without disturbing its operation.
Incremental code loading - Users can control in detail how code is loaded. In embedded systems, all code is usually loaded at boot time. In development systems, code is loaded when it is needed, even when the system is running. If testing uncovers bugs, only the buggy code need be replaced.
External interfaces - Erlang processes communicate with the outside world using the same message passing mechanism as used between Erlang processes. This mechanism is used for communication with the host operating system and for interaction with programs written in other languages. If required for reasons of efficiency, a special version of this concept allows e.g. C programs to be directly linked into the Erlang runtime system.
2.Erlang 語言上的概觀
書籍: ( http://pragmaticprogrammer.com/titles/jaerlang/index.html )
[ Sequential Erlang ]
Exam1:
Consider the factorial function N! defined by:
N!=N*(N-1) when N>0
N!=1 when N=0
-module(math1).
-export([fac/1]).
fac(N) when N > 0 -> N * fac(N-1);
fac(0)-> 1.
Exam2:
-module(math2).
-export([sum1/1, sum2/1]).
sum1([H T]) -> H + sum1(T);
sum1([]) -> 0.
sum2(L) -> sum2(L, 0).
sum2([], N) -> N;
sum2([H T], N) -> sum2(T, H+N).
[ Concurrency Programming ]
Exam3:
-module(concurrency).
-export([start/0, say /2]).
say (What, 0) ->
done;
say (What, Times) ->
io:format("~p~n", [What]),
say_something(What, Times - 1).
start() ->
spawn(tut14, say, [hello, 3]),
spawn(tut14, say, [goodbye, 3]).
Exam4:
-module(area_server).
-export([loop/0]).
loop() ->
receive
{rectangle, Width, Ht} ->
io:format("Area of rectangle is ~p~n",[Width * Ht]),
loop();
{circle, R} ->
io:format("Area of circle is ~p~n", [3.14159 * R * R]),
loop();
Other ->
io:format("I don't know what the area of a ~p is ~n",[Other]),
loop()
end.
We can create a process which evaluates loop/0 in the shell:
Pid = spawn(area_server,loop,[]).
Pid ! {rectangle, 6, 10}.
Pid ! {circle, 23}.
Pid ! {triangle,2,4,5}.
4. Erlang –style process or event-based model for actors ( http://lambda-the-ultimate.org/node/1615 )
( http://lamp.epfl.ch/~phaller/doc/haller07coord.pdf )
Message passing
Each process has its own input queue for messages it receives. New messages received are put at the end of the queue. When a process executes a receive, the first message in the queue is matched against the first pattern in the receive, if this matches, the message is removed from the queue and the actions corresponding to the the pattern are executed.
However, if the first pattern does not match, the second pattern is tested, if this matches the message is removed from the queue and the actions corresponding to the second pattern are executed. If the second pattern does not match the third is tried and so on until there are no more pattern to test. If there are no more patterns to test, the first message is kept in the queue and we try the second message instead. If this matches any pattern, the appropriate actions are executed and the second message is removed from the queue (keeping the first message and any other messages in the queue). If the second message does not match we try the third message and so on until we reach the end of the queue. If we reach the end of the queue, the process blocks (stops execution) and waits until a new message is received and this procedure is repeated.
Of course the Erlang implementation is "clever" and minimizes the number of times each message is tested against the patterns in each receive.
五. Erlang相關資源
Website:
Open Source Erlang
http://www.erlang.org
http://www.process-one.net/en/projects/
Mail List:
Erlang-questions -- Erlang/OTP discussions
http://www.erlang.org/mailman/listinfo/erlang-questions
BOOK:
Concurrent programming in Erlang
http://www.erlang.org/download/erlang-book-part1.pdf
Programming Erlang Software for a Concurrent World
http://pragmaticprogrammer.com/titles/jaerlang/index.html
1.RFID Middleware
2.jabber (xml::stream http://zh.wikipedia.org/wiki/Jabber)
3.ejabber (http://www.process-one.net/en/ )
二. 現在的商業環境(web server)所面臨的問題
1.連線的數量不斷的攀升
2.連線的時間很長
傳統上httpd 使用Prefork的方式來解決,短時間時密集連線的問題,在現在的環境愈到了嚴重的挑戰,比如: HTTP_Streaming、Server Push、COMET 這些需要長時間連線的架構,使得httpd 能夠服務的連線變少了,而fork process 最大的問題是,他所需要佔用記憶體的空間過於龐大,於是其他的伺服器架構崛起(lighthttpd ghttpd …)
The C10K problem( http://www.kegel.com/c10k.html )
It's time for web servers to handle ten thousand clients simultaneously, don't you think? After all, the web is a big place now.
And computers are big, too. You can buy a 1000MHz machine with 2 gigabytes of RAM and an 1000Mbit/sec Ethernet card for $1200 or so. Let's see - at 20000 clients, that's 50KHz, 100Kbytes, and 50Kbits/sec per client. It shouldn't take any more horsepower than that to take four kilobytes from the disk and send them to the network once a second for each of twenty thousand clients. (That works out to $0.08 per client, by the way. Those $100/client licensing fees some operating systems charge are starting to look a little heavy!) So hardware is no longer the bottleneck???
三. Concurrent Programming
1. fork
原始的程式
(程式+資料) --fork(複製一份)(程式+資料)
當程式fork 後,child 繼承原來的資料,此後彼此不相關,如果要傳遞資訊,需要使用pipe sharememory 或是 unix socket 來做資料交換
2. thread
事實上在Linux 系統下，執行緒只是一個light weight process：Linux 核心是以fork() system call 來產生一個新的行程（process），而執行緒是以clone() system call 產生的。fork()和clone()的差別只是在clone()可以指定和父行程共用的資源有哪些，當所有資源都和父行程共用時就相當於一個執行緒了。因為Thread 的使用會讓子父行程共用資源,因此非常容易引發dead lock / race condition …這類的問題
3. lightweight Threads ( http://www.defmacro.org/ramblings/concurrency.html)
Erlang process 是一個輕量級的Thread,因此他可以非常輕易的去開啟或是結束且快速在彼此做切換,因為掀開他的底層,他只是一個簡單的function罷了,process節省了大量的context switching浪費僅在一些function上做切換的動作(Erlang 的Thread 是 vm level thread)
這份文件簡單的提到了Erlang的概觀
http://mirror.linux.org.au/pub/linux.conf.au/2007/video/talks
/252.pdf
四. Erlang ( http://www.erlang.org/ )
1.以下是 about Erlang 對他自己的簡述
Erlang is a programming language which has many features more commonly associated with an operating system than with a programming language: concurrent processes, scheduling, memory management, distribution, networking, etc.
The initial open-source Erlang release contains the implementation of Erlang, as well as a large part of Ericsson's middleware for building distributed high-availability systems.
Erlang is characterized by the following features:
Concurrency - Erlang has extremely lightweight processes whose memory requirements can vary dynamically. Processes have no shared memory and communicate by asynchronous message passing. Erlang supports applications with very large numbers of concurrent processes. No requirements for concurrency are placed on the host operating system.
Distribution - Erlang is designed to be run in a distributed environment. An Erlang virtual machine is called an Erlang node. A distributed Erlang system is a network of Erlang nodes (typically one per processor). An Erlang node can create parallel processes running on other nodes, which perhaps use other operating systems. Processes residing on different nodes communicate in exactly the same was as processes residing on the same node.
Soft real-time - Erlang supports programming "soft" real-time systems, which require response times in the order of milliseconds. Long garbage collection delays in such systems are unacceptable, so Erlang uses incremental garbage collection techniques.
Hot code upgrade - Many systems cannot be stopped for software maintenance. Erlang allows program code to be changed in a running system. Old code can be phased out and replaced by new code. During the transition, both old code and new code can coexist. It is thus possible to install bug fixes and upgrades in a running system without disturbing its operation.
Incremental code loading - Users can control in detail how code is loaded. In embedded systems, all code is usually loaded at boot time. In development systems, code is loaded when it is needed, even when the system is running. If testing uncovers bugs, only the buggy code need be replaced.
External interfaces - Erlang processes communicate with the outside world using the same message passing mechanism as used between Erlang processes. This mechanism is used for communication with the host operating system and for interaction with programs written in other languages. If required for reasons of efficiency, a special version of this concept allows e.g. C programs to be directly linked into the Erlang runtime system.
2.Erlang 語言上的概觀
書籍: ( http://pragmaticprogrammer.com/titles/jaerlang/index.html )
[ Sequential Erlang ]
Exam1:
Consider the factorial function N! defined by:
N!=N*(N-1) when N>0
N!=1 when N=0
-module(math1).
-export([fac/1]).
fac(N) when N > 0 -> N * fac(N-1);
fac(0)-> 1.
Exam2:
-module(math2).
-export([sum1/1, sum2/1]).
sum1([H T]) -> H + sum1(T);
sum1([]) -> 0.
sum2(L) -> sum2(L, 0).
sum2([], N) -> N;
sum2([H T], N) -> sum2(T, H+N).
[ Concurrency Programming ]
Exam3:
-module(concurrency).
-export([start/0, say /2]).
say (What, 0) ->
done;
say (What, Times) ->
io:format("~p~n", [What]),
say_something(What, Times - 1).
start() ->
spawn(tut14, say, [hello, 3]),
spawn(tut14, say, [goodbye, 3]).
Exam4:
-module(area_server).
-export([loop/0]).
loop() ->
receive
{rectangle, Width, Ht} ->
io:format("Area of rectangle is ~p~n",[Width * Ht]),
loop();
{circle, R} ->
io:format("Area of circle is ~p~n", [3.14159 * R * R]),
loop();
Other ->
io:format("I don't know what the area of a ~p is ~n",[Other]),
loop()
end.
We can create a process which evaluates loop/0 in the shell:
Pid = spawn(area_server,loop,[]).
Pid ! {rectangle, 6, 10}.
Pid ! {circle, 23}.
Pid ! {triangle,2,4,5}.
4. Erlang –style process or event-based model for actors ( http://lambda-the-ultimate.org/node/1615 )
( http://lamp.epfl.ch/~phaller/doc/haller07coord.pdf )
Message passing
Each process has its own input queue for messages it receives. New messages received are put at the end of the queue. When a process executes a receive, the first message in the queue is matched against the first pattern in the receive, if this matches, the message is removed from the queue and the actions corresponding to the the pattern are executed.
However, if the first pattern does not match, the second pattern is tested, if this matches the message is removed from the queue and the actions corresponding to the second pattern are executed. If the second pattern does not match the third is tried and so on until there are no more pattern to test. If there are no more patterns to test, the first message is kept in the queue and we try the second message instead. If this matches any pattern, the appropriate actions are executed and the second message is removed from the queue (keeping the first message and any other messages in the queue). If the second message does not match we try the third message and so on until we reach the end of the queue. If we reach the end of the queue, the process blocks (stops execution) and waits until a new message is received and this procedure is repeated.
Of course the Erlang implementation is "clever" and minimizes the number of times each message is tested against the patterns in each receive.
五. Erlang相關資源
Website:
Open Source Erlang
http://www.erlang.org
http://www.process-one.net/en/projects/
Mail List:
Erlang-questions -- Erlang/OTP discussions
http://www.erlang.org/mailman/listinfo/erlang-questions
BOOK:
Concurrent programming in Erlang
http://www.erlang.org/download/erlang-book-part1.pdf
Programming Erlang Software for a Concurrent World
http://pragmaticprogrammer.com/titles/jaerlang/index.html
Tuesday, April 24, 2007
xmpp 實做的分享
最近在寫jabber server, jabber 是建構在xmpp protocol 上的一個IM,因為RFC的規格制定曠日費時,所以;jabber 以xmpp 為基礎,自己又定義了約200 個協定XEP-0001~0214,而 xmpp 主要由五個protocol所組成,分別是RFC-3920~3923 RFC-4622
我目前的進度已經可以讓像 Exodus or Pandion(IM Client) 連接上我自己實做的jabber server,預計下星期我就能讓im client 直接在上面talk,且訂閱彼此的狀態...在寫的過程成中越來越覺得他的複雜,其實,這大概是我目前寫過最複雜的伺服器,不過有一點心得可以先分享給大家,其實有一些人有一個疑問,jabber長的像什麼? 如果我們撇開他在IM的實做(處理訊息的傳遞也是一種運算資源),我們可以把它看成是一家公司,一家公司會有他對客戶的服務,而當產品要製作時,它需要資源,需要應徵人員,每個應徵的人員需要依照公司的制度來運行(component plug-in),每個人員應徵後需要報到,然後正式工作,依照給個人的專業知識分派工作 ....循環不已,當新的產品要製作時,這個公司可以再應徵新的 不同專業領域的資源,而且可以重新制定新的工作規則...而公司組織裡的這些運行其實都是靠制度,而這個制度相對於jabber 就是他的protocol,所以我把xmpp形容成是一個資源/運算的分散者,因此他可以建構一個基本的Grid Computing 環境,把每個運算工作分散到無限台機器上....如果你要問他可以做什麼? 事實上在jabber 的protocol 裡幾乎定義了絕大部分
的應用,voip 影音...,所有的運算資源都可以在事後 plug in 進去,google_talk 所實做的部份可能還不到整個jabber 的1/20,由此;我們可以看出他的規模/擴充性之大... 總之把他想成是一個有組織的公司,只是公司的規模有大有小罷了,xmpp 的工作資源分配真的跟這個描述很像,有機會自己實做一次體會一下囉!
我目前的進度已經可以讓像 Exodus or Pandion(IM Client) 連接上我自己實做的jabber server,預計下星期我就能讓im client 直接在上面talk,且訂閱彼此的狀態...在寫的過程成中越來越覺得他的複雜,其實,這大概是我目前寫過最複雜的伺服器,不過有一點心得可以先分享給大家,其實有一些人有一個疑問,jabber長的像什麼? 如果我們撇開他在IM的實做(處理訊息的傳遞也是一種運算資源),我們可以把它看成是一家公司,一家公司會有他對客戶的服務,而當產品要製作時,它需要資源,需要應徵人員,每個應徵的人員需要依照公司的制度來運行(component plug-in),每個人員應徵後需要報到,然後正式工作,依照給個人的專業知識分派工作 ....循環不已,當新的產品要製作時,這個公司可以再應徵新的 不同專業領域的資源,而且可以重新制定新的工作規則...而公司組織裡的這些運行其實都是靠制度,而這個制度相對於jabber 就是他的protocol,所以我把xmpp形容成是一個資源/運算的分散者,因此他可以建構一個基本的Grid Computing 環境,把每個運算工作分散到無限台機器上....如果你要問他可以做什麼? 事實上在jabber 的protocol 裡幾乎定義了絕大部分
的應用,voip 影音...,所有的運算資源都可以在事後 plug in 進去,google_talk 所實做的部份可能還不到整個jabber 的1/20,由此;我們可以看出他的規模/擴充性之大... 總之把他想成是一個有組織的公司,只是公司的規模有大有小罷了,xmpp 的工作資源分配真的跟這個描述很像,有機會自己實做一次體會一下囉!
Friday, April 20, 2007
Tuesday, April 17, 2007
Saturday, April 14, 2007
javascript 縮減語法的工具
http://dean.edwards.name/packer/
這是一個javascript 縮減語法的工具,他的原理其實就是把 javascript 內不必要的斷行 空白...去除,這個網站還提供php .net ...的語法縮減工具,縮減語法的好處可以降低網路頻寬的使用,參考google 的首頁就知道,google 不只把js 縮減,也把html 也縮減了,這樣可以節省大量網頁下載時所使用的頻寬
另外google 還會對輸出網頁做gzip 的壓縮,這一點我就比較不明白,事實上;使用cache 的機制會比做gzip來的更節省(http hrader: cache-control expire if-modify-since etag...尤其是針對不常變動的首頁),然而;ie 6 有一個bug,那就是當網頁使用gzip 時,上述的標頭都會失效(firefox 則不會有此問題),因此;這一點是我比較不解的地方
這是一個javascript 縮減語法的工具,他的原理其實就是把 javascript 內不必要的斷行 空白...去除,這個網站還提供php .net ...的語法縮減工具,縮減語法的好處可以降低網路頻寬的使用,參考google 的首頁就知道,google 不只把js 縮減,也把html 也縮減了,這樣可以節省大量網頁下載時所使用的頻寬
另外google 還會對輸出網頁做gzip 的壓縮,這一點我就比較不明白,事實上;使用cache 的機制會比做gzip來的更節省(http hrader: cache-control expire if-modify-since etag...尤其是針對不常變動的首頁),然而;ie 6 有一個bug,那就是當網頁使用gzip 時,上述的標頭都會失效(firefox 則不會有此問題),因此;這一點是我比較不解的地方
Subscribe to:
Posts (Atom)