PInetd2 is closing to release with the addition of two elements, a new launcher, and logs.

The new launcher is the first step toward the ability to dynamically start and stop subprocesses without restarting the whole daemon. The old “start.sh” has been replaced with a “daemonctl” tool, like in the old pinetd. This allows to start pinetd2 in the background (if forking is enabled), see processes status, and stop/restart the daemon directly. However since daemon can now be started in background, another feature was needed, which is…

Logging to a file

Of course, the whole system was made to centralize logs. Each child transmits logs via IPC to its parent, and the main launcher display logs on screen. It wasn’t difficult from there to write logs to a file. This means that even a FTP client chrooted somewhere can still send log events to its parent and get them written to the main logfile.

What’s next?

Now, I need to make those two elements less hack-like (ie. add some config options, add ability to log to syslog, etc), then I’ll have to finalize PMaild’s IMAP daemon thanks to the fact I added mailparse to the required elements in the PHP distribution. MailParse will handle the difficult mime-decoding part, and will make the next steps much easier. Parsed data will be cached in database (one new table per domain), and this should finally makes PMaild usable in imap-based webmails like Horde IMP.

If you never heared about PInetd, it probably means you never tried to search on google how to make a DNS server in PHP, for example.

PInetd is a generic framework allowing to run a wide range of daemons, mostly answering on TCP or UDP ports.

The main goal is to make generic protocol interfaces that can easily be overloaded to implement new features, new auth mechanism, etc… For example the FTP daemon is extended as FTPd_SQL for SQL auth.

The pinetd2 project is under development since October 4th 2007 (will be 2 years old soon), and has reached an appreciable maturity level. Most bugs were cleared and right now the system is running well. Some expected features (http admin interface, transports, etc) are not completed yet, but should appear within the next weeks, before the first official release with version 2.0.0.

PInetd includes framework tools to easily make a daemon listening on a TCP or UDP port, and also include some pre-made modules: FTP, HTTP, Mails (pop/smtp), DNS, NetBatch…

While official documentation is still mostly missing, previous posts here explain how to configure a DNS daemon, and the wiki contains basic documentation.

The official PInetd website is now open, and it is possible to subscribe to news feed to receive updates either by RSS or by mail.

While the PHP team is currently working hard getting PHP 5.3 stable, I can officially announce that PInetd’s old branch is no longer supported.

All efforts will be put on pinetd2 with its new framework, and improvements required to support transports.

A new pinetd branch will eventually be launched with a complete rewrite of the IPC subsystem (which has become too complex with the different new features added along the line). New branch will most likely be named 2.1.

As you maybe know, PInetd2 is my new internet daemon framework, totally rewrote to be more stable, make use of PHP5.3′s namespaces, etc…

I was planning to release PInetd2 as soon as PHP 5.3 gets stable (which will maybe be the case with 5.3.1 ?), until it happens I’m adding new features and fixing bugs.

I’ll start with something I already coded (NetBatch) before introducing the new feature I’m working on (Transports).

NetBatch

NetBatch is a new daemon allowing to run applications on a local or remote server. It has many possible uses, since it even allows running stuff as root, or run stuff in background.

With NetBatch, reloading apache’s configuration, restarting a server, etc… from an apache process is no longer a dream. You can just connect to whatever server you want, and run the stuff. NetBatch is running in async mode, meaning you can even run interactive applications and provide input when you feed the need to.

NetBatch also has a different operation mode, named “Persistent”. It allows you to run a (potentially long) operation, and connect later to get the result. You have to poll the application (or NetBatch will kill it), and you can control it like a normal application. The only difference is when you disconnect/reconnect, the application is still there.

Transports

Transports is a new feature allowing implementation of read/write resources. The two first transports I’ll implement are IRC and RSS. Both will provide some kinds of resources (irc_channel, irc_user, rss_feed) and the ability to read objects/write objects on those.
rss_feed will be a read-only resource providing rss feed polling. irc_channel will be a read/write resource allowing to control a generic bot on (a) given channel(s). irc_user will provide full control over the bot, private messages, etc (and will cause a dedicated connection).

Other transports are also planned, including but not limited to jabber, twitter, msn, etc… (anything that can emit or receive “events”).

Configuring a transport is done via an XML node. Each transport will have its own settings (for example “refreshTime” for rss_feed), and will have a “connected state”. When a transport is “connected”, it may have a separate process launched handling the connection.

Transports are handled using pinetd2′s internal IPC system, however they don’t require polling to work. Once connected, incoming data will be sent directly from the transport, as soon as it is made available.

The goal is of course to allow creation of various things with pinetd: bots, notifications, etc

Future

One of the biggest missing features in pinetd2 is an administration interface. A way to view logs, start/stop daemons, edit configuration, etc… I’m planning to do this using pinetd2′s HTTP daemon, and started up to some extent, but haven’t found anyone to help me by working on the HTML interface (I’m really bad with HTML).
Once pinetd2 gets an administration interface, I’ll be able to go through the documentation process before starting a release cycle (RC, then release) to finally get a stable release of pinetd2.

After my initial announcement of a PHP DNS Daemon, and some performance tests, and since it has been widely reported over internet (Zend DevZone, and today on Nexen which is deeply involved with French php community), I finally found the strength to write a guide on “how to get started with DNSd”.

Basically, you’ll need subversion, gcc, make and most prerequisites for PHP (ICU, etc). This supposed you already have some knowledge in system administration.

The first step will be to fetch the latest SVN code somewhere in your Linux box (I usually put that in /usr/local).

$ cd /usr/local
$ svn co http://ookoo.org/svn/pinetd2/trunk pinetd2
(checkout lines)
$ cd pinetd2/php
$ ./do_php.sh
(will download & compile PHP 5.3.0, and maybe complain about missing stuff,
 just install whatever is missing, if you need help, post the last error
 lines here or contact me)
$ cd ..
$ ./start.sh
Please edit config.xml and remove the line containing this text

Arrived at this point, a file config.xml has been created and needs to be edited. I added comments inside it to help you. You’ll need to do a few things:

1. Remove the <RemoveMe> tag, and edit the <Name> tag to include your machine’s name.
2. Change the storage engine to use SQLite3. PInetd’s MySQL driver is not compatible yet with DNSd. You can copy the line from the example, but remember that if your zone file is in /tmp, it might get erased at startup (depends on your linux distribution).
3. There is an empty <DNSd> tag, fill it with the second choice in the comment (the <PeersArray> one). Change the Signature, peer name, etc to fit your needs for the Type=”control” line : this is the definition of “who will be able to create/remove records on this DNS server”.
4. At the end of the file, remove all processes that do not have Daemon=”DNSd”. There sould only be 3 remaining processes.
   If you are not root, set PortOffset to an arbitrary value greater than 1024, like for example 10000.

Once you’ve reached this point, you should be ready to go. Try starting the daemon.

[2009-02-19 10:10:33:30945] DEBUG: pinetd v2.0.0alpha running on...
[2009-02-19 10:10:33:30945] WARN: SUID security level is defined...
[2009-02-19 10:10:33:30945] WARN: Warning: Chroot security level...
[2009-02-19 10:10:33:30945] DEBUG: My name: localhost
[2009-02-19 10:10:33:30947] INFO: Loading Daemon\DNSd\UDP on port 10053, bound to ip 127.0.0.1
[2009-02-19 10:10:33:30948] INFO: Loading Daemon\DNSd\TCP on port 10053, bound to ip 127.0.0.1
[2009-02-19 10:10:33:30949] INFO: Loading process Daemon\DNSd\Process

As you can see, I am not root, so PInetd will complain about the impossibility of chroot()ing or setuid()ing, however it’s just for testing, so we don’t really care.

At this point, if you configured the DNS daemon like me with a PortOffset of 10000, with a peer named “MyPeer” and a secret of “qwerty”, running “dnsd_test.php” in the “test” directory will create an “example.com” domain.

$ php dnsd_test.php
Connected to localhost

Now, you can test it:

$ dig +short -p10053 @localhost example.com
127.0.0.1

Of course the PHP DNS Daemon is not completed yet (fixed a potential denial of service yesterday) and probably still have many bugs, so I strongly advice against using it on any production system yet.
Yes, I am using it for this blog, and for another ~200 domains, to find bugs and make the solution more stable, however if you wish to contribute by testing on a production system too, make sure you are ready to have all your websites becoming down and your server taking fire.

The next step for you is to look in test directory at the “dnsd_test.php” script, and make your own pages using DNSd. You do not need to use PHP 5.3.0 to use this class, so you can basically use it anywhere. PHP 5.3.0 is only required for the DNS daemon itself.

If you have any question, feel free to leave a comment, I’ll try to complete this article as problems are raised by people who use this.

Many people seems to have taken interest into my weird idea to write a PHP DNS daemon in PHP, so I decided to run some experiments. The first one was to setup ~200 domains to this dns server (including my blog’s one), the second one is to run queryperf against it, and against bind9 running on the same host, with the same configuration (ie. same domain list, no recursion, etc).

First, the host:

• CPU: 2x Intel Xeon E5405 (2GHz) ; a total of 8 cores
• RAM: 8GB RAM (4x 2GB DDR2 @667Mhz)
• Hard Disk: 2x1TB HDD (RAID 1, 3ware Inc 7xxx/8xxx-series PATA/SATA-RAID) ; total of 1TB usable
• OS: Linux Gentoo 64bits 2008.0 (multilib) with Linux Kernel 2.6.27-gentoo-r2

The test itself will be a 1 million random queries generated by gen-data-queryperf.py with 40% of random domains.

Some words on results

First, I’d like to say that pinetd2 is still under development, some parts are still not implemented (the DNS server is able to act as a DNS server, that’s the important part for me), and also some optimizations weren’t done yet (for example a query will always cause the same SQL statements to be run, I could prepare those).
The fact I’m running SQLite means the SQL server isn’t able to cache results (the db file might be modified by anyone, anytime, however I don’t know the exact internals of SQLite), and I don’t cache anything either.

When I started writing DNSd, I didn’t especially try to go on performances, features were importants, and realtime was too. Many improvements to speed can still be done (I’m thinking “prepared statments” right now, but also caching domains list, etc) and would help to get those numbers closer to ISC BIND.

The fact DNSd is 1/4 the speed of BIND (2531.89 queries/sec instead of 10071.2 queries/seq, my dns server is runnnin at 25.14% the speed of bind) is impressive. I guess we’ll need more tests, with different backends (MySQL is also supported, in theory) and different hosts, but I was supposing the database overhead would be bigger than that (well, SQLite is fast, but I wasn’t expecting that fast).

To tell you the truth, I am surprised by those results, however these are results on a real host, really running domains (like my blog’s domain), which makes me believe those results are the closest I could get from DNSd performances on a real host.

Now, the raw test results with both bind and PHP DNSd, running from the same host (to avoid network latency, and since I got 8 cores with almost no CPU usage as it’s morning in France, it shouldn’t make a big difference).

Other test results with other hardwares are welcome. I’ll try running the same kind of tests on less powerful hardware too, just to see what I get.

ISC BIND 9.6.0-P1

DNS Query Performance Testing Tool
Version: $Id: queryperf.c,v 1.12 2007/09/05 07:36:04 marka Exp $

[Status] Processing input data
[Status] Sending queries (beginning with 91.121.45.45)
[Status] Testing complete

Statistics:

  Parse input file:     once
  Ended due to:         reaching end of file

  Queries sent:         1000000 queries
  Queries completed:    1000000 queries
  Queries lost:         0 queries
  Queries delayed(?):   0 queries

  RTT max:         	0.605333 sec
  RTT min:              0.000035 sec
  RTT average:          0.001974 sec
  RTT std deviation:    0.002666 sec
  RTT out of range:     0 queries

  Percentage completed: 100.00%
  Percentage lost:        0.00%

  Started at:           Wed Feb 18 06:36:21 2009
  Finished at:          Wed Feb 18 06:38:00 2009
  Ran for:              99.293069 seconds

  Queries per second:   10071.196409 qps

PHP DNSd (revision 301) with PHP 5.3.0beta1 and SQLite3 (bundled libsqlite)

DNS Query Performance Testing Tool
Version: $Id: queryperf.c,v 1.12 2007/09/05 07:36:04 marka Exp $

[Status] Processing input data
[Status] Sending queries (beginning with 87.98.170.177)
[Status] Testing complete

Statistics:

  Parse input file:     once
  Ended due to:         reaching end of file

  Queries sent:         1000000 queries
  Queries completed:    1000000 queries
  Queries lost:         0 queries
  Queries delayed(?):   0 queries

  RTT max:         	0.645355 sec
  RTT min:              0.000036 sec
  RTT average:          0.007884 sec
  RTT std deviation:    0.004824 sec
  RTT out of range:     0 queries

  Percentage completed: 100.00%
  Percentage lost:        0.00%

  Started at:           Wed Feb 18 06:38:41 2009
  Finished at:          Wed Feb 18 06:45:16 2009
  Ran for:              394.961920 seconds

  Queries per second:   2531.889657 qps

One (or more) last word(s)

The test method is inspired from a link given by James Collins: “the choices for a nameserver“. While comparison can’t be done between the results there and mine (bind’s result are similar, but as said, there were problems with PowerDNS and anyway we are not running in the same conditions) it still looks like I got some chances into getting closer to be a “real” dns server, with PHP code!

Anyway, remember that “there are two sort of lies, lies and benchmarks.” (source: the previous document).

Now, I guess I have no other choice than writing documentation about “how to install DNSd” and “how to setup a DNSd slave”, that’s going to be fun (if anyone can help, I’d be happy, got a public wiki where the doc can be publied).

As the subject suggests, I just wrote an opensource DNS daemon in PHP. I already know what any sane person is thinking right now:

[18:25:06] <Derick> MT`AwAy: you’re mad

Now that it’s said, let’s move on ; if you want to tell me I’m crazy, you can post it in a comment here, it makes me happy. I had some reasons to dislike bind9 which finally made me write my own DNS daemon, and I’ll explain that here.

My need was to have a stable dynamic DNS server working in most environnements, with an easy to configure master/slave relationship (with realtime synchronisation), and a way to change records instantly from PHP…

For those who already played with bind9, there’s a feature called dlz which basically allows to ask bind to get data from an SQL server. I could just configure another MySQL per slave, and put replication there, but it’s not that fast and I had some stability issues (both with MySQL replication, and with bind crashing in some weird cases).

So, instead of trying to fix bind9′s code (which would also include fixing MySQL replication – or trying newer MySQL’s row-based replication) or searching for another solution (there are zillions of dns servers around there, but they all have features I’ll never need), I decided to write my own DNS server (built on top of PInetd, my PHP networking framework for PHP 5.3.0), with only the features I needed.

So here are some of the features of this DNS daemon:

• Supports RFC 1035, and some others too (IPv6 AAAA records, and DNS OPT, with the goal of supporting DNSSEC at some point).
• Does NOT support AXFR nor IXFR, I have no need for the standard zone transfer protocol
• Data can be updated realtime with a provided client class, connecting to the server via a shared secret authentication (client computes checksum of his name, timestamp and secret, server replies with the same kind of data). Once authentication is finished data is sent cleartext, but adding SSL encoding wouldn’t be that hard (just need to create a “STARTTLS” function, that’s on the TODO list).
• Slave stays connected to master (keepalive packet sent every 15 minutes), and gets updates realtime.
• On first connection, slave will get all zones/domains/records from the master. If it gets disconnected later and reconnect, it will search for his last update, and will ask only for newer data from the master.
• Slave synchronisation is done in a separate process, meaning that even when processing a lot of updates, service is available.

As a test, I ran AFNIC’s zonecheck on a virtual zone I created on the server, and it works!

dig @dyndns1.ookoo.org version.dnsd ch txt

If you want to test this, I got a test domain with a record creation page, however I won’t post it here (or it might get abused) so if you want to test, feel free to contact me.

And finally the conclusion is simple: php can do anything you want to do, and even things you never wanted to.

PS: I’m looking for someone to look after PInetd‘s website, like removing this lipsum news, etc… (I don’t have time for that).

Ever wanted to give meaningful names to your processes when you pcntl_fork() with PHP ? proctitle is the extension you’re looking for!
Adapted from bug report #29479 and code initially wrote by Midom for Wikipedia, the proctitle extension allows for a process to change its own displayed title in the system’s process list.

Is the ability to rename a process relevant to PHP? I’ll just say that an image is worth a thousand words…

So, here it is, my good old PInetd project is soon under public beta (as soon as PHP 5.3.0beta1 gets out), and, I hope, will have people using it. PInetd is basically a framework for creating TCP, UDP or other kinds of daemons. It has a lot of features (internal IPC, fork, chroot() and setuid() helpers, process management, etc) and contains some example daemons (FTP, POP3, SMTP, DNS, HTTP, …). I think it may give ideas to a lot of people and open new possibilities for PHP.

Anyway, tonight’s goal was to make it easier to see what’s happening on pinetd without too much troubles, and allow display of some basic informations with some daemons (for example for FTPd, display name of file currently being downloaded/uploaded, etc). So, the proctitle extension is relevant to PHP!

You can get it via SVN: http://ookoo.org/svn/proctitle/ (tested with PHP 5.2.8 and PHP 5.3.0alpha3).

Finally, starting tommorow evening (if we’re lucky the new server will be ready by this time), pinetd2 will go in production mode on ooKoo.org

This is a big step toward debugging, and will help me to kill most existing but yet unknown bugs in pinetd2.

Remember that pinetd2 is multicompatible. When updating, tables structures will be updated, but if downgrading they will be reverted to their original state.

As you may know, I’m currently rewritting the whole pinetd project to a new project called pinetd2.

Right now, we finally have partially reached the functionnality level of pinetd (being able to receive mails, and check them with pop), and now IMAP4 is also working (you can receive mails via imap, but not yet search and do nice stuff imap allows).

pinetd2 is still missing a MTA (this should be a matter of days) and more imap4rev1 functions (this is a matter of one or two weeks). Once this is done and once PHP 5.3 is released, pinetd will be officially OBSOLETE, and pinetd2 will become the new officially supported version. Please note that extra attention has been ported to make pinetd2 compatible with pinetd, without even having to run an update tool. Just extract pinetd2 to a new directory, copy your config file from pinetd, and run it. Your config file will merged in the XML config used by pinetd2, and you’ll have to check it a last time before running the daemon for real.