الْحَمْدُ لِلَّه, Fedora 16 has been released!  You can read the full release notes or just go straight to the torrents for downloading.  See the release announcement after the fold…

The Fedora Project is pleased to announce the release of Fedora 16 (“Verne”). For detailed information, see the release notes:

http://docs.fedoraproject.org/en-US/Fedora/16/html/Release_Notes/

Fedora is a leading-edge, free and open source operating system that continues to deliver innovative features to many users, with a new release about every six months. Fedora 16 brings exciting new features for desktop users, for system administrators, and for developers. Highlights of these features can be found below. If the mere mention of Fedora 16 is all you need, download options can be found at:

http://fedoraproject.org/get-fedora

Dedication

During the preparation of Fedora 16, the computing world lost one of its great contributors: Dennis Ritchie. Ritchie co-invented Unix and the C language. He also co-authored The C Programming Language, a book that taught many programmers just at the time personal computing was exploding. Without Ritchie computing would be nothing like it is today.

A humble man, not well-known outside his field, Dennis will always be remembered by those of us who practice the craft. Thank you, Dennis.

What’s New in Fedora 16?

For desktop users

A journey to the center of the desktop:

• GNOME 3.2, the latest update to the most popular desktop environment, brings new features, including:
  • System Settings gains an “Online Accounts” panel, which provides a central point for managing online accounts like Google, Facebook, etc.
  • A new contact management application is integrated with Empathy, Evolution and the new “Online Accounts” settings panel
  • A new document management application provides a simpler alternative to traditional file management for both local and “in-cloud” documents.
• KDE 4.7, the most recent version of this feature-rich desktop environment, brings substantial updates, including:
  • DigiKam 2.0 adds face detection and recognition, geotagging and more.
  • An updated Plasma Workspaces window manager (KWin) makes KDE better suited for mobile devices, and also adds improvements for desktop users.
  • A new shutdown dialog allows users with multiple operating systems to select the OS to boot next.

For developers

Twenty thousand lines in C:

• Perl 5.14, a new version of Perl, brings many enhancements, including Unicode 6.0 support.
• GCC Python Plugins extend GCC with Python 2 and 3, without dealing with the C internals of GCC.
• D2: Fedora 16 is the first Linux distribution to include the newest version of D, a systems programming language combining the power and high performance of C and C++ with the programmer productivity of modern languages such as Ruby and Python.

For system administrators

Around the world in eighty columns:

• GRUB2: This leap forward allows better configuration options and better support for non-x86 architectures.
• New system account ID numbering: Fedora 16 starts user IDs at 1000, providing more room for system accounts and making it easier for administrators not have services run as root. This improves interoperability with other Linux distributions that start user IDs at 1000.
• Chrony: Chrony provides network time protocol (NTP) client and server pieces which are more tolerant of unstable clocks and Internet connections which are not always on.
• ext4 driver mounts ext2 and ext3: Fedora 16 uses the ext4 driver to mount ext2 and ext3 file systems, reducing the size of the kernel code.
• Improved virtualization tools:
  • Fedora 16 provides improvements to virtual networking, making large deployments easier.
  • Virt-manager guest inspection is a unique-to-Fedora tool allowing read-only access to guest file systems, applications, and Windows registry.
  • Fedora 16 provides locking of virtual disks to prevent a disk being used by multiple virtual machines simultaneously.

This is only a taste of what is included in Fedora 16. A more detailed list can be found at:

https://fedoraproject.org/wiki/Releases/16/FeatureList

Alhamdulillaah, I’ve discovered a new WordPress blogging client packaged with Fedora – namely, Lekhonee. There aren’t a lot of formatting options, but it appears you can post using HTML, so that’s a plus, I think. Let’s see how it goes.

Alhamdulillaah, the Fedora Project has released Fedora 11!  You can get the torrents and/or check out the new features!

(Sorry for the rather uninspired & lazy post…I usually like to share more details, but I’ve got work to do…;) )

Alhamdulillaah, I’ve successfully loaded the Fedora 11 Beta Live CD onto an SD card.  This allows me to boot directly from the card using my USB card reader, much as if it were a USB key/flash drive.  In short, after a few attempts, everything just works.

The few attempts mostly related to loading the Live CD image onto the SD card, as my flash memory reader is somewhat fussy and I was experimenting with different filesystems – my preference being to use a free software filesystem before using one of Microsoft’s.  I was able to do this using the ext2 filesystem (no need for journaling on a Live USB key, right?), as I have no need to store anything else on it.  Incidentally, I created an overlay image for persistence, so any changes made to the system while running should be saved and persist between reboots.  We’ll see how that works.

One of the driving factors for getting the Fedora 11 Beta Live image working is because I want to give my father, who just recently purchased a new netbook (Lenovo), a nice tour of a recent version of Fedora.  A Live USB version would be ideal for this.

Update: I’ve since gone ahead and installed Fedora 11 Beta on my laptop, as well (from the DVD install, though).

]]> http://blog.basilgohar.com/2009/04/06/fedora-11-beta-livecd/feed 0 http://blog.basilgohar.com/2009/03/10/my-first-fedora http://blog.basilgohar.com/2009/03/10/my-first-fedora#comments Tue, 10 Mar 2009 08:01:07 +0000 Basil http://www.basilgohar.com/blog/?p=424 Bismillaahir Rahmaanir Raheem

Alhamdulillaah, to my pleasant surprise, I discovered that I started using Fedora personally with Fedora Core 4, and not 5 like I had previously thought.  This makes me feel better as I’m glad I started using Fedora earlier, because I like to think I recognized the importance of a completely (or as close as possible) free operating system (free here means freedom, not just free of cost).

At the time, I was content running it as a dual-boot system and just playing around with it.  By the time Fedora Core 5 came around, though, I was giving serious thought to using Fedora (Core, at the time) as my primary OS.  The complete transition happened, I believe, with Fedora 7 (which is also the year that Fedora Core & Fedora Extras combined, creating…Fedora!), where I relegated Windows XP to just being a spare OS for the very few things it does that I am unable to do with a completely free operating system (yet).

My experience with Fedora Core 4 was overwhelmingly positive.  I am sure I was impressed by the sheer amount of Free Software that was available for it as well as the elegance of a system that was not originated in Redmond.  I was and still am very pleased with the way Gnome works, and that was a big factor for me at the time as it is now.  I do wonder, though, if I downloaded & installed Fedora Core 4 today, what my experience would be link….hmm…

I remember seeing Fedora first in use as Fedora Core 2, when I was surprised someone was using it, as the name “Fedora Core” made me think it was the core of an operating system, but not the complete one.  I don’t know how or why I picked that up, but it is likely related to the perception that Fedora is not really meant to be used, and it is just a development platform (a blatantly false accusation).  I do know that I heard of Fedora Core when it came out, but again, my perception was likely incorrect at the time, thus causing me to adopt using it until quite late.  I had been using different GNU/Linux distributions since I was a freshman in college (can anyone say 1999?).  At the time, I recall trying “Corel Linux” – yes, I…don’t know what I was thinking.  I do remember it being based on Debian, though.  I also remember it significantly for a large amount of disk activity.  I don’t remember much else other than me deleting it shortly afterward.    Other flavors of GNU/Linux I tried were Red Hat, and that’s about all I can remember now.

So, anyway, this might be a good idea for a meme.  My first Fedora.  Talk about when you first started using it, why you did, and how the experience was.

I have the bad habit of starting things and not completing them.  This is bad for numerous reasons, just a few of which I’ll list shortly.  I will warn you now that as this post is meant to help me organize my thoughts, it will list-heavy, as I do find lists and other types of grouping structures to be beneficial and help in making relationships (e.g., database schemes).

I already know that as I type this I will revisit this post in the future as I will be unable, in one sitting, to be think of all of unfinished projects.  Therefore, I will come back and add to the list as I recall more things (and find the time and desire, a rare combination these days, to update the blog).

For

• Unfinished projects tend to clutter your mind
  • I tend to turn to unfinished projects whenever I work (i.e., get paid to do something) and hit a stumbling block.  This severely hinders my progress on work, which ultimately leads to less getting done.
  • Time when I could be brainstorming solutions is usually spent wondering about these myriad unfinished projects, which, lacking much focus, tend to yield few to no tangible solutions.  This includes walking/riding to the masjid, showering, and other activities that tend to leave the brain free and, ideally, well-suited for addressing beneficial thoughts.
  • If I try to focus on something, I can have a hard time preventing my mind from wandering to these unfinished projects.  This is related to the point about work above.
  • It’s just not healthy to have so many things on the mind at once.  There is a point at which the benefit of multitasking is overshadowed by the overhead of just trying to keep track of everything.  It’s not efficient.
• Having so many unfinished projects means that it can be hard to pick just one to work on.
• Playing catch-up may give a false sense of achievement, because the value of these unfinished projects are not equal, and therefore, I may complete one that is really not that important while one that is more valuable was never completed.  Therefore, just trying to clear out the backlog of projects may yield a net result of close to zero.
• There are many more, and I may amend this list in the future as I recall them.

So, here is the list of my unfinished projects (at least those that I can recall at one point or another).  I am including some thoughts on the different ideas as well.

• Redesign the Audio Islam website
• Write a script to automatically tag all the audio files, especially the ones containing Qurʾān recitation, with reciter & sūrah information
• The Fedora Documentation manpage coverage project
  • This is basically an idea I had related to the Fedora Documentation project to ensure that all executables had an associated man page (manual page, a low-level documentation system for Unix-like operating systems such as GNU/Linux).
  • The first step in this is to canvass the existing packages & executables and see what already has a man page.  I have already started on this, but there is much work left to be done.
  • There are plenty of sources for documentation that can be used as baseline manpages, including the Debian project.  So, this project doesn’t require as much work as it does just coordinating and facilitating.
• A Paltalk replacement
  • Paltalk is a voice conference & chat application (video is also supported, but I don’t care about that right now) that is widely used, but terribly proprietary.
  • My goal is to create or synthesize a replacement for Paltalk using only free software and open technologies
    • The best candidates for this are SIP (for multimedia) and Jabber/XMPP (for text chat).
    • The landscape for software that can do this in a simple fashion is rather bleak, but there are some solutions.
    • I’m currently focused on FreeSWITCH as the SIP server and ejabberd as the chat backend.
    • Linking them such that it will be one service will be quite challenging, and still requires a free software client that has solid support for both SIP & XMPP
      • I’ve been looking mostly at Qutecom, SIP Communicator, and Psi
    • • On GNU/Linux, the Empathy client for the Telepathy framework is outstanding, but, is limited to free-software operating systems.  I think Windows support is a long way out still.
• AlMaghrib in Malaysia
  • I am trying to catalyze the establishment of AlMaghrib Institute seminars in Malaysia (almost surely in Kuala Lumpur, at least at the beginning).
  • I’ve already purchased the domains almaghribinmalaysia.com/net/org, and I just need to get the site setup.
    • To start off, I think just a blog with some information about AlMaghrib is fine.  It can be fleshed-out later.
  • Alḥamdulillāh, there is already a lot of interest both within and without Malaysia.  I have several contacts here that are also excited and may be able to facilitate the first few steps.
    • I see the first step being getting a group of excited & enthusiastic locals to get behind the effort.
    • After that group is identified (and it is really an ongoing process), see about getting the support of existing organizations.
      • From an initial perspective, it seems like this is going to be easy, in shāʾ Allāh.
    • With this support, we will want to schedule some kind of event to gain a wider audience interest.  Something like a conference or teaser seminar, perhaps a week-long program with some of the AlMaghrib instructors.
      • Many of the AlMaghrib instructors are already known or well-known here
      • Having Suhaib Webb might be particularly nice, as his wife is Malaysian.
      • Waleed Basyouni has expressed an interest in coming to Malaysia, and some work is already in progress regarding that, but it may have stalled (temporarily only, in shāʾ Allāh)
      • Yasir Qadhi is another instructor that has some reknown here, apparently (from what I’ve been told).
• HidayahTech
  • My own technology consulting business
  • Although HidayahTech is currently the name of my blog, I have also resolved to use this as the name for my own consulting business.
  • Services will include
    • Audio recording, editing, mastering, and production (I did this, for example, for the Texas Dawah Convention in 2007)
      • I am consider splitting this and other audio-related services into it’s own business, namely, Hidayah Audio
    • Web technologies
      • Hosting (this is generally quite easy)
      • Moodle (an open-source learning mangement system, for putting schools online, for example)
      • Audio services, including hosting my “replacement for Paltalk” solution above
    • Consulting
      • This will include a wide variety of services ranging from system administration, general IT, infrastructure, to web development
      • I may take some partners or hire some friends for this aspect of this business, as I will need help for some points of it
• Global Homeschooling
  • This is my premiere project, if I had to pick one.  I consider this to be something great if I can achieve it, by the Permission of Allāh.
  • To sum it up, I want to develop a system whereby Muslims (and truly, anyone) can homeschool their own children easily.  Homeschooling may be too specific of a term, though, because I want the scale to include a plan that can be implemented by governments.  I would like children, and really anyone, to be able to be educated within their own homes, utilizing the bevy of technology that exists.
  • Technology creates options & opportunities, and this is a belief I take strongly.  Therefore, I think we should utilize it.  I see education as one of the most important investments that can be made, whether it is Islamic or related to the worldly sciences, both of which are important.  I actually believe the distinction to be artificial, but it will take time to get there.
  • So, to put it in a few words, I would like to develop a system of education leveraging technology that is globally accessible allowing education to be released from the confines of school buildings.
• Learn the Python programming language
  • I think learning Python has a high return value on the types of applications I would like to write, including graphical ones.  Also, it is quite the preferred language for many platforms, not the least of which is Red Hat & Fedora.
• Memorize the Qurʾān
  • Please keep in mind that this list isn’t in order of importance…;)
  • I think this is a goal every Muslim should have.  I started this goal, seriously, while I was staying in Egypt with my parents, even semi-regularly visiting the imān of the nearby masjid to review my memorization.  Sadly, since arriving in Malaysia I’ve not really resumed, and I am afraid I may start regressing.  This is the kind of project that can be achieved through a little, regular & constant work every day.  I really should commit to something, even if it is as simple as one āyāh per day.  That would be better than none.

Alhamdulillaah, I’ve just managed to produce my first HDR (high dynamic range) image the results of which I’m pleased with.  Once again, I have CHDK to thank for enabling this feature easily.

Note:  I begin this post with a mildly detailed explanation of the human visual system, HDR & LDR images, and the process to create HDR & tone-mapped images.  The pictures themselves are near the bottom of this post.

First, a little background about HDR.  The way our bodies capture and process imagery is, to put it lightly, amazing.  Starting from when light first hits the cornea through to when we perceive the visual object (not to mention how our brain’s store such visual images, which is far from just keeping a “bunch of bits”), the Words of Allāh resound with truth regarding His Creation:

And it is He who spread the earth and placed therein firmly set mountains and rivers; and from all of the fruits He made therein two mates; He causes the night to cover the day. Indeed in that are signs for a people who give thought. (Alqurʾān 13:3)

Thus, in the way that they are created, our eyes can capture an amazing amount of detail and range in their own way.  The range of regular (so-called “LDR” – low dynamic range) images is quite limited in comparison.  And this is, actually, the vast majority of image-related technologies, be they cameras, display devices, or even image formats such as JPEG.

HDR images, however, are those which do or have the ability to capture image details, specifically color & brightness information, at a much higher level than normal technologies.  Since normal technologies, such as our digital cameras and screens, can not capture nor display HDR images, we have to take a different route to get the information.  One technique is exposure bracketing where one takes a picture of the same scene at different exposure levels, one after another.  The reason for doing this is that, while the normal dynamic range of a camera may be unable to capture the bright, medium, and dark details of an image all at once, it does have the ability to capture these details separately with different settings.

Alhamdulillaah, there is also software that exists that allows you to take these separate LDR images and combine them into one now-HDR image that contains the details from all the separate exposures.  Doing so is not trivial, and it may take one multiple attempts to get something that looks good.  More often than not, the result is eery, because of a technique called tone-mapping.  Tone-mapping is a technique whereby the colors and range of an HDR image are brought down into a simulation of what our eyes would perceive in the scene.  In reality, that’s all but impossible, but it yields some interesting, beautiful, and sometimes striking results.

The specific software I use is the unpronounceable Qtpfsgui (I just say it one letter at a time).  Qtpfsgui is a front end for pfstools, which is an outstanding image toolkit for these kinds of images.  It takes care of aligning, merging, and finally tone-mapping the images one step at a time.  I short-circuited the first two steps (because Qtpfsgui can be fussy sometimes)  by using the command-line tool align_image_stack to align the different exposures (since I moved a bit between exposures) as well as creating the HDR image from the three separate images.  Then, I used Qtpfsgui just for the tone-mapping stage.

The first row of pictures are the original images, taken at -4, 0, & +4 EV settings.  Looking closely, you’ll notice that each range brings out different details: the darkest shows details in the clouds, the middle shows details in the sea, and the brighest shows details in the sand.  These are the separate LDR images at different exposures that I talked-about above.

The second “row”, containing only a single picture, is the result of the several steps I mentioned above to produce an HDR image.  I make no claims into how “real” this image looks, but it does show the details that I was looking for – from the clouds to the sand – in one image.  I also think it looks cool. 

This is the first HDR image I’ve created that I’ve wanted to share, and so I took this opportunity to explain the process too.  If you have any questions, then by all means, please post a comment and then subscribe to the comments so you’ll know when I’ve replied.

Note: This is for future reference as I try to make some progress on an old project.  I do not think the last value is a default for Fedora, but rather, something added later.  If someone knows otherwise, please correct me!

Update: I’m starting to dislike fixed-width templates…corrected the last $PATH value so it will fit on my blog without overflowing into CSS “Elsewhere”.

Update 2: Ahhh!  WordPress!  WYSIWYG, not WIWIWYG (What I Want is Why You Get)!

Note: This is a post I started writing back in November, but only got around to updating now.

Alhamdulillaah, I cannot believe I’ve lived for so long without knowing about screen (Wikipedia article).  To put it as simple as possible, think of screen as a “tabbed” interface for terminals/shells.  This is a weak analogy, however, because it enables so much more than tabs do for browsers, for example.  Onwards for more details.

History

I had first heard of screen through references and then through some of my friends, but I didn’t give it much heed because it seemed like something too complex for something so simple as a terminal application.  Little did I realize that, like most applications with their roots in Unix, it’s both simple and extremel useful.

Invocation

screen is invoked by typing it’s name.  It will appear initially as though you’ve reset your terminal, but this is just a new screen session, which is effectively a new login. Each screen window (which I will explain later) is independent and has it’s own scrolling buffer, independent also of your own terminal’s buffer. This means that screen as an application stores previously displayed text on the machine on which it is ran, so if you’re running it remotely, it won’t affect your local machine.

Basic Usage

Commands in screen are almost always prefixed by ctrl-a.  For example, creating a new window in screen is “ctrl-a c”.  Switching to the next window, if you have more than one, is “ctrl-a n”.  Previous is “ctrl-a p”.  screen windows can be exited by simply typing “exit” as if they were any other terminal session.  When the last screen window is exited, screen itself terminates.  Screen windows can also be killed by typing “ctrl-a K” (note the capitalization of “K”).

It’s also easy to jump to windows 0 – 9 by typing “ctrl-a” followed by the window number.  So, for example, if you want to jump to window 3 of your screen session, just type “ctrl-a 3″.

Detach and resume

The real beauty is using screen is its persistence.  Chances are, if you’ve used remote login shells frequently, you’ve been disconnected while working.  Most shells will respond by terminating whatever was actively running in the foreground of that terminal.  This means your vim session was interrupted.  It also means whatever service you had started is probably dead by now.  screen takes care of all of these things, because it behaves as a persistent application.  Thus, whenever you disconnect from the terminal, screen stays running, and it allows you to resume a previous screen session with the command line option “-R”, which means reattach an existing screen session, else start a new one.  Additionally, the “-d” argument might be useful, because it will detach the session from an existing login, in the case it is still attached elsewhere, such as a remote login from another location that you’re no longer using.

For example, if you are logged-in to your server from home with an active screen session, and then go to work, calling “screen -d -R” will detach the screen session from your home login and reattach it, where you left off, to your login from work.

Another advantage is that screen effectively allows you to make a daemon (background running process) from any command-line application, including those for which it normally isn’t possible.  An example of this would be rtorrent, which is a torrent downloading application normally meant to run in the foreground.  I personally like to start an rtorrent process running in one screen window and just leave it running.  Then whenever I login to that server and attach the screen session, I can check on the status of my torrents very easily by ctrl-a p/n to the appropriate window while doing other work.

Conclusion

So there you have it!  I realize this is a very introductory article with regards to screen, but it should be enough to get your started.  If you run into any problems with it, just type “ctrl-a ?” (that’s a question mark) and a friendly window will pop-up explaining, albeit briefly, all the different commands and what they do.

For a long time now I’ve been wanting to setup an LDAP back end for various Hidayah Online Network resources, especially the Columbus Dawah project.  This has become particularly pressing as the Columbus Dawah website has three primary public services – the blog, the forums, the school.  Additionally, in the future I want to roll out webmail (publicly) for volunteers & administrators within the project.  The problem with running these disparate services is that they all have their own authentication built-in, which means someone needs to register on each site independently to access any services that require one to be logged-in.  Therefore, a system that would unify user authentication for all services (i.e., LDAP) would save a lot of hassle for both the users and myself.

Native support

Naturally, the first challenge is preparing the different software packages that run the various parts of the Columbus Dawah website for utilizing LDAP for authentication purposes.  The relevant bits of software are WordPress for the blog, phpBB for the forums, & Moodle for the school.  Additionally, I will use RoundCube for the webmail, but since it will act as an LDAP client, it will not need any special setup on it’s own – the back end IMAP e-mail server (Dovecot) will be the one that interacts with LDAP.  That will be described in another post, in shaaʾ Allaah.

Of these mentioned bits of software, it appears that only WordPress comes without some kind of native LDAP support.  While this will be a challenge moving forward, one plug-in exists that at least offer some support, but I’m not sure if it’s updated for the latest version(s) of WordPress.

Structure

Although I haven’t finalized it, there is definitely the hope that I can use LDAP for authorization as well as authentication.  While authentication would allow me to let someone login to the site, authorization would allow me to give different users different levels of access.

For example, let’s say we have a simple organization that has three “ranks” of members – member, volunteer, & officer (this is roughly based on the plan I am working on for Columbus Dawah).  Furthermore, we want to grant different levels of access to the site’s resources depending on a user’s rank.  For a regular member, we would like to allow to him to login, post to the forums, post comments on the blog, and enroll in courses in the school.  For a volunteer, we would like to add to that the ability to have an e-mail account.  Finally, for officers, we would like them to have access to make blog posts and create & edit courses in the school.

Authentication alone would only allow us to grant or disallow access site-wide using one account.  Authorization gives us that additional feature of granularity to allow access to some or all of the site based on permissions.  LDAP supports all of this, but it is up to the underlying software to enable such features through LDAP authorization in a way that makes sense for the application.  As all the aforementioned applications have a concept of groups or roles with different sets of permissions or capabilities, and as I intend to utilize such features as the project grows, the ability to authorize users as well as authenticate them is tantalizing at the very least, and down right essential the more I think of it.

Fedora Directory Server

The specific LDAP server I intend on using is the Fedora Directory Server (website).  First and foremost, naturally, it’s open source.  And, the fact that I’m choosing an LDAP server that is related to the Fedora Project really should come as no surprise to anyone with even the slightest familiarity with this blog.

However, FDS is also quite an advanced LDAP server, supporting a slew of enterprise-level features while, according to most accounts, being reasonably easy to work with.  Therefore, it is quite a good choice, and may very well be the best one.  Time will tell how close to the truth that statement is.

What’s next?

So, what’s next?  Well, the most glaring obstacle right now is that I have zero experience working with and/or configuring an LDAP server, so I’m going to start by just playing around with the protocol and see if LDAP will work as smoothly as I hope it will.  If I feel comfortable enough with that, then my first foray into LDAP will likely be to configure my new e-mail server to use it as a backend.  That should prove to be quite interesting, moving forward.  So, stay tuned and I’ll do my best to keep things updated!

I have my new server (aalimraan.hidayahonline.net – the one hosting Audio Islam) setup to log web accesses each day to its own file which is then bzip2-compressed.  This is convenient for a variety of reasons.

• Firstly, I can easily access statistics for each day by processing the appropriate file.
• B, I can see the relative activity on each day at a glance by seeing the size of the file, keeping in mind it’s only a rough estimate since compression can skew the results (e.g., many requests for the same, exact URL may compress much more than fewer, disparate URL requests, resulting in a smaller file and a seemingly less active day)
• 3, by being compressed text files, they take up very, very little space.

The fact that the files are compressed using bzip2 means they are relatively tiny (by an order of magnitude compared with their uncompressed forms).  On top of this, I can easily access their contents using bzcat, which simply decompresses the files on the fly, allowing me to redirect the uncompressed text stream to whatever utility I’m hoping to process them with – such as grep, wc, or whatever.

So, I wanted to download the files in bulk to my laptop at home so I could write a PHP script that will process the access logs and store them in a database so I can extract all kinds of goodies from them such as which sites link to Audio Islam the most, which URLs are the most popular, and so on.  So, rather than download all the separate files individually, I wanted to concatenate the log files created thus far into one larger file so that I could just download it in one go.  Additionally, several text files concatenated together would naturally yield a smaller file (at least, so one would assume).

So, I ran the following command (or something like it):
cat access_log*.bz2 | bzip2 -c | ~/aalimraan.hidayahonline.org-access_log-20081226.bz2
The intention being that I wanted to create one large bzip2-compressed file as the concatenation of all the other, smaller daily files by first decompressing them into a continuous text stream (as if it were one large log file) and then recompressing them into the larger, single file.  But when I got around to processing that file on my local machine, the uncompressed output was garbage!  I was surprised, and then an idea hit me – I decided to run the uncompressed output through bzcat once again (the idea being that I am now decompressing it twice).  Lo and behold, the output of that invocation was something that looked astonishingly like a web server access log!  So what happened?

Look closely at my command, and you’ll see that I made the quite silly mistake of using plain old

cat

instead of

bzcat

.  Thus, I was recompressing a stream of already bzip2-compressed files, which is, to say the least, quite pointless.  In fact, the resulting file is all but useless to me, because I cannot really tell where one file begins or ends, so I only get output from the first bzip2 file.

Needless to say, let this be a lesson to make sure, when piping data around, you know what you’re doing.

This has been a public service announcement from you local system admin.  Thank you for listening!

Alhamdulillaah, while waiting for an answer an #fedora for the question of where the custom spellcheck files for hunspell, the unified spellchecking engine for Fedora 9 and upwards, reside, I discovered that they are stored in .config/enchant, which isn’t very intuitive, but makes sense after discovering that enchant is related to the Fedora hunspell integration.  Click to see the link that lead to this discovery.

Alhamdulillaah, I’ve (finally!) installed Fedora 10 on my laptop, but it’s not been without its bumps.  I cannot blame this on Fedora 10 nor the anything else related to the release.  But, in the end, I don’t have a DVD to install from, only the LiveCD (which I booted from my SD card reader), and that left me without a lot of packages and also without Ext4!  And now, after having copied my Evolution profile directory over, I discovered that the new version does (2.24.1) does not supported the “unmatched” search folder feature which I had come to rely on in recent weeks.  This is a real bummer as it all but nullifies the advantage of using search folders (for me, at least) in Evolution.  Alhamdulillaah, at least there’s a bug report open with one developer sympathizing.  However, a fix for it may not be forthcoming until the next release.

I think the time has come to take a look at Thunderbird again…

Alhamdulillaah, Fedora 10 has been released!  Torrents are available.  I have all of my servers seeding now (including the newest one).  Read up on the new features to find out what’s new.

Well, what are you waiting for?  Just go download it already!

If you don’t want to use the torrent (although you should), then you can also download both the x86-64 build as well as the i386 one from my new server.  My server should be able to deliver the files as fast as you can grab ‘em, in shaaʾ Allaah.  Let’s find out. 

(Famous last words, eh?)

I had really wanted to seed the initial few hours of the release on my new server because of its great amount of network bandwidth, but that’s an opportunity I lost because of a crash like this:

Caught Segmentation fault, dumping stack: KB] [Port: 6924]                                                                                                                                                                                                                   [U 76/0] [D 0/0] [H 0/32] [S 3/86/768] [F 4/128]
0 rtorrent [0x41ea5d]
1 rtorrent [0x421488]
2 /lib64/libc.so.6 [0x3a76a322a0]
3 /usr/lib64/libtorrent.so.10 [0x3a78e4caf4]
4 /usr/lib64/libtorrent.so.10 [0x3a78e50683]
5 /usr/lib64/libtorrent.so.10 [0x3a78e4f704]
6 /usr/lib64/libtorrent.so.10(_ZN7torrent9PollEPoll7performEv+0x5d) [0x3a78e24d55]
7 rtorrent [0x440379]
8 rtorrent [0x41f645]
9 /lib64/libc.so.6(__libc_start_main+0xfa) [0x3a76a1e32a]
10 rtorrent(_ZNSt8ios_base4InitD1Ev+0x49) [0x40cc79]
Aborted

that kept happening.  Up to this point, I have never had a problem with rTorrent, alhamdulillaah, but still – that one time you really need it, and it goes?  The worst part is, when searching the problems I was having, it was due to memory consumption.  4GB is not enough to seed a couple of hundred peers?  I could seed almost 100 on my older hardware with only 1GB of RAM, and I never had the mainline client (which is written in Python) give me such problems.  rTorrent is written in C++, which I thought would help with these kinds of cases.  Shouldn’t, theoretically, a compiled application require less memory than one running within a virtual machine environment — maybe my understanding on computer science is lacking, but I always thought that was the prevailing wisdom.

Anyway, I didn’t want to turn this into a rant about rTorrent, but the emotions are strong enough to drive me to at least say something, especially because the crash happened during (my) nighttime, so I couldn’t restore the seeding until after I woke up and got back from fajr prayer.

If anyone has any recommendations on how to avoid the problem I had with rTorrent or on a better, more stable client that can handle such massive transfers and load, please let me know.

Alhamdulillaah, following a recommendation from a close friend, I decided to start leasing a server from 10TB.com for Audio Islam.  As the name implies, they are big on bandwidth, which is exactly what Audio Islam needs.  Needless to say, I’m thrilled at having a new beefy server to play with and to meet the current & future needs of this food of the Islamic audio-consuming masses.  And the best part is they installed Fedora for me!  How awesome is that!

Read on for the specs & details on the transition.

The plan I got (accurate to the time of this posting, at least) is really impressive, maa shaaʾ Allaah.  The catch for all of this value is minimal support.  Alhamdulillaah, with more than 5 years of managing GNU/Linux boxen, as long as I have a remote reboot, I think I’ll be fine, in shaaʾ Allaah. 

Hardware specifications

The specs for the new server:

• Single Quad Core Xeon 3220 – 2.40GHz
• 4GB of RAM
• 250GB of disk space
• 1Gbps (!) network connection
• 10TB monthly bandwidth limit
• $199/month

The specs for the old server were quite modest in comparison:

• 2.4GHz Celeron (single CPU, single core )
• 1GB of RAM
• 80GB of disk space
• 20Mbps network connection
• No monthly limit
• $239/month

Bandwidth

It should be quite obvious to see why I had little hesitation in the upgrade.  A 20Mbps connection without a monthly limit can yield a theoretical monthly throughput of 20Mb/s * 3600s/hr * 24hr/day * 30day = 6.48TB.  Interestingly enough, last Ramadhaan, Audio Islam did nearly 6TB – so it was just the right time to move on as well.

Of note is that it may be that the connection on the old server might have been limiting the speed of downloads at the peak times.  Therefore, with a 1Gbps connection (that’s fifty times faster than the old server), we may just see some surprises in the coming months.  I may have to instill some artificial bandwidth throttling if I do not have another solution (say, a second server).  We’ll just have to see…

Fedora

The most exciting part for me is that they were more than happy to install the latest release of Fedora (9) for me, which is the same operating system I have running on my laptop.  Fedora is a great distribution for running a server on, and it makes me want to help out with the proposal for a Server Special Interest Group (SIG) within the Fedora Project, in shaaʾ Allaah.

Some people may hold the opinion that a server distribution should be on a more stable platform than Fedora, because Fedora is a cutting-edge distro, running the latest releases of software generally.  I won’t argue that point, but I will also say that running the latest stable release of software brings advantages that are not available to someone running older releases of software, especially server software.  Being that Fedora has a huge & helpful community around it with many already using Fedora in a server configuration, any problems encountered will also be well supported by others like myself, in shaaʾ Allaah.  So I am not concerned in running bleeding-edge stable software on a server, which is just what Fedora provides, alhamdulillaah.

If all goes well, I may have reason to get a second server from 10TB for some business I am undertaking, in shaaʾ Allaah, and with Fedora 10 scheduled to be released in a few days, in shaaʾ Allaah, I am looking forward to running it as well.  Crazy?  Perhaps.  But I’m loving it!

Cacti

Wait, what?  Yes!  My same friend referenced above also showed me Cacti running on his server.  I immediately feel in love with it and got to work setting things up.  See for yourself.

To put it simply, Cacti is a PHP/MySQL-based tool that, when running, monitors & produces information graphs for servers, mostly communicating through SNMP.  I have it setup for my three current servers (soon to be two, in shaaʾ Allaah).

Server naming scheme for the Hidayah Online Network

Incidentally, with the growing number of servers within the Hidayah Online Network I decided that I needed a naming scheme.  My oft-referenced friend elected to use the names of cities of importance within Islaam – so he named them makkah, madinah, etc.  Another one of my friends has opted for the names of historic battles in Islamic history – so badr, uhud, etc.  After consulting with my wife, I decided on something even better – the names of the chapters (suwaar) of the Qurʾaan – so my first server (formerly just called “The Hidayah Online server”) has now become alfaatihah.hidayahonline.net.  My second server (formerly just called “The Audio Islam server”) is now, for the remainder of it’s life within the Hidayah Online network, albaqarah.hidayahonline.net.  And, the newcomer, if you haven’t guessed, is called aalimraan.hidayahonline.net.

I have a sense of sadness that, within two weeks, the name albaqarah is going to have to be retired.  Therefore, I may rename aalimraan to albaqarah, as the server is really replacing the old one in purpose, and also, I would hate to see the name of one of the greatest suwaar of the Qurʾaan be abandoned, albeit for a very arbitrary purpose.

Also, as should be obvious now, I decided that the formerly unused domain name, hidayahonline.net, will be the administrative name for everything related to the network itself.  hidayahonline.org will remain the Islamic portal.  This leaves hidayahonline.com, which I may retain for commercial services should that opportunity arise (and arise it may yet, in shaaʾ Allaah).

Conclusion

I think that pretty-much wraps-up what I’ve wanted to discuss over the past couple of weeks regarding the Audio Islam migration.  Alhamdulillaah, it went very smoothly, and working with a very powerful server on a super-fast connection has been most enjoyable, even though I’ve been undergoing some bandwidth trials myself (the current connection is 64kb/s symmetric DSL – something I didn’t even know existed before).  We should have a more reasonable connection restored within a few days, in shaaʾ Allaah.

Look, in the future, for some significant updates for Audio Islam, including the years-overdue upgrade to the CMS.  Now that the server is running Fedora, I have even less excuses, as everything I need is conveniently packaged already!

michael.dehaan (his spelling, not mine!) posted something that you can take or leave (I don’t want to start a flame war on my own blog):

michaeldehaan.net | Popularity Contests Are So Junior High School.

But what is intriguing are the ideas that develop in the comments.   Please take some time and read those first, and then come back here. 

What I wanted you to read were the ideas floated in the comments regarding raising awareness of the Freedom of Free Software for the common people, and the analogies made to such programs as NOVA & Cosmos on PBS.  This touched me personally because of how important such programs were to me and to nurturing my love & interest for Science & Nature.

These programs, it seems, were designed to take what may have been seemingly esoteric topics that were originally avoided by regular people and make them accessible.  People like Carl Sagan devoted their lives towards making deep topics such as the universe accessible and understandable by common people.  In no small part, I think they were successful.

So, if success can be had in such an area as Science, then why not also for Freedom?  Why not devote time to “marketing” the advantages that Free software and software freedom bring, both practical and social?  But the noveleity I found was in pointing towards the successful (if not “American Idol” successful) models of Public broadcasting, which did succeed in reaching  intended audiences and affecting them positive ways that brought about positive changes, in my humble opinion.

What’s the next step, then?  Let’s at least foster the idea that we should be working on producing media that fun, easy to understand and follow, and serves to promote what we, the users of Free software, feel is important about it.  I’m going to think some more about this and get back to you all, in shaaʾ Allaah.

Alhamdulillaah, while reading about integers and, subsequently, prime numbers last night, I decided the time had come for me to write a primality calculator in PHP.  The test I use in my implementation is about as a naïve as one can possible get, but it was fun, nonetheless.

Interestingly enough, I discovered that there already exists a primality calculator for *nix-based systems – primes.  It comes with the bsd-games package for Fedora.  I assume the same is true for other distributions.  Mine isn’t quite as fast yet…but it does accept, optionally, one or two arguments. If there are two arguments, then it will calculate all primes starting from the first going until the second. If there is only one argument, it will calculate from PRIME_START (defined to be 2) through to the largest integer PHP can handle on your platform (usually the maximum value of a signed int on your machine).

There are loads of potential optimizations that can be performed, and this was just something written on a whim as a proof of concept. It seems to be accurate, however. The code, released under the GNU GPL v. 3.0, can be found below.

#!/usr/bin/php
< ?php

define('PRIME_START', 2);   //  One, by definition, is not prime

if (isset($argv[2])) {
    $start = (int) $argv[1];
    $max = (int) $argv[2];
} elseif (isset($argv[1])) {
    $start = (int) PRIME_START;
    $max = (int) $argv[1];
} else {
    $start = PRIME_START;
    $max = PHP_INT_MAX;
}

if ($start < PRIME_START) {
    $start = PRIME_START;
}

if ($max < PRIME_START) {
    $max = PHP_INT_MAX;
}

echo "Using $start as start value and $max as max value\n";

$int = $start;

while ($int < $max) {
    $start_time = microtime(true);
    if (is_prime($int)) {
        $total_time = round((microtime(true) - $start_time), 2);
        echo "$int is prime (calculated in {$total_time}s)\n";
    }
    ++$int;
}

echo "All possible primes lower than $max have been calculated!\n";

/**
 * A very naïve test to determine the primality of a given integer
 *
 * @param int $int
 * @return boolean integer is prime
 */
function is_prime($int)
{
    if (! is_numeric($int)) {
        return false;
    }

    $int = (int) $int;

    $test_divisor = PRIME_START; //  By coincidence, the first divisor to determine prime numbers is also the first prime number

    while ($test_divisor < $int) {
        if (is_divisor($int, $test_divisor)) {
            return false;
        }
        ++$test_divisor;
    }

    //  If we are here, then that means no divisors were found
    return true;
}

/**
 * Very simple function to determine if one integer is the divisor of another
 *
 * @param int $dividend
 * @param int $divisor
 * @return boolean $divisor is divisor of dividend
 */
function is_divisor($dividend, $divisor)
{
    if (0 === $dividend % $divisor) {
        return true;
    } else {
        return false;
    }
}

Alhamdulillaah, I’ve been able to make some major- and sorely-needed-upgrades to Audio Islam.  Interestingly enough, for the vast majority of people, these changes won’t really be that visible as they are all behind-the scenes.

The desire to upgrade Audio Islam has been around for a long time, and I still have many more changes, particularly in relation to the code, that are pending.  But I decided to focus on performance due to the vast surge in visitors that will come during the month of Ramadhaan, in shaaʾ Allaah.  Therefore, I had to polish some things, upgrade others, and in some cases, fundamentally change them!

Prior to today, Audio Islam was running a typical LAMP setup – Red Hat Enterprise Linux 4, Apache HTTP Server, MySQL database server, & PHP Hypertext Preprocessor.  Also typically, at the behest of the PHP developers, my Apache HTTP server was configured to operate in prefork mode, which, to put it simply, causes the web server to assign one process per web request.

For a site like Audio Islam, where currently the programming logic is rather limited and the bulk of the action involves downloading of large files, the cautions that call for using the prefork mode, as opposed to the more performant multi-threaded worker mode, are less urgent, so I opted for the more efficient mode due to Audio Islam‘s need for very high concurrency on rather modest server hardware.  I’ve had to set the maximum number of clients in my webserver to over 700 and also offset the more popular mode of download to be FTP just to keep the website accessible, so I am very keen to seek out as much optimization as I can until I can move to another server that will sustain the site as well as this one in terms of bandwidth (currently, 20Mbit/s unmetered is keeping the site afloat, alhamdulillaah).

However, here is where the multi-threaded madness of PHP reared its ugly head.  In truth, I am not entirely sure that I can lay the blame solely on PHP here, but as PHP is not generally multi-threaded enabled and intends to stay that way, I don’t feel so keen on removing them from the blame completely, either.

Before going further, here’s a summary of the specific versions of each package (before/after)

• Apache HTTP Server: 2.2.0/2.2.9
• MySQL database server: 5.0.16/5.0.67
• PHP Hypertext Preprocessor: 5.1.6/5.2.3

As I install all my servers from source on my aging Red Hat platform in order to get the latest versions easily, that means I have the pleasure of compiling all my packages from source (I haven’t figured-out how to make my own RPMs yet…).  So, my first step was to compile Apache HTTP server, ensuring I enabled the worker MPM mode to enable multi-threading (a simple mistake I noticed when I saw the config summary included the old prefork MPM).

My configure command was as follows:

./configure --enable-modules=all --enable-mods-shared=all --with-mpm=worker

This went along very well. My next step was to compile MySQL, as I needed access to the client library in order to compile PHP with support for the database backend.  Note that the coming of PHP 5.3 should obviate this as it will feature a native MySQL driver for PHP which shares the same license as PHP, which sparked the whole problem of the separation between PHP the MySQL client library.

Before I show the MySQL configure command, let me present the PHP one:

./configure --with-apxs2=/usr/local/apache2/bin/apxs --with-mysql --with-zlib --with-bz2 --enable-mbstring --with-mysqli --with-pdo-mysql --enable-zip

This is a pretty basic PHP configuration script, rather slim, but sufficient for what I needed.  However, the configure script complained about a missing library for MySQL named “libmysqlclient_r”.  I looked for it on my system, and while I found it, it was the wrong version.  This is when I realizzed that I had to enable the thread-safe client in MySQL.  So, I did this, using this configure command:

./configure --enable-thread-safe-client

After a rather lengthy make & make install process, PHP was still complaining.  I looked for libmysqlclient_r and found it, but PHP still kept complaining that it wasn’t under “/usr/local”.  It didn’t matter that I explicitly instructed it to look at the install directory of my MySQL installation.

Anyway, after spending several hours trying to figure this out by researching online and trial-and-error, I decided enough was enough, and I was going to follow PHP’s recommendation of running it under FastCGI.  However, I do not have a lot of experience getting Apache to run under FastCGI.  Morever, as I didn’t need Apache’s power & flexibility for Audio Islam, I decided now was the time to go where I’ve wanted to go for a while – to host Audio Islam using lighttpd.

lighttpd (pronounced lighty) is a lightweight yet powerful HTTP server that is designed for low overhead and high performance. Despite these goals, it still hosts a slew of features that sufficed it to be the backend for Audio Islam, alhamdulillaah.  Once I realized that I was going to go through with it, I stopped all my efforts at getting Apache HTTP working and focused on installing lighttpd.  Amazingly enough, I discovered that the latest stable version, 1.4.19, was already available via yum for RHEL4!  I quickly went ahead and install it and its fastcgi module.  And, a short while later, after some configuration, I had Audio Islam up-and-running on lighttpd!  Alhamdulillaah!  It was rather surprising to see how it worked perfectly out of the box once I redirected it to my newly-fastcgi-compiled PHP.  For reference, here’s the final configure command I used for compiling PHP to ultimately work on lighttpd:

./configure --enable-fastcgi --enable-discard-path --enable-force-cgi-redirect --with-mysql --with-zlib --with-bz2 --enable-mbstring --with-mysqli=/usr/local/mysql/bin/mysql_config --with-pdo-mysql=/usr/local/mysql/bin/mysql_config --enable-zip

The first three arguments are from the lighttpd documentation, and the rest are what I wanted in the first place.

So, there you have it! Audio Islam is now running on top of lighttpd, alhamdulillaah.  In shaaʾ Allaah this setup will enable the server to sail through Ramadhaan far smoother than previous years.  I guess we’ll find out in a couple of weeks, in shaaʾ Allaah.