Random, shuffled, endless playlists with mplayer and sh

It's portable and easy. There's no need to do shell / environment specific stuff, mess with arrays in shell scripts or worry about spaces in filenames: this works in any semi-POSIX compliant environment that can run mplayer.

while `true`; do find "$1" -name \*.mp3 | while read filename; do echo $filename >> /tmp/pls.$$; done; mplayer -shuffle -playlist /tmp/pls.$$; rm /tmp/pls.$$; done

I have this plopped in ~/bin/play and run play ~/mp3 which works well for me. You could also do 2>&1 >/dev/null & or >& /dev/null & (for sh/csh like shells respectively) to kill output and background the process if you want.

Have fun :)

A Better ROBOT9000

Introduction

Quite a while ago, XKCD posted an idea about an auto-moderating IRC bot to maintain some sense of order in his channel. The basis for the idea is that as social communities grow, they start to suck. In every online social network I've been involved in, this has been true -- it has also been true in part for some social activities I enjoy outside of the computer / internet realm. In this post, I'd like to touch both on the sociological as well as technical aspects of this subject.


Sociological Aspects of Online Communities

It's probably universally accepted that as a community becomes larger, some form of government should be set up to manage it (anarchists aside). This is evident in the smallest of groups to the largest: we see political order in everything from clubs to our jobs to the management of our society our own countries.

Initially, a very small community consisting of only a few friends may function well without any intervention. After all, these are people who are familiar with each other and thus have experience interacting with each other socially. They know the quirks of the others; they are aware of the boundaries of what is found socially acceptable to each other. A community like this is self-governing: there is no need to instantiate regulations upon such a small community because there is little to no cause for social discord in their normal interaction.

If the idea behind the community is a good one, others may be invited to join who are strangers to the others. As the community grows, more nuances and social quirks are introduced. People become less familiar with one another and we start to see social discord when people (perhaps unintentionally) overstep their boundaries. This becomes more prevalent as the community continues to grow. However, the group remains closed: everyone has a common interest (which is the centerpiece of interaction in the community), and it is assumed that everyone is a friend of someone else there. It is due to this closed nature that, though social boundaries may be traversed from time to time, there is no need for regulation: the group can function normally without moderation.

Now let's examine an open group. Imagine that we have a public service: a swimming pool. In general, most of the attendants will not know each other and, while some bonds may form between them, each attendant is largely unaware (and even complacent) about the social boundaries and expectations of the others. In this case, rules for safety are instantiated with an active moderator (or more), who we see manifested as lifeguards. Frequently, these rules are not limited to physical safety: social safety is moderated as well! (Next time you are at a public swimming facility, look at the rules board: most of the time, there are rules forbidding screaming and profanity.) From personal experience, most people are aware that as the size of the group at such a facility grows, more intervention is needed to ensure the safety of all around.

On to open anonymous groups. This is a fairly new concept (within the last several decades) made possible by advances in communication. BBSes and the Internet have made it possible for people to communicate without knowing anything about each other. It is in this form of communication that moderation is direly needed: as interpretation of communication is largely non-verbal (or, in this case, non-textual), we are making it in a sense more difficult to communicate our ideas. Additionally, due to the implied anonymity of an alias, people are more apt to extend their behavior to extremes as any consequence is fairly benign. Rules are placed and moderation is available, but this is frequently not enough, and largely due to the inconsistencies of moderators in these communities. As moderation is unpaid, their consistency suffers, both in the enforcement of their policies and the actual time they spend moderating. Additionally, what one moderator may consider socially acceptable, another may not, leading to inconsistency between moderators. Finally, since there are inconsistencies in the time they are willing and able to spend performing the moderation, times frequently arise in which no moderation is available, and anarchy ensues.

A lot of what ends up being socially unacceptable in a situation turns out to be repeated conversation. If experience is any indication, most things that could be considered annoying, profane, or otherwise socially harmful are indeed repetitive. Internet memes are repeated to the point of not being any kind of funny; profanities and insults -- well, there are a finite combination of these. A solution proposed by XKCD for IRC chat communities is to create an auto-moderation bot that penalizes all violations in a well-defined and consistent manner.


Implementation

The original implementation of ROBOT9000 is in Perl with a MySQL back end. This implementation is extremely suboptimal: SQL is not good for text searching, and in test use, the bot proves itself to be rather unreliable. It is being run on a server for the ScoreHero community, connecting to the same MySQL database used for the site. The site has over 235,000 members at this time and with its score management and forum features, has a very heavily loaded MySQL database. Because of this, the bot seems to lose its socket to the database after short periods of inactivity, causing it to die and require restarting. Running the bot on a different server seems somewhat counterproductive since the channel is officially moderated by the ScoreHero staff and the bot should be run on assets belonging to the site.

Hence, I'm working on a ROBOT9000 implementation in C. Its functionality is the same, but it uses a red-black tree data structure, storing two 32-bit hashes for each unique line of text, penalizing if the hashes already exist in the list. There are a couple issues to hammer out regarding best practices, manual intervention, and collisions (which I'll probably leave alone). Right now, it's semi-usable. Source code is available at http://testbed.dh0.us/~dho/crobot/

Getting back in the swing of things

Intro
It's been a while since I last blogged. A couple of years, in fact. I've been busy with work stuff and my own life, as happens with pretty much everybody. It has had its ups and its downs, but I suppose I'm in a pretty good place right now.

Building Python Extensions on OpenSolaris
I've got a new job, doing web development stuff. It's not OS development, but it pays the bills (and the people here are pretty cool). Since I've recently been getting back into working on / using OpenSolaris, I thought I'd go ahead and set up an environment a new project of ours at work. We use Python by-and-large (though we do have some PHP projects), and we use Django for most of the things we do. So I was setting up a local server for our latest project, and I needed to install psycopg2.

The install went well enough, but when I ran

python manage.py syncdb

I was greeted with a traceback stating that libpq.so.5 could not be found. I had installed PostgreSQL from the Blastwave package, so initially when building psycopg2, I ran into the problem of it not knowing where my Postgre headers were (I had forgotten to plop /opt/csw/postgresql/bin into my PATH for pg_config). But here, I was stumped.

In FreeBSD, I've had to run ldconfig(8) a few times, but ldconfig doesn't exist on OpenSolaris. I found out that the analogue to this is crle(1), but I was advised not to use it. Instead, it was suggested to rebuild psycopg2 with LDFLAGS containing -R /opt/csw/postgresql/lib. I clobbered the previous build and ran:

LDFLAGS='-R /opt/csw/postgresql/lib' python setup.py build

Everything went smoothly after the installation -- the runtime linker's paths were updated and my syncdb worked like a charm. (Well, after I went into Postgres and created the database, which I hadn't done yet. My brilliance never ceases to astound me.)

OpenSolaris Syscalls
I'm currently bashing my head against a brick wall trying to get cap-eye Install to work for me (I keep trashing my ability to boot the system) while I work on RFE 4616466. I think I have a working syscall for it, but untill I get Install to work and actually get to boot the kernel to run a test program to use the syscall, I have no idea. It was a pretty interesting challenge: from the kernel, how do you go from a local / remote address:port tuple to find what PID is responsible for that TCP connection.

It's kind of a tough answer, actually. Initially, I thought it was pretty straightforward. The tcp_s structure contains a field called tcp_cpid, which is the process that initially created the described TCP session. However, processes may hand off a socket to other processes (plenty of ways to do this), so the PID currently responsible for the socket may not be the one that created it. Indeed, the one that created it need not even be running, which not at all an unlikely possibility. Additionally, multiple processes may manage the socket in question, but I chose to not support this: It would require the caller to know beforehand how many processes use the socket as I would then have to copyout all the PIDs. I could alternatively let the caller specify a maximum number of PIDs they are interested in, but this seems too arbitrary to need to support.

My implementation goes something like this:

1. First, we get a reference to the current IP netstack by calling netstack_get_current() and dereferencing its nu_ip member.
   
2. We get a connection record by looking in the list of connections for the IP stack, using a handy macro that turns the remote address and local/remote port tuple given the IP stack into an index into this array: ipst->ips_ipcl_conn_fanout[IPCL_CONN_HASH(remote_addr, ports, ipst)];
3. We then loop over all matches looking specifically for TCP connections on the local/remote tuple, matching with:
   IPCL_CONN_MATCH(connp, IPPROTO_TCP, raddr, laddr, ports)
   
4. Once we've found a matching connection, we dereference its conn_tcp member to get our tcp_cpid as a hint.
5. Now the fun part. We loop over all processes in the active zone.
   
1. If the process' PID matches the tcp_cpid field, we return that value, as the process still exists and still has control over the socket (and this saves us from needing to do a bunch of lookups into the process' file table).
2. We loop over the process' file table looking for the socket holding the connection. This was a particular bitch to figure out, but I think I got it:
1. The file table is accessible by using the P_FINFO() macro on the struct proc.
2. Starting from 0 and looping while less than filetable->fi_nfiles, we
   walk an ugly list of structures.
   
1. We get the desired fp from filetable->fi_list[i]->uf_file.
   
2. We access its associated vnode (as sockets are implemented using sockfs, we have a vnode for the socket) from fp->f_vnode.
3. The socket node (struct sonode) is stored in the private vn->v_data field.
4. The tcp_s struct for the socket is stored in the sonode's private so->so_priv field.
5. If the pointer of the tcp_s struct here matches that of the tcp_s struct we grabbed from the connection, we've got a winner.
6. If we don't find anything, return ESRCH

I got a good bit of help regarding implementation of syscalls from Eric Schrock's blog entry on adding new syscalls to OpenSolaris. After reading this and doing my own poking around, I also found SYSCALL.README, which has other useful information on the subject.

Anyway, that's about all for this entry. Perhaps more later :)

--dho