I’ve taken to carrying around a RawWorkflow WhiBal card when I’ve been out and about practicing my photography, and had a few people ask me what it’s for and how it helps.
One of the nice things about how we see is that our brains are adept at adjusting our perception of what we see according [...]
I ran in to Smylers’ Talking at Conferences: A Beginners’ Guide the other day. It’s a very useful distillation of things you should consider if you’re thinking of giving a talk at a technical conference, such as YAPC Europe 2007, where the talk was delivered.
One recommendation in particular struck me, in re ordering the [...]
If you’re a frequent (or even semi-frequent) traveller then you may be interested in a website that I’ve been using for the past few months.
TripIt exists to make it easy to manage one or more travel itineraries, and to share those itineraries with other people (who can be passive viewers, or active collaborators).
Signing up is [...]
I note that British Airways e-receipt e-mails are probably going astray for a lot of people.
I’ve had to book a few flights with BA recently. Up until a couple of weeks ago their acknowledgment e-mails came through fine. And then I stopped receiving them. Taking the time to delve in to the [...]
It’s been a bit quiet around here for the last couple of months. A quick brain dump.
In February I left my job helping to run the mail systems at Citigroup, having sorted myself out with a contracting role doing software development in Perl (with some Autosys and Murex mangling on the side) for the [...]
If you publish SPF records, send mail to Korean ISPs, and use SPF mechnisms other than ip4:, you may face a problem.
Apparently (and this is second-hand, so treat it with some caution), the Korean Information Security Agency (KISA) is producing an RBL of domains to blacklist. This is complemented by a whitelist, called WhiteDomain, [...]
I’m a long time (x)emacs user. I use vi for small day-to-day stuff (vi TODO, anyone?), but as far as possible I do any development work with emacs, and have done for the past 12 years.
So, a few days ago I was pleased to discover a new mode that I’d not previously known about.
ecb, [...]
Following on from yesterday’s discussion of new features in Sendmail 8.14.0, today I’m writing about Sendmail’s GreetPause feature, and some additional logging for it that’s been added in Sendmail 8.14.0.
Sendmail has supported pausing before displaying the greeting as an anti-spam measure since 8.13.0.
To use it, add a line like:
FEATURE(`greet_pause', `10000')
to your sendmail.mc file, and rebuild [...]
Sendmail 8.14.0 was recently released, and it includes a small handful of patches that I sent in. The documentation explains what these options do, but doesn’t explain why you might want to use them. So I thought I’d do that in a series of entries here.
First, the new HeloName option.
When Sendmail needs to [...]
Well that Kubuntu experimentation didn’t last long.
After my last post I proceeded to install VMWare Server in Kubuntu. That went relatively easily, and I prepared a Windows XP VMWare image.
The image creation was also relatively smooth, although it took 2 or 3 times longer than a normal Windows install. For some reason anything involving [...]
The Ultra 40 that I won arrived on Wednesday, but I’ve been too busy since then to do much with it until now.
I have spent a little bit of time carrying out some research. My initial plan was to triple boot this server. First as a FreeBSD machine — it’s my preferred Unix [...]
Last Thursday (the 18th) I attended two open source events in London. The first was the third Subversion UK User Group meeting, and the second was Google’s second Open Source Jam.
This was almost a disaster for me. I was coming from docklands, and the DLR was suspended due to the storms that hit [...]
I’m very pleased to say that I’ve just been granted with a partial commit bit for the Subversion source code. The developers have had enough of me popping up on the mailing list talking about issues with the Perl bindings and posting patches, and have given me the necessary access rights to fix and [...]
I’ve given Planet Subversion a facelift, and decided to splash the cash on a domain for it too — planet-subversion.com. I’ve also recently added (but neglected to mention here) a feed from Mark Phippard, who has been writing some very interesting articles about the Subclipse project.
Share This
I write and maintain quite a lot of Perl software. Much of it is open source and available on CPAN. Most of it consists of libraries designed to be used by other applications, rather than applications in their own right.
One of Perl’s strengths is that “There’s more than one way to do it” [...]
One of the common pieces of FUD used to argue against the use of open source software is “There’s no vendor support. If it breaks, who do we turn to?”.
There are a number of answers to this particular concern.
First, there’s the community of developers that develop the software, and the wider community of other [...]
After a slightly longer development process than I would have liked (I had several hard disks choose inopportune moments to go south — no data loss thanks to backups, but I took the opportunity to shuffle some hardware around), SVN::Web 0.50 has been released, and should now be available on CPAN.
There are quite a few [...]
If you’re in London on Thursday 18th January and you’ve got an interest in Subversion then you might want to come along to the next Subversion UK User Group meeting. It’s at 3pm, there are directions and sign up instructions at that link.
I’ll be there, giving a tour of the Subversion related applications and [...]
A great deal of spam is sent by “botnets“. These are (typically) Windows PCs that have been compromised in some manner, and are now illicitly controlled by a third party. This third party uses the network of thousands of PCs that they have compromised to:
Send spam
Host phishing sites
Carry out denial of service attacks
There [...]
I’ve just discovered that I’ve been an unwitting participant in an identity theft.
But not, perhaps, in the way that you might imagine.
As already chronicled, some of my writing recently made it in to The Guardian. As is the way of these things The Guardian like to pay their writers, so I sent off my [...]
One of the sites I make a point of reading regularly is use.perl, and in particular, the user journals / blogs. They don’t take too long to read, and there’s normally a couple of posts a day that teach me something I didn’t know about Perl, or that highlight a new module that’s doing [...]
You’re probably aware of Google’s code hosting service. They use Subversion as their revision control system, so if you want to contribute to a project hosted there you really need a Subversion client.
I was reading the FAQ for the hosting service the other day and a particular entry struck me.
Do you have support for [...]
I’ve been experimenting with Plagger, a tool for plugging together chains of filters, pumping RSS/ATOM feeds in one end, and getting transformed output at the other end.
This doesn’t have to be as simple as chaining a few XSLT transformations together, as Plagger filters can carry out additional actions (such as e-mailing the results to you, [...]
Two spam related pieces of information today.
The first concerns what happens if you’re hosted at an ISP with an anti-spam policy, an itchy-trigger finger, and a support desk that is devoid of clue.
It appears as though the fine folk over at The Weekly had their infrastructure on a shared server at their ISP, HostingPlex. [...]
Charles Arthur’s wondering why spam came through his CAPTCHA system, and concludes that people are probably being paid to sit there and fill out CAPTCHAs.
There are a couple of other possibilities. The first is that the CAPTCHA system he’s using might be compromised. Some OCR systems can be surprisingly effective on them.
The second [...]
I was pinged on IRC earlier today by someone who was having an e-mail discussion with Charles Arthur of the Guardian, in response to this article on Six steps to stopping spam. Since I spend a lot of my day job doing anti-spam engineering for a large organisation, Robbie thought that I might have [...]
… or so I thought.
After I sent in my submission to Sun’s Try and Buy contest, I didn’t hear anything back from them for some time. So I thought it wouldn’t hurt to drop them a quick note and check that they received it. After all, it’s not unknown for something to [...]
This system will be going back to Sun soon, while I wait to find out whether or not they’ve decided to grant me the system. In the meantime, here are some final thoughts on the last 59 days.
DTrace
DTrace is remarkably useful. The ability to trivially trace what an application is doing with a [...]
Yesterday’s tests show that using gcc on both FreeBSD and Solaris yields a marked improvement in the time taken to compile Perl.
However, despite the big difference in compile times, the run-times of Perl’s test suite aren’t dramatically affected. The worst performer, Perl running on Solaris, compiled with Sun’s cc and optimisation is 6% slower [...]
Yesterday I looked at performance compiling Sendmail on Solaris and FreeBSD using Sun’s compiler (on Solaris) and gcc (on both systems).
In the tests gcc come out handily ahead, with gcc on FreeBSD being 16% faster than gcc on Solaris with low optimisation options, and 12% faster than gcc on Solaris with optimisation turned on. [...]
As I explained on day 55, I’ve been comparing GCC and the Sun Studio compiler on Solaris, to GCC running on FreeBSD to see if there are any significant differences in the time taken to compile the applications, and if there is, whether that difference is reflected in the time taken by the applications to [...]
Fetching and installing FreeBSD was relatively painless. I downloaded the ISO image for FreeBSD 6.1 (AMD64 build), and wrote it to a blank CD.
When I reinstalled Solaris some weeks back I made sure to leave some space on the disk for FreeBSD. So then it was a case of booting from the CD.
This [...]
The last week has been quite busy with work that’s not related to this project. Mindful that the 60 day time limit is almost up, and aware that I’ve not done any actual benchmarking of this workstation — vis a vis “How does Solaris on this hardware compare against another OS on this system?”, [...]
I’ve been looking at different queue sort strategies to see what their overhead is. Since all the messages are going to be delivered to a single host these results aren’t necessarily going to be indicative of what you would see on a production server. However, they should serve to illustrate any inherent speed [...]
It’s definitely a bug.
Specifically, in the default case, and contrary to the documentation, sendmail does not run one queue runner for every queue directory. It runs precisely one. I brought this up on the Sendmail mailing list, sendmail-2006@support.sendmail.org. The most recent message in that discussion (at the time of writing) follows.
From: Nik [...]
That’s odd.
You may recall that my most recent tests have involved sending 30,000 messages, split over 1, 5, 10, 20, 30, and 40 queue directories respectively.
The Sendmail Operations Guide says:
2.3.1. Queue Groups and Queue Directories
There are one or more mail queues. Each mail queue belongs to a queue group. There is always a [...]
I’ve started to get data about the effect of multiple queues with multiple queue runners.
As before I’m using 1, 5, 10, 20, 30, and 40 queue directories, and I’m instrumenting with queue-run-duration.d. This time I’m starting queue runners with the command sendmail -q30s. This will cause Sendmail to create a new queue runner [...]
Today I’m looking at the results that I’ve obtained from the latest round of tests. These tests used sendmail -q to deliver 30,000 messages to a different zone. There were 10 runs to each test, and the different tests collected data on timings for 1, 5, 10, 20, 30, and 40 queue directories.
Highlevel [...]
Previously I’ve written about variables that may affect how rapidly Sendmail can process the mail queue. I’ve now started working to gather data on exactly how much influence these variables have.
Basic methodology
The relay-zone is configured to accept messages from the internal-zone, and relay them.
The internal-zone runs smtp-source(1) to generate a large number of messages. [...]
Note: If you’re not familiar with sendmail queues, the sendmail queue primer I wrote might be useful.
There are two aspects of mail queue management to consider with Sendmail. The first is the process that puts messages in the queue. I’ve looked at that in some detail already, and written a number of D [...]
Or: “How do I use DTrace with programs that fork?”
With some help from the dtrace-discuss[1] mailing list I’ve now written a couple of D scripts that can trace what Sendmail is doing between probe points. There’s a writeup, and sample output, below the fold.
[1] Note — the forum archive doesn’t seem to link to [...]
Back on day 28 I looked at the effect of multiple queue directories with concurrent senders.
These results showed that there was considerable benefit with 10 senders and 10 queue directories. The benefit going to 20 queue directories with 10 senders was negligible.
At the time I wondered whether this was a general rule — i.e., [...]
Having established that there’s a significant increase in the amount of taken by the fdsync() and open() system calls when Sendmail creates queue entries with a single queue directory I’ve set about tracking down what that bottleneck is.
To confirm that these system calls were the ones most dramatically affected by multiple queue directories I re-ran [...]
Earlier posts have shown that using a single queue directory imposes a significant bottleneck when processing concurrent connections with Sendmail. Yesterday I posed some questions, and today I’ve started work on answering the first one.
The first question was:
What is responsible for the dramatic slow down in the single-queue case (test 4)?
To start answering this [...]
Yesterday I looked at the effect of multiple queue directories when processing messages over a single connection.
Today I’ve been looking at how multiple queue directories can help when processing concurrent connections.
The methodology was identical to the previous tests. The only change was to the smtp-source(1) command line. The previous tests were run with [...]
I’ve commited the first sets of results to the repository in the aptly named results/ directory.
To refresh your memory, the question I intended to answer was:
does the number of queue directories (on a single disk) make a significant impact on the time taken to create new entries in the queue?
They’re quite surprising.
First, let me point [...]
It’s time to run an instrumented Sendmail, throw some messages at it, and see how it performs. Specifically, does the number of queue directories (on a single disk) make a significant impact on the time taken to create new entries in the queue?
To do this, I need to instrument Sendmail’s queue creation functions, and [...]
I’ve (finally) got Sendmail built, zones configured, DTrace working for functions declared static, and a mechanism for creating test SMTP sessions.
So it’s time to start putting this together, instrumenting Sendmail, and seeing whether or not I can use this to prove (or disprove) some common advice given when configuring Sendmail.
First, I’m going to look at [...]
It’s the school holidays, and my two children have had friends staying over this past week, which meant that there hasn’t been much opportunity to work on this project, and even less opportunity to write about it. So these next few posts are going to be something of a catch up.
I’ve previously documented running [...]
I’ve now got Sendmail built and installed, and adjusted the SMF so that it uses my local version of Sendmail (with DTrace probes) in favour of the system version.
There were four chunks to this work.
Configuration build infrastructure
Sendmail configuration files (/etc/mail/sendmail.cf) are normally built by building a simpler configuration file (sendmail.mc) that uses a vast range [...]
Now that Sendmail is building and correctly installing in to a custom directory it’s time to start looking at how I get my version of Sendmail used instead of the version that’s supplied with Solaris.
For that, I need to delve in to the Solaris Service Management Facility (SMF).
At the risk of oversimplifying, the SMF is [...]
The issues with M4 have been resolved. A colleague, Andre Lucas, took up the challenge and worked out a fix which he describes in detail. And ministat’s now looking much better. It’s grown some useful new options, a lot of documentation, and can now (optionally) generate plots in colour. Look below [...]
I’ve spent some of today porting some useful statistics reporting software from C to Perl.
ministat reads in two or more files of data and uses the Student’s t test to determine if there is any statistical difference between the means of the datasets. This is especially useful when comparing benchmark results.
For instance, I figure [...]
Which is a problem because the Sendmail build system is written in it.
Here’s the problem I’m trying to solve.
As mentioned previously, Sendmail supports a DESTDIR variable, which you can set on the make command line, to specify a prefix to be added to the all the installation paths.
The problem is that the build infrastructure doesn’t [...]
I’ve just had to carry out a complete reinstall of the OS, which was uncommonly tedious.
If you recall, I’d mentioned wanting to upgrade to Solaris 11b40 a few days ago, to get some of the latest and greatest DTrace functionality.
Having downloaded and burned the CD ISO images I was ready to go. So I [...]
I’ve spent some time today getting Sendmail+DTrace to install properly.
This wasn’t quite as straightforward as it could be, requiring a little build infrastructure hacking.
The general stanza to build and install Sendmail after extracing the source code is:
% sh Build
% make install
This will overwrite your existing Sendmail installation. That’s not what I want. For [...]
I’ve been a bit busy with other work over the past few days, and haven’t made quite as much progress as I’d like.
There are a few things that have moved forward though.
First, I now have all three zones running with my home directory mounted read-only. This was trivial to do, and the changes to [...]
Following Monday’s info dump about queues, I’ve spent some time over the last few days reading the DTrace documentation in detail. In particular, the Solaris Dynamic Tracing Guide. This is the DTrace handbook, with a great deal of information about how to use DTrace.
It also contains the information about how to add custom [...]
The time has come to start adding DTrace functionality to Sendmail. Of course, there’s no point in just diving in and adding code left, right, and centre, so over the last couple of days I’ve been thinking about what I should be instrumenting first.
One of the issues that I see with Sendmail when I [...]
I’m just starting to get my feet wet with DTrace. Does anyone know of a decent Emacs mode for editing .d files?
Share This
As of Saturday night I’ve installed new lenses at the Midland Drive-In. The new ISCO lenses replace the probably 40 or more year old lenses which included Ultra Panastar adjustable anamorphic lenses. Bottom line… there have apparently been some improvements in projection lens technology in the last few decades (go figure!). The [...]
Scotiabank’s ScotiaOnline online banking application has a major flaw. If you schedule a bill to be paid in the future (it’ll be listed under “Pending Transactions”) and then after 6pm on the day that transaction is to occur delete that pending transaction it’ll still be processed as if you hadn’t deleted the pending transaction [...]
A bunch of my friends who teach at St. Theresa’s High School and I have been working on a musical… Into The Woods. It runs from April 30th to May 3rd at the school. Show starts at 7pm each night. Go see it (and my lighting design).
Wow. They must have this number on a hunt group. About 2.5% of the spams I’ve handled in the last 2 days (and about 10% of the spams to my own account) have had this phone number — more and more obfuscated as time goes on. It’s already surpassed this week’s volume [...]
Back when I was in university I had, one day, a customer that went from no rush, whenever, to can this application be live in three hours. The application required a decent server and a couple megabits of bandwidth. I didn’t have any machines in a data center at the time with both [...]
Microsoft Antigen for SMTP found a message matching a filter. The message is currently Purged.
Message: “Re_ Pbl.spamhaus.org down_”
Filter name: “KEYWORD= profanity: piss”
Sent from: “Daryl C. W. O_Shea”
Folder: “SMTP Messages\Inbound”
Location: “psp/TRACYSV05″
Piss! Oh noes! The utter profanity of replying to someone who said “Am I blocked? Did I piss someone off?” is simply unacceptable.
I’m simply amazed at [...]
Reuters is reporting that the US Navy says they hit their failed spy sat three hours ago (at 22:26 PM EST). I guess it won’t be going on eBay as others had hoped.
My nightly SA mass-checks have been hanging up this week on a 1MB email (not sure how a 1MB message got in my mass-check corpus, but that’s not important). It turns out that it was Mail::DKIM v0.29 that was taking about 150 seconds to process the message, while the rest of SA was only [...]
One thing I’ve forgotten, twice now, to do before attempting to run a job on Hadoop clusters that I’ve setup in a hurry to demo something is to make sure that all of the nodes are registered in DNS or at least have entires in their hosts files about every node (datanodes, the namenode, the [...]
In early December I was nominated and elected to become a member of the Apache Software Foundation (ASF). I’ve been contributing to SpamAssassin, an Apache project, since just after the release of 3.0.0 in October 2004 (and a committer since March of 2005). I’ve been using and working on SpamAssassin since Justin Mason’s [...]
My buddy Theoretical Ken just posted what might be the most useful blog post I have ever come across! How to tie your shoes so they don’t come undone, YET are easy to undo when you want to take them off!
Yes, you can do a double knot, but that is a pain to get undone. [...]
Yahoo! has been losing exec’s faster than, well, AOL used to. If you have been following the Terry Semel firing and the failed MSFT take over news, you know that Yahoo has really lost a lot of talent lately. Techcrunch.com had an article on a site that makes the resignation easier for people still there. [...]
Check this out!
“It’s like Deja Vu all over again.” - Y.B.
There has been a lot of talk lately from the cable broadband providers (Comast, TW Cable, etc) that they are looking to implement bandwidth caps on their cable internet customers. Some will begin capping your download to 5GB a month in the fall, which is a VERY [...]
Steve and Erin are some of our closest friends. Rylan hired both of them when she worked at AOL back in the 1990’s. They have an awesome house down on Lake Anna in Virginia and invite all the friends down for a weekend every year.
This year we had a blast swimming and tubing and water [...]
I bought a pair of Audio Engine A5 speakers and they arrived today. I like music a lot and I enjoy good quality sound. I have tried a number of things to find something reasonably small with a self contained (self-powered) amplifier that sounded great. While there may be some competitors out there with good [...]
A few weeks ago, I borrowed a robotic DVD/CD duplication machine (made by Primera Technologies) from one of my clients. As of the first of the year, Primera released some software that would allow the machine to be used for CD ripping (instead of burning). I thought this was great and I could finally rip [...]
In March, I had the privilege of supporting Davy Crockett in the Moab 100 race. At the race, I met Jarom Thurston of Payson, an ultra-running friend of Davy’s and who was also running in that race.
I happened to get an email from Jarom the other day announcing that he would be running the Bad [...]
My birthday is in March….March 29th, every year. And if you read this blog you likely know I celebrated 40 years this time around by sailing the Caribbean with my family and then taking the kids skiing and hiking in Utah for a week. Pretty nice celebration.
It is now June. My birthday is a good [...]
I think this is the best portrait ever of my wife, Rylan. I love window light.
(click image for larger)
Seit Anfang 2002 unterhält die österreichische Regulierungsbehörde eine Liste von E-Mail-Adressen, deren Inhaber kein Spam-Mail erhalten möchten. Bislang wurde dieses durch das E-Commerce-Gesetz eingeführte Verzeichnis als einfache Text-Datei geführt. Wer eine Art “Spammerbekenntnis” ablegte, erhielt eine Kopie der Liste, um seinen Datenbestand anpassen zu können. Ab sofort werden die Daten nur noch in Form von [...]
Ein Bericht über social engineering zur (angeblichen) Nutzung von Erotikangeboten, zu der nun versucht wird, Geld einzutreiben. Ungewöhnlich dran ist, dass dies nur vordergründig so zu sein scheint. Es scheint mehr so zu sein, als dass versucht wurde, den vermeintlichen Urheber dieses Betrugsversuchs ins schlechte Licht zu setzen.
In den Mails wird behauptet, dass der [...]
Der Beitrag, der mich zu diesem Eintrag inspirierte und aus dem ich gleich ungewöhnlich lang zitieren muss, führt das Wort Zensur nicht in Anführungszeichen. Ich habe sie hier im Titel eingeführt, um von dem eigentlichen Akt des Zensierens abzulenken, und darauf zu verweisen, das in manchen Ländern verbotene Inhalte (zensierte Inhalte) ein Handelshemmnis sind, da [...]
Matthew Prince von Unspam hat sich bisher vor allem durch das Projekt Honey Pot und das Betreiben von Antispam-Registern in (bundes-)staatlichem Auftrag hervorgetan.
Aus dem Projekt Honey Pot ist nun eine Klage vor einem Gericht in Virgina entstanden, bei der Matthew Prince anführt, dass hier nicht ein Provider, sondern eine Vereinigung von 20.000 Spam-Empfängern (=Opfern) klage.
Unspam-Chef [...]
Hier ein Video, das bei YouTube eingestellt wurde. Es handelt nicht von Spam, sondern thematisiert künstlerisch Spam dar. “The Human Spam” - in etwa “Der menschliche Spam” - zeigt zwei Kerle, die vom Outfit her an die Mario Brothers, den Figuren aus den bekannten Jump-and-Run-Spielen. Der ein nervt den anderen mit sinnlosem Rumgehüpfe und Generve. [...]
Anwalt mit Sachkenntnissen im Wettbewerbsrecht (müssen das nicht alle sein? oder tun die nur so?) dürfen die Kosten ihres Tätigwerdens nicht mehr selbst in Rechnung stellen, wenn sie gespammt werden und dann abmahnen.
Bundesverband der Verbraucherzentralen (vzbv) wärmt Kritik an Telemediengesetz mit Umfrage zu eingestellter Spam-Beschwerdestelle auf.
Ich habe eine Email-Adresse bei Gmail … ups, aus markenrechtlichen Gründen heißt es auf Deutsch natürlich bei Googlemail. Ich habe auch kein Spam-Problem, auch keine mit false negatives oder false positives. Meine Frau hat auch einen Zugang, nutzt die Email-Adresse aber nicht. Gerade hat Sie mir die Emails in ihrem Googlemail-account gezeigt: Lauter Spam. Pamela’s [...]
Der niederländische Regulierer OPTA hat nun ein Bußgeld gegen ein Spammer verhängt, der neun Milliarden Spam-Emails Pillen, Sexartikel und Pornografie versandt hat. Der Spammer hatte sich dabei in großem Umfang (Botnet? Zombies?) fremder Rechner bedient:
Als besonders belastend wertete die niederländische Behörde, dass der Mann durch Manipulation Hunderter von Computern die Systeme nichtsahnender Internetnutzer für den [...]
EIne neue und überraschende französische Rechtsvorordnung regelt die Domainvergabe französischer Registries und macht sie zu Handlangern im Auftrag des Staates.
