You can view the whole article here:

http://today.reuters.com/news/newsArticle.aspx?type=technologyNews&storyID=2006-04-07T181554Z_01_N07376126_RTRUKOC_0_US-COLUMN-PLUGGEDIN.xml&src=cms

There are some great quotes in there about how the future looks as mobile phones, ubiquitous (unmetered) internet connectivity and GPS converge, plus Niclas is the first person who seems to have noticed that my A2B business card has a latitude and longitude instead of a normal address on it.

In my opinion, this means that splogs are big business and the spammers are making a lot of money out of them. Four hundred and forty-two dedicated web servers afterall! These machines aren’t doing anything else (as far as we can tell) but hosting splog sites and sending out pings for them.

We now have 370 web server IP addresses in our blocklist. Yesterday we received 474,394 pings of which 161,094 were blocked immediately based on the web server being on the blocklist. Of the rest, 179,467 were picked up by our incremental “delay buffer” which means we won’t go out and index them for up to 8 days if we get another ping for the same URL.

Here’s a rough explanation of how our splog blocklist works. I already posted this text as a comment to a Spam Huntress post a while ago:

At A2B (see http://www.a2b.cc for more) we run a search engine with a ping interface. Bloggers who have geo-located META tags in their HTML (see http://www.a2b.cc/help-searching-addurl-blogping.a2b for more) can ping us and we’ll pick up (parse) their page and index it in our geosearch engine. We receive pings from many individual bloggers, a full ping feed from pingomatic.com, and bulk pings from several other sources, usually around 700,000 to 1 million pings per days. With approximately 200 IP addresses in the blocklist, about 37% of daily pings are blocked immediately.

To generate the list, we recorded the URL of each website (read blog) we were pinged with and also converted it into the IP address of the web server for each URL. We recorded the IP address and added to a counter every time we received a ping for a URL on the same web server. We soon noticed that we were getting many thousands of pings for the same IP addresses, so pulled a script together to list the top IP addresses by number of pings.

We built another script which showed all the URLs associated with each IP address. In order to decide which web server IP addresses are serving splogs, which we block, we open this script and manually have a look at a random sampling of the URLs - it’s usually pretty easy to tell if they’re splogs as they’re just full of advertising links or are quite random in their choice of subject matter. Any web server which has real blogs tends to stay off the blocklist (so that rules out blogspot.com even though people are using it for splogging).

As soon as we’d blocked the first 112 IP addresses, the amount of traffic we were using parsing blogs dropped from 27GB per day (it was so high that it was costing us money in hosting charges) to 6GB/day. Of course, it began to creep up again soon after, so we’re realising it’s an ongoing effort and are beginning to think about blocking whole ranges of IP addresses.

Hope that’s of interest to everyone.

Ann Elisabeth also pointed out that we’d forgotten to put “nofollow” tags on the links to the splog URLs. We’ve now fixed it. Thanks again!

Recently the number of pings and the number of new URLs had been shooting up every day, presenting us with a problem. Because we go out and parse every new URL, we had been generating an average of about 27 GB of traffic a day (yes, that’s twenty-seven gigabytes). Time for some action.

A couple of months ago we’d already introduced a system where we wouldn’t parse a URL again for a while if the first pass didn’t contain geographical meta information, and this had reduced our traffic somewhat initially, but it was back up and climbing fast. We ran some analysis, and found out that a huge proportion of the URLs we were getting pings for turned out to be autogenerated spam blogs - endless variations on “Real Estate in Florida News Blog” and “Hair Loss Removal Therapy in Seattle Blog” and other typical spammer themes. We discovered that many of these URLs seemed to be resolving to a smallish number of web servers set up specifically for hosting spam blogs. Time to block some IP addresses!

We came up with a blacklist of banned IP addresses, based on which web servers the largest numbers of received ping URLs resolved to, and wrote a script blocking anything matching those IP addresses. We banned 114 IP addresses, and our traffic dropped immediately from 27 GB per day to 6 GB! Result! One in the eye for blog spammers! At least on A2B anyway…. You can check out which IP addresses we banned, how many pings we’re getting for each web server, and which URLs they’re sending us, here.

Now we’ve got a list of most popular web server addresses each day and we’re adding to the blocked list all the time! But it’s a struggle as the spammers are constantly changing IP addresses and their exact system is difficult to figure out.

Perhaps the most interesting thing is that we’ve had to ban the web server that Loic Lemeur’s English blog sits on. He’s the founder of Typepad and runs the Les Blogs conferences! The server only holds a few URLs in our database, they all seem to be spam apart from his, and we were getting a lot of pings resolving to it. Does he have friends who are blog spammers? I can’t imagine so, but lots of the URLs are typepad.com domain names.

****
Update - I have just found out these spam blogs are called splogs - we’re now fighting splogs!

Setting up Opera as a first-time user can be a slightly intimidating experience - almost every single aspect of the look, feel and background workings of the browser is configurable, and it can take ages to work out how to customise (customize, whatever) it so that it is as useful as possible. Recently, I’ve had a few newer users ask me how I got Opera working well. I put together a few tips for one of them - I hope they’re useful:

Appearance

- IMO the best way to customise the look/feel of the browser is by right-clicking on one of the toolbars and selecting “customize” to get the Appearance menu. I then like to have a “floating” panel placement with Mail, Bookmarks, Contacts etc selected, and “show panel placement at edge of window”. Then when you click on e.g. the Received view in the mail panel, the window will open up.
- Selecting “Restore All” in the main Window menu will give you all your windows/tabs detached from the background, then you can move them around
- In the “customize” menu mentioned earlier, you can select the buttons tab and drag the buttons anywhere you want - dragging “All accounts” and “Start search” into the mail panel and the mail view window can be very useful as you can then restrict a view (or a search) to just one account. Also having the “Check” button in the mail menu is useful as you can “Check All” or check just one account.
- I think the main toolbar panel needs some buttons dragging into it as it misses back and forward and some other by default - you can do this using the technique above.
- There are various keyboard shortcuts and mouse gestures like F11 for full screen, plus you can define your own. See Tools -> Preferences -> Advanced -> Shortcuts where you can click on edit to see all of them.
- The Opera forums at http://my.opera.com are a goldmine - people are usually very responsive when you’ve got a problem (as long as you’ve searched the forum for an answer already)
- On the Appearance -> Toolbars menu I have the progress bar pop up at the bottom of the page, select “images and text below” in Style and “No wrapping” in Wrapping. I don’t bother with the status standard toolbar, but do have a personal bar (you can put bookmarks on it, or whole bookmarks folders)
- Skins are in the Appearance menu, I have Zephyr running. Lots more on the Opera site

Main Preferences

- The Tools -> Quick Preferences menu is quite useful
- To get PDFs viewable in an Opera window you might need to install a plug-in DLL in the right directory - details in Opera forum (plus you need 8.51 as there was a bug with it in 8.50)
- In the preferences -> general menu, selecting “Continue from last time” as a Startup option is good - it will even continue from where you were if it crashes (it doesn’t crash too too often these days). I’ve also got “Open pages in tabs” selected as the pages option. If your panel placement is “floating” in the appearance menu then I find this is really flexible as you can move your windows around.
- I don’t bother with Wand (tools -> preferences) but I do have the auto-complete section filled out so that I don’t have to type in my street address every time I fill in an HTML form
- The Tools -> Preferences -> Advanced settings are where the fine stuff happens. I’ve got “Remember last size” selected for Browser -> New Pages so that all new pages open in a standard size - you may need to “restore all”, change the size of the window to the way you want it, then exit/restart Opera to get it working just the way you want it. Also, “Cycle Pages” in recently-used order allows you to cycle through the open windows using CTRL-TAB. Reuse existing page is clicked, as is show scroll bars. Can’t remember why, but I have smooth scrolling turned off - think for CPU or memory reasons on my crappy PC at home.
- I don’t bother with this, but Preferences -> Advanced -> search allows you to define a URL like “g andre stiphout” to search Google for your name when you hit return. I don’t bother because I have “Google Search” and “Find in Page Search” buttons dragged into each browser window from the Appearance menu, so I just type in the Google search term there and hit return.
- In preferences -> advanced -> history, I’ve got the memory cache set to automatic.
- Proxy servers are under network in advanced preferences

M2 Mail Client

- M2 is great for running multiple mail accounts on one machine - you can check them all individually or the whole lot in one go
- Most of the mail settings are in tools -> mail and chat accounts
- When sending from my laptop, I automatically BCC myself so that my home PC will always pick up a copy of any mail I send. I then have “Leave messages on server” selected in accounts -> edit -> incoming for the laptop, but not the home PC. Guess I could use IMAP for this…
- If you want to use multiple “From:” addresses with the same POP3 account, then set up another account and uncheck the “Include this account when checking manually” box in accounts -> edit -> incoming.
- I have the “Check/Send”, “Compose”, “Start Search” and “All Accounts” buttons in the mail panel in the floating panels. Then I can select which account to check from the drop-down, only look at one account by selecting an account from the “All Accounts” drop-down, and search for mail by typing in a keyword - it will automatically open a search window when you hit return. If you then click on All Messages -> Received in the panel, it will change the contents of the mail window to the received view,although you can come back to the search by selecting it under the “Searches” views.
- When you hit compose you just select the From: address and it will send the mail from the account and automatically change the signature you have defined in the “outgoing” settings for that mail account.
- I have unselected “Automatically wrap outgoing messages” in the outgoing settings as I found that wrapping was getting nasty, but I do have “Add contacts when sending message” selected as then you can just type in the first few letters of a contacts name and Opera will give you a list of options.
- Whilst only one mail/news view window may happen to be open at the same time, Opera will remember the settings separately. So you can “view -> threaded” for a mailing-list by clicking on the view button in a mail view and “view -> flat” for a general window (which may or may not contain the same mailing-list messages.
- I’ve also got “Display” set to “List and message below” in the mail window view settings.
- Mailing-lists are recognised by some kind of X-mailing-list header so some of them don’t show up as a mailing-list. However, the ones that do have the right headers make use of this incredibly useful feature in the mail panel where you can just click on “Mailing-lists” and then select the list to see the latest messages. You can also remove mailing-lists from your normal “Received” view if you only want personal mail to be shown.
- I only have “read”, “mailing-lists” and “filtered” showing in my received view. You can click on the “spam” button to dump something into the spam view - the filter learns quite quickly, but you need to scan your spam folder every so often for false positives and then click the “not spam” button. If any of the buttons aren’t in any windows just drag them from the appearance menu.
- Because I have something over 200,000 mails on my PC at home I’ve got Period set to 1 month in the view -> period menu in the Received mail view which improves the speed of the application. This doesn’t stop mails from “Forever” showing up when I search for a keyword.
- “Redirect” is equivalent to “bounce” in Pine.
- I can’t remember how I set the quote mark as a “>” in the composer window, but each composer window has it’s own view menu where you can add header fields.
- You can also add RSS feeds in the feeds menu - I’ve got about 30 in there! When you view a weblog or another site with an RSS feed in your browser window, a small RSS icon pops up in the URL bar. Click on this to subscribe to the feed.
- When you import mail from an mbox file it gives it its own account/view.
- Empty trash just by right-clicking on the trash view in the panel. I use CTRL-A to select all mail in my spam view, then hit the delete key, which dumps the spam into the trash view/folder. Then right-click to delete.
- Saving attachments used to be easy with the standard windows save menu but now it’s a bit of a pain as Opera introduced its own menu. Lots of people complained on the forums though, and that often has an effect, so maybe it will be changed at some point.

Okay, hope that helps people out a bit. Feel free to drop me a comment if you’ve got more questions.

The script can be run from the command line and allows you to define the ping target or destination (the server) you want to monitor, the name of the machine you are pinging from (useful if you’re pinging to and from a matrix of machines) and the email address you would like to send any alarm emails to. It seems to work well on a couple of Linux machines I tried it on.

You run the script using e.g. the following command:

…where you need to put the path to the php package in your setup instead of “/usr/bin/php”, you need to put the path to the PHP script (I named it ping-host.php but you could call it something different), -h is the “host” you are pinging, -l is the name of the “localhost”, i.e. the machine you are pinging from, and -e is the email address (or multiple email addresses) you want to send any alarm mails to.

I have an “email to SMS” address as one of the alarm message recipient, so I get a text (SMS) message on my GSM mobile phone if one of the servers I’m monitoring has been down for 10 minutes or 25 minutes. You could also run a whole network monitoring project with this, using it to ping routers or switches.

I’ll put the script into the posting below, but in order to run this, you also need a table in a database which looks like this:

- You should just be able to cut-and-paste that into the database

I just run this script on a cron job once a minute (feel free to alter the values you use) for each server to be monitored. The line in the crontab looks like this:

Every time you add a new server to the ping list in your crontab you’ll also need to add a row in the mon_host_ping table in the database with its mon_host_ping_factor set to zero:

Okay, so now here’s the code for the script, ping-host.php :

I also learnt how to use the PHP getopt() function properly to import command-line arguments into a PHP script. If you’ve read this far, you may have noticed that the script used the following PHP line:

This finally worked, but I had to try various combinations to get it working:

I finally worked out that you need to put all the arguments you’re expecting in the original getopt() function as it can only be called once on a script run from the command line - in this case the arguments are all dumped into the $getoptarray array and you can then pull them out individually using something like this:

Any queries, just post a comment. Better still, have a look at A2B, download the free GPS software, add your blog, pass on the recommendation to a few other people, and then post a comment.

Well, it’s up to you to read the posting and draw your own conclusions, but we will be adding the ability to post a URL and/or a POI in the very near future (especially now that this author has returned from a stint working on another project in the UK).

By the way, A. Sergio, feel free to go ahead and share your calibrated maps through the forum’s thread on this topic.

I also note that A. Sergio has a link to GPSDash on his site, which is another great GPS application. Some day I hope GPSDash will also build A2B functionality into their application.

What you will need for this example:

- Windows 98/2000/XP machine (other OSes work too)
- SSH client which allows port forwarding (we use Putty, which is free, on our Windows machine)
- SSH server which has port forwarding configured, running on a remote machine (this example is using OpenSSH sshd on SuSE Linux) - usually you can run this machine at home
- SMTP server name and port number (usually 25) which you use for sending mail. If you’re trying to do VoIP, you can just substitute your VoIP port(s) here.

Step 1 - Set up the SSH server to allow port forwarding
This is as simple as unhashing the following line in your sshd_config (normally either /etc/sshd_config or /etc/ssh/sshd_config) file which controls the configuration of the ssh service on your server:

From:
#AllowTcpForwarding yes

To:
AllowTcpForwarding yes

You then need to restart the ssh daemon - if you’re running SuSE Linux you can usually do that using the YAST configuration tool. The SSH service will now send all traffic forwarded to it off to the correct destination.

Step 2 - set up your SSH client to port forward:
In Putty, this is very simple. Create an SSH session to your ssh server by entering the IP address and port number (usually the default port 22). Now configure SSH tunnels under Connection -> SSH -> Tunnels on the left-hand side of the Putty session configuration screen.

Pick a high port on the local machine which the email client can be configured to connect to - I recommend adding 3000, 4000 or 5000 onto the existing port, giving you 3025, 4025 or 5025 - we will use 3025. Now, enter the port and host in “Add new forwarded port”. For “source port”, enter the high port you have just picked on the local machine, so “3025″. For “Destination”, enter the hostname of the mail (or VoIP) server and the port used on that machine, with a colon between them - e.g. “smtp.example.com:25″. Then click on “Add”, go back to “Session” on the left-hand side of the configuration screen, name your session in the “Saved Sessions” box, and click on Save. It’s also worth reading the Putty help file for port forwarding.

You now have an SSH session which is capable of taking traffic sent to port 3025 on the local machine, forwarding it through the SSH session, and sending it to smtp.example.com port 25 once it reaches the the SSH server. Now you just need to set your Windows PC and email software up to be able to use it.

Step 3 - configure the hosts file on your Windows PC
In order for your PC to know that traffic for a certain destination has to go via a local port, you have to change the hosts file to override the DNS it uses to work out the IP address (e.g. 192.168.23.5) for a particular hostname (e.g. smtp.example.com). In Windows, you do this by editing the hosts file, which is normally C:\WINDOWS\system32\drivers\etc\hosts in Windows XP, and similar in other Windows OSes). Edit this (in Notepad or Wordpad) file to add a “localhost” line entry for your chosen remote machine:

127.0.0.1 localhost smtp.example.com

Now save this file (making sure it doesn’t become “hosts.txt” instead of just plain “hosts”). Your PC now thinks that “smtp.example.com” is a local machine (known as “localhost”) - when you send traffic to it on port 3025 the SSH tunnel will pick it up and forward it to the correct destination.

Step 4 - configure your email program
Now that you’ve configured your PC to think that smtp.example.com is local (or localhost), and Putty to forward all traffic on port 3025 to the remote destination you specified, you can set up your mail client to send traffic for smtp.example.com to port 3025 instead of port 25. Most mail clients will allow you to define the port that your SMTP server uses (I’m using the M2 client which comes with the free Opera browser) - just go in and change the SMTP server entry settings from port 25 to port 3025.

Now you can get everything running - start the SSH session to the SSH server using Putty, and when you send mail your email client will send it to smtp.example.com via your SSH session - great if port 25 access is restricted from where you are, but SSH is open.

The only thing to watch out for is that you can only have port 3025 running once on your PC - if you want to connect to multiple SMTP servers (for example) use other local ports such as 4025, 5025 etc etc.

Enjoy!

Having signed up and received my key, I tried a few things out with the mapping service last night (and had to join their mailing-list to get a couple of things sorted out.

This is what I’ve come up with so far - putting markers on a map of my choosing.

The main disadvantage so far is that there’s no way to specify a map which specifies longitudes (West and East) for its left- and right-hand edges, and latitudes (North and South) for it’s top and bottom. Instead, you have to do this by centering the map on a latitude and longitude, then choose a zoom level appropriate for the number of markers you have on the map. Oh well, guess I can do something with it…