Updated to Ubuntu 16 and WordPress 4.7

Been fighting for about 9 hours since I decided to update to Ubuntu 16.04.1 LTS. That also brought up MySQL to 5.7.16, and crappers they did a whole security password thing that made killed all the current passwords.  So now I had to go in and drop users and recreate them so I could grant privileges back to the WordPress databases.

Docker got all thrashed out so I have to recreate all my machines from the devicemapper storage engine to aufs.  At least I think that is the problem. That’s a problem for another day. And that’s how I got to writing more in my blog.

Using ModPerl to Keep Zencart and phpBB2 Hackers Away

I’ve seen a bunch of attempts to hack at my zencart admin area and an old phpBB2 install that is no longer there. Both of these don’t work, and just return a 404 error, but it’s kind of annoying to keep spending compute cycles on this sort of behavior. Besides, now that I have installed the statsd and graphite packages, I am set up to provide a graph of whatever I want. So why not track the hackers, lock them out, and provide a graph of activity at the same time? So here goes.

2016-12-28 edit: see my github repo at https://github.com/dminear/apache_hack_check

I also run mod_perl in the Apache2 server, so I wanted to hook in to the PerlAccessHandler and do the work. The following is the Apache2 config lines:

        PerlModule ModPerl::DanHandler
        <Location />
                #SetHandler perl-script
                PerlAccessHandler ModPerl::DanHandler
        </Location>

And the following is the Perl module that is in the /usr/local/lib/site_perl/ModPerl directory on my Ubuntu machine:

package ModPerl::DanHandler;
use strict;
use warnings;
use FileHandle;
use IO::Socket::INET;

use Apache2::Log;
use Apache2::RequestRec ();
use Apache2::Connection ();

use Apache2::Const -compile => qw(FORBIDDEN OK :log);

BEGIN {
mkdir "/tmp/bad_ips";
chmod 0777, "/tmp/bad_ips"
}

my $sock = IO::Socket::INET->new(PeerPort => 8125,
				PeerAddr => '127.0.0.1',
				Proto => 'udp');

sub handler {
	my $r = shift;

	my $str = $r->connection->remote_ip();

	# if there is an attempt to access "zencart/admin", then put the ip on the block list
	if ($r->unparsed_uri() =~ /zencart\/admin$/ ||
		$r->unparsed_uri() =~ /zencart\/+admin\/+/ ||
		$r->unparsed_uri() =~ /phpbb2/i ) {
		#$r->log_error("BAD IP: $str");
		$sock->send( "hacker.unparsed_uri." . $r->unparsed_uri() . ":1|c\n" ) if defined $sock;
		my $fh = FileHandle->new( "> /tmp/bad_ips/$str");
		if (defined $fh) {
			print $fh "dummy";
			$fh->close;
		}
	}

	# check the block list
	if (-e "/tmp/bad_ips/$str") {
		$sock->send( "request.blocked:1|c\n" ) if defined $sock;
		return Apache2::Const::FORBIDDEN;
	} else {
		$sock->send( "request.allowed:1|c\n" ) if defined $sock;
		$sock->send( "request.hostname." . $r->hostname() . ":1|c\n" ) if defined $sock;
		return Apache2::Const::OK;
	}
}
1;

So now when someone comes in and tries to access /zencart/admin (or some gyrations thereof), the IP address gets stored in the tmp directory as a filename. On every request, the remote IP address is checked, and if found returns a 403 Forbidden response. The nice thing is that this happens for any request thereafter.  Because it’s early in the request stage, there’s not too much overhead. Plus I get the satisfaction of watching the banned IP addresses grow.

Then there’s some logic to update the statsd server based on good or bad requests. Here’s a screen capture of it in action (click on the image to enlarge):

Monitoring Graph

Processing Quicken QFX Files

I recently wanted to download my bank transactions from E*Trade, but they do not support Excel format for the bank statements (only brokerage).  You can, however, download 3 months of Quicken QFX data at a time.  I downloaded several files spanning the year-to-date, but then needed a program to parse up the files and output a tab delimited text file.  So, for a try in formatting code, here’s what I got:

#!/usr/bin/perl -w
# Dan Minear
# 2011-09-01
#
# process Quicken QFX files and output tab delimited text file
#
# call 

use strict;
use FileHandle;
use Data::Dumper;

my $data = {};
my $capture = 0;		# don't capture
my %txntypes = ();

if (@ARGV < 1) {
	die "Syntax:  $0 \nExample:  $0 09*.QFX\n";
}

my $fo = FileHandle->new( "> out.txt");
if (! defined $fo) {
	die "Cannot write file";
}

while (my $fname = shift) {
	print $fname . "\n";
	my $fi = FileHandle->new("< $fname");
	if (defined $fi) {
		my $txn = {};
		while(<$fi>) {
			chomp;
			chop;
			# look for  
			if (/^/) {	#start of record
				$capture = 1;
				$txn = {};
				next;
			}
			if (/<\/STMTTRN>/) {	# end of record
				$capture = 0;
				# add to data hash
				if (defined $data->{$txn->{FITID}}) { 
					print "FITID $txn->{FITID} already defined\n";
				} else {
					$data->{$txn->{FITID}} = $txn;
					$txntypes{$txn->{TRNTYPE}} = 1;
				}
			}
			if ($capture) {
				/<(\w+)>(.+)$/;
				$txn->{$1} = $2;	
			}
		}
	} else {
		die "cannot open $fname for reading";
	}
	$fi->close;
}

print "there are " . keys(%$data) . " transactions in files\n";

#print $fo Dumper( $data );
print $fo "DATE\tTYPE\tCREDIT\tDEBIT\tNAME\tMEMO\n";
foreach my $i (keys(%$data)) {
	my $t = $data->{$i};	# ref to hash
	my $date = $t->{DTPOSTED};

	if ($t->{TRNAMT} < 0) {	# it's a debit
		print $fo substr($date,0,4) . "-" . substr($date,4,2) . "-" . substr($date,6,2) . "\t" .
			$t->{TRNTYPE} . "\t" .
			"\t" .
			$t->{TRNAMT} . "\t" .
			$t->{NAME} . "\t" . 
			$t->{MEMO} . "\n";
	} else {	# it's a credit
		print $fo substr($date,0,4) . "-" . substr($date,4,2) . "-" . substr($date,6,2) . "\t" .
			$t->{TRNTYPE} . "\t" .
			$t->{TRNAMT} . "\t" .
			"\t" .
			$t->{NAME} . "\t" . 
			$t->{MEMO} . "\n";
	}
}

$fo->close;

=head1 uncomment to print out transaction types
foreach (keys(%txntypes)) {
	print $_ . "\n";
}
=cut

Automating my sprinklers

I have been spending a few hours creating a sprinkler controller.  The basic structure is to have a Perl program do the hard core logic, and then issue a webservice request to trigger the sprinkler controller on for a watering cycle.  So far, I have the embedded controller ready to interface to a hacked sprinkler controller .  The controller just takes a simple command to water a zone for so many seconds.  There is no higher level sequencing that is in place yet.  In reality, I probably need to send a sequence to the embedded controller so the power supply only powers one water solenoid at a time. The alternative is  to put that logic in the Perl program, so that’s where I am now.

I have been working on the Perl program to figure out my watering algorithm.  So far, I just make a request to a NOAA weather site for an airport  about 2 miles from me. The document is slurped as XML data that XML::Simple throws into a hash.  From there, I take the relative humidity and integrate the “dryness”,  or 100 minus the humidity, for the given time span.  I have decided to water my grass every 3 days as a ballpark until I see how things are working. The program just prints out a trigger message for now.  In the future, this will spawn off a sequence of webservice requests to water the lawn.

So far, we have been having some interesting weather.  There have been off-and-on rain storms coming through, and this week is setting up for a rapid change into hotter weather.  A weather description that matches “Rain” will cause a reset of the integration so the lawn is not watered.  The script logs some pertenant data to the end of the script so I can rerun a scenario in the future with real world data.

A side product of this project is to tie it to my Doggie Dumpcam, which captures realtime image changes on my front lawn.  The tree is growing bigger lately, which is creeping outside of the mask area and triggering a lot of image captures.  Some interesting things have been captured, including dogs, birds, and hornets.  This Perl program will have the capability to trigger a particular sprinkler zone between, say, the 6am and 8am hours for a small timespan.

Honda Civic Hybrid & Toyota Software

I came across an article this evening, and based on an experience this afternoon, it made me think. How much software testing is enough?

This all started out this afternoon.  A couple of times a week I go out to lunch with my friend Chris, and we spend time talking about all kinds of different things.  Sometimes we find amusing things so we like to share them.  This afternoon, Chris was showing me that he stumbled across the speech recognition of his Honda Civic Hybrid.  He thought that it was mainly to control the GPS navigation, but it turns out that it does a lot more than that.  We found ourselves trying to command “radio, 88.2” (nonsensical FM frequency) or “radio, 88.5” (it only goes to 88.7) to see the response.  It turns out that you can also turn on/off the air conditioner.

This banter quickly degenerated to “damn car, tune the radio to 870!” which amazingly tuned the radio to 870AM.  I tried yelling “Eject! Eject!” but nothing happened.  For a brief instant, I was considering “set speed to maximum,” but I did not say it, just in case. The Honda Hybrid does not have a real throttle cable; just a potientiometer that is hooked to the gas pedal. This is essentially an input to the computer, which controls the electric motor & gas engine.

So, it’s quite possible that there could be a software bug, and some condition might cause a seriously undesirable side effect.  Thus, the above article. Now, I’m not saying that’s what happened, but it makes you wonder. Is there a governing computer that independently shuts down the system when the wrong outputs are seen?  If you peer into a traffic signal computer box, you will find one computer that does the traffic signals, and another that is monitoring everything and in the slightest instance of trouble (say, green lights for perpendicular directions), it shuts the thing down into flashing reds for all directions, and requires human intervention to reset. I think that’s why you tend to see flashing reds in rainstorms — things get wet, currents flow where they should not, and the governing computer senses some malfunction state.

So how do you solve this?  Test, test, and more test. The ECZ space station code was path tested for every possible execution path.  That’s a lot of testing, but a manned space program requires it.  That’s a lot of cost, too. A huge amount. So, what do you do if you have a competitive environment with fast turnaround times and new models always coming out due to customer demand? How far does the testing get taken? When is the test done?

Did I Tell You About My Apple?

While I’m in the writing mood, I just can’t say how much I love Apple computers.  I have gone through a Performa, grape iMac (sold at a swapmeet to nice guy), half basketball articulated monitor iMac (current print server as of tonight), a 17 inch monitor articulated basketball iMac (the kids computer as of tonight), a 24 in metal iMac (Cindy’s current workstation), and I have my MacBook laptop. I probably missed one in there.

Don’t get me wrong — I’ve had my share of PC’s.  From a $2000 Leading Edge 80286, to a 80386 clone (welcome to protected mode!), to a 486 (several running Galacticomm’s BBS software), to Pentium II, Pentium III Sharp laptop (Cindy’s computer while in the hospital with Jared & Riley’s pregnancy), to a Pentium 4 clone with a ASUS motherboard that died — I’ve had a few.  In fact, the current Ubuntu server is running on a IBM Pentium 4 workstation (which seems to be really nice) and handles the home web server and internet router (among other things).

But after the death of the clone, I just decided that I didn’t need a PC. Back before green was so cool, I saw the specs: the PC took about 110 watts, and the Macs about 40 watts. I used the PC in the upstairs to keep warm in the winter nights. It was just, so, PC !  After it died, I decided to see if I could make good with my laptop. And you know what?  It’s worked out.  Just about that time Parallels started getting good, so I bought it and it mostly worked. The one thing it did not do was have Direct-X 9.0 integration, so MS Flight Simulator didn’t work.  Major bummer!  But now the latest version does have Direct-X 9.0 support, and FS works pretty good! So good, in fact, that I don’t need a PC!  And my USB Propedals and Saitek joystick work (well, sort of — the pedals are developing an annoying full left turn intermittant — not fun in a helicopter).

So now I’m left with a hoard of Apple computers that seem to keep cranking. I didn’t plan it this way.  I just keep the things that work good and don’t break (not sure how the Isuzu Rodeo fits here).  The Time Machine backup has saved me two times now with crazy unstable systems.  Yes, Apples will at times go all crazy and get messed up. But there’s Time Machine.  Where’s that for PC’s?  I used to keep a box of PC software so when the PC got messed up I would just format the drive, load up the OS, and then crank through the application CD’s (my data was stored on a shared Apple SAMBA drive).  Time Machine just reloads the drive from where it last saved (or before, or before, or …).

The whole Apple software update just works.  Just before this (and probably the thing that spawned this post) I received a whole mess of software updates for iLife, Safari 4.0 (seems to be trying to be more like Google Chrome — cool!), and various other updates.  It’s not some service pack thing that has some crazy different functionality differences that wants to report back home about the genuine software program that —– baaahhhh!

Someday I’ll write about my love for Ubuntu. Where else can you get a firewall, mailserver, secure web server running mod-perl, Tomcat, Java SDK, and mySQL working in under a day?

GEEK OUT!

SOA Musings

Every so often, the world rolls by, and you hear bits and pieces of things.  Most often, they don’t mean too much, so you just make a mental note and that’s about it.  Sometimes, though, it totally catches you offguard, and you wonder how you managed to miss the whole boat.  SOA is one of those things. Well, maybe half the boat.

When you talk SOA, I think you are also talking Java, but folks will not tell you that.  For some extra reading, you can check out What is SOA. I haven’t yet, so I can’t say I know anything. After you figure all that out, then check out SOA Principles for some more. Then, you can check out SOA Patterns when you get really board.

See, these links are really so I can go read those things, and then I’ll try to update this post.  In my naive thinking,  SOA is just doing what you were taught.  But then you really do it that way, and don’t cut corners.  Remember all that talk about design up front, test driven development, and developing to an agreed upon interface?  That’s really what SOA is.  But I need to check the previous link to see if I’m right.

The other thought is that SOA is doing what you were not taught.  I can’t remember being taught all the things in The Pragmatic Programmer. But it seems to be full of what you should do, and other things I wasn’t taught. But hey, maybe that’s because I’m an EE.