Updated to Ubuntu 16 and WordPress 4.7

Been fighting for about 9 hours since I decided to update to Ubuntu 16.04.1 LTS. That also brought up MySQL to 5.7.16, and crappers they did a whole security password thing that made killed all the current passwords.  So now I had to go in and drop users and recreate them so I could grant privileges back to the WordPress databases.

Docker got all thrashed out so I have to recreate all my machines from the devicemapper storage engine to aufs.  At least I think that is the problem. That’s a problem for another day. And that’s how I got to writing more in my blog.

Using ModPerl to Keep Zencart and phpBB2 Hackers Away

I’ve seen a bunch of attempts to hack at my zencart admin area and an old phpBB2 install that is no longer there. Both of these don’t work, and just return a 404 error, but it’s kind of annoying to keep spending compute cycles on this sort of behavior. Besides, now that I have installed the statsd and graphite packages, I am set up to provide a graph of whatever I want. So why not track the hackers, lock them out, and provide a graph of activity at the same time? So here goes.

2016-12-28 edit: see my github repo at https://github.com/dminear/apache_hack_check

I also run mod_perl in the Apache2 server, so I wanted to hook in to the PerlAccessHandler and do the work. The following is the Apache2 config lines:

        PerlModule ModPerl::DanHandler
        <Location />
                #SetHandler perl-script
                PerlAccessHandler ModPerl::DanHandler
        </Location>

And the following is the Perl module that is in the /usr/local/lib/site_perl/ModPerl directory on my Ubuntu machine:

package ModPerl::DanHandler;
use strict;
use warnings;
use FileHandle;
use IO::Socket::INET;

use Apache2::Log;
use Apache2::RequestRec ();
use Apache2::Connection ();

use Apache2::Const -compile => qw(FORBIDDEN OK :log);

BEGIN {
mkdir "/tmp/bad_ips";
chmod 0777, "/tmp/bad_ips"
}

my $sock = IO::Socket::INET->new(PeerPort => 8125,
				PeerAddr => '127.0.0.1',
				Proto => 'udp');

sub handler {
	my $r = shift;

	my $str = $r->connection->remote_ip();

	# if there is an attempt to access "zencart/admin", then put the ip on the block list
	if ($r->unparsed_uri() =~ /zencart\/admin$/ ||
		$r->unparsed_uri() =~ /zencart\/+admin\/+/ ||
		$r->unparsed_uri() =~ /phpbb2/i ) {
		#$r->log_error("BAD IP: $str");
		$sock->send( "hacker.unparsed_uri." . $r->unparsed_uri() . ":1|c\n" ) if defined $sock;
		my $fh = FileHandle->new( "> /tmp/bad_ips/$str");
		if (defined $fh) {
			print $fh "dummy";
			$fh->close;
		}
	}

	# check the block list
	if (-e "/tmp/bad_ips/$str") {
		$sock->send( "request.blocked:1|c\n" ) if defined $sock;
		return Apache2::Const::FORBIDDEN;
	} else {
		$sock->send( "request.allowed:1|c\n" ) if defined $sock;
		$sock->send( "request.hostname." . $r->hostname() . ":1|c\n" ) if defined $sock;
		return Apache2::Const::OK;
	}
}
1;

So now when someone comes in and tries to access /zencart/admin (or some gyrations thereof), the IP address gets stored in the tmp directory as a filename. On every request, the remote IP address is checked, and if found returns a 403 Forbidden response. The nice thing is that this happens for any request thereafter.  Because it’s early in the request stage, there’s not too much overhead. Plus I get the satisfaction of watching the banned IP addresses grow.

Then there’s some logic to update the statsd server based on good or bad requests. Here’s a screen capture of it in action (click on the image to enlarge):

Monitoring Graph

Processing Quicken QFX Files

I recently wanted to download my bank transactions from E*Trade, but they do not support Excel format for the bank statements (only brokerage).  You can, however, download 3 months of Quicken QFX data at a time.  I downloaded several files spanning the year-to-date, but then needed a program to parse up the files and output a tab delimited text file.  So, for a try in formatting code, here’s what I got:

#!/usr/bin/perl -w
# Dan Minear
# 2011-09-01
#
# process Quicken QFX files and output tab delimited text file
#
# call 

use strict;
use FileHandle;
use Data::Dumper;

my $data = {};
my $capture = 0;		# don't capture
my %txntypes = ();

if (@ARGV < 1) {
	die "Syntax:  $0 \nExample:  $0 09*.QFX\n";
}

my $fo = FileHandle->new( "> out.txt");
if (! defined $fo) {
	die "Cannot write file";
}

while (my $fname = shift) {
	print $fname . "\n";
	my $fi = FileHandle->new("< $fname");
	if (defined $fi) {
		my $txn = {};
		while(<$fi>) {
			chomp;
			chop;
			# look for  
			if (/^/) {	#start of record
				$capture = 1;
				$txn = {};
				next;
			}
			if (/<\/STMTTRN>/) {	# end of record
				$capture = 0;
				# add to data hash
				if (defined $data->{$txn->{FITID}}) { 
					print "FITID $txn->{FITID} already defined\n";
				} else {
					$data->{$txn->{FITID}} = $txn;
					$txntypes{$txn->{TRNTYPE}} = 1;
				}
			}
			if ($capture) {
				/<(\w+)>(.+)$/;
				$txn->{$1} = $2;	
			}
		}
	} else {
		die "cannot open $fname for reading";
	}
	$fi->close;
}

print "there are " . keys(%$data) . " transactions in files\n";

#print $fo Dumper( $data );
print $fo "DATE\tTYPE\tCREDIT\tDEBIT\tNAME\tMEMO\n";
foreach my $i (keys(%$data)) {
	my $t = $data->{$i};	# ref to hash
	my $date = $t->{DTPOSTED};

	if ($t->{TRNAMT} < 0) {	# it's a debit
		print $fo substr($date,0,4) . "-" . substr($date,4,2) . "-" . substr($date,6,2) . "\t" .
			$t->{TRNTYPE} . "\t" .
			"\t" .
			$t->{TRNAMT} . "\t" .
			$t->{NAME} . "\t" . 
			$t->{MEMO} . "\n";
	} else {	# it's a credit
		print $fo substr($date,0,4) . "-" . substr($date,4,2) . "-" . substr($date,6,2) . "\t" .
			$t->{TRNTYPE} . "\t" .
			$t->{TRNAMT} . "\t" .
			"\t" .
			$t->{NAME} . "\t" . 
			$t->{MEMO} . "\n";
	}
}

$fo->close;

=head1 uncomment to print out transaction types
foreach (keys(%txntypes)) {
	print $_ . "\n";
}
=cut

Automating my sprinklers

I have been spending a few hours creating a sprinkler controller.  The basic structure is to have a Perl program do the hard core logic, and then issue a webservice request to trigger the sprinkler controller on for a watering cycle.  So far, I have the embedded controller ready to interface to a hacked sprinkler controller .  The controller just takes a simple command to water a zone for so many seconds.  There is no higher level sequencing that is in place yet.  In reality, I probably need to send a sequence to the embedded controller so the power supply only powers one water solenoid at a time. The alternative is  to put that logic in the Perl program, so that’s where I am now.

I have been working on the Perl program to figure out my watering algorithm.  So far, I just make a request to a NOAA weather site for an airport  about 2 miles from me. The document is slurped as XML data that XML::Simple throws into a hash.  From there, I take the relative humidity and integrate the “dryness”,  or 100 minus the humidity, for the given time span.  I have decided to water my grass every 3 days as a ballpark until I see how things are working. The program just prints out a trigger message for now.  In the future, this will spawn off a sequence of webservice requests to water the lawn.

So far, we have been having some interesting weather.  There have been off-and-on rain storms coming through, and this week is setting up for a rapid change into hotter weather.  A weather description that matches “Rain” will cause a reset of the integration so the lawn is not watered.  The script logs some pertenant data to the end of the script so I can rerun a scenario in the future with real world data.

A side product of this project is to tie it to my Doggie Dumpcam, which captures realtime image changes on my front lawn.  The tree is growing bigger lately, which is creeping outside of the mask area and triggering a lot of image captures.  Some interesting things have been captured, including dogs, birds, and hornets.  This Perl program will have the capability to trigger a particular sprinkler zone between, say, the 6am and 8am hours for a small timespan.

Mom & Dad’s 50th Wedding Anniversary

Today our family is celebrating Mom and Dad’s 50th wedding anniversary.  We are trying to put together something to say at the gathering, which is taking place at my sister Pam’s house.  There will be five out of the eight kids there. I don’t know if we’ve had all the kids there together for some time.  It is quite difficult, as my brother and sisters are spread apart.  I have siblings in the Netherlands, France, and the rest are in the US.  In the US, we are in Ohio, Oregon, and California.  Today Netherlands, France, and Oregon will not be there.

So back to the topic at hand, we have been trying to figure out what we want to say. We decided on having each sibling say a bit or two about what they want — a story, what mom and dad have taught them, or some memory we have had.

So I don’t know where to begin about this.  I mean, our family really doesn’t talk about our emotions, so I find it difficult to talk about how I feel.  Better stick to a story, I suppose. Past, present, and future — maybe use that as a guide.

I think overall my parents have shown me the value of family. I haven’t noticed things that were in our family until I grew older and began to see how other families related, or had to make decisions on how I wanted my family  to relate.  For example, take dinnertime.  For the most part, we had a family dinner together.  Sure, there were hectic times when older sisters were working or had a school activity, but for the most part, I don’t remember not having a dinner with everybody there. The  other thing I remember was that my mom cooked the meals.  Nowadays, I look back and think how difficult that must have been.  And there were plenty of times where I was “What are we having for dinner?  Yuck!”  Now being a parent, I didn’t know what I was saying back then.

Family finances were always lumped together.  There were not separate bank accounts. I remember lots of nights where Dad had that green fluorescent digit calculator and was working bills. Once again, when I started seeing other couples that have separate accounts I had to choose how I wanted finances to be in my life.

Spiritual development was another thing shown to us by our parents. There were plenty of times when I could not stand to go to mass when I was young, but I never won that battle, despite my kicking heels upside-down on the wall.  I guess I don’t consider our family overly spiritual, but faith is an intrinsic part of who we are.  And once again, I did not really see that until after I was away and saw how much that is a part of our family.

So here we are today, celebrating 50 years.  I think we are here for a lot of reasons, but overall the lessons I have learned are to give more than you take, celebrate the good, discount the bad, and head into the garage when you have to.

Honda Civic Hybrid & Toyota Software

I came across an article this evening, and based on an experience this afternoon, it made me think. How much software testing is enough?

This all started out this afternoon.  A couple of times a week I go out to lunch with my friend Chris, and we spend time talking about all kinds of different things.  Sometimes we find amusing things so we like to share them.  This afternoon, Chris was showing me that he stumbled across the speech recognition of his Honda Civic Hybrid.  He thought that it was mainly to control the GPS navigation, but it turns out that it does a lot more than that.  We found ourselves trying to command “radio, 88.2” (nonsensical FM frequency) or “radio, 88.5” (it only goes to 88.7) to see the response.  It turns out that you can also turn on/off the air conditioner.

This banter quickly degenerated to “damn car, tune the radio to 870!” which amazingly tuned the radio to 870AM.  I tried yelling “Eject! Eject!” but nothing happened.  For a brief instant, I was considering “set speed to maximum,” but I did not say it, just in case. The Honda Hybrid does not have a real throttle cable; just a potientiometer that is hooked to the gas pedal. This is essentially an input to the computer, which controls the electric motor & gas engine.

So, it’s quite possible that there could be a software bug, and some condition might cause a seriously undesirable side effect.  Thus, the above article. Now, I’m not saying that’s what happened, but it makes you wonder. Is there a governing computer that independently shuts down the system when the wrong outputs are seen?  If you peer into a traffic signal computer box, you will find one computer that does the traffic signals, and another that is monitoring everything and in the slightest instance of trouble (say, green lights for perpendicular directions), it shuts the thing down into flashing reds for all directions, and requires human intervention to reset. I think that’s why you tend to see flashing reds in rainstorms — things get wet, currents flow where they should not, and the governing computer senses some malfunction state.

So how do you solve this?  Test, test, and more test. The ECZ space station code was path tested for every possible execution path.  That’s a lot of testing, but a manned space program requires it.  That’s a lot of cost, too. A huge amount. So, what do you do if you have a competitive environment with fast turnaround times and new models always coming out due to customer demand? How far does the testing get taken? When is the test done?

Facebook is killing my green computer

So, I can’t help but see that all the Javascript/Flash/Whatever stuff running on a Facebook page is keeping the CPU utilization up, preventing the savings that supposed to be happening while my computer is idle.

Well, actually, it’s Cindy’s computer.  It is a 24 inch iMac that seems to get really hot on top, especially on the top left corner.  So hot that it is sometimes painful to touch.  Yup, that hot.

The thing that keeps it that hot is that I find it sitting on a few Facebook pages after she has left the thing and gone somewhere else.  It has the auto screen dimmer going (the CCFL tubes contribute a lot to the heat when the screen is on), but still remains hot.  The best I can see is that all the scripts still running on the Facebook pages keep the CPU going, and that is just, — stupid.

Such is life.  Just when you think things are getting more efficient on the server side, the client side is spiraling out of control.  You just can’t win.

So here’s the call to all browsers!  Put some code it them that times out after a predetermined time and shuts down the useless processing.  It’s a power saver for the browser.  After 10 minutes, just kill the dumb threads that are just spinning doing stupid stuff.

Such a lot of Pain on the Earth

Wow, I’ve had the recurring idea of the multitude of pain on the earth, and some time ago Rush came out with Snakes and Arrows. I don’t know if I wrote about the  May 11, 2008 concert at Irvine Meadows (or Verizon Wireless whatever its called).

I came across this post, and I could not have said it better.