Backup to S3 using duplicity

I recently put a nice big hard drive in my file server.  I got a 1TB drive, which is kind of silly because I don’t have anywhere near that much data.  But, I figured that it’s better to be prepared.  I wanted to backup the data on the drive and Amazon’s S3 seemed like a good choice.  However, I wanted the ability to only send changed files across the wire, and a simple way to encrypt all of the data as well.  I chose to use Duplicity to manage the backups as it was able to only send changes as well as encrypt everything using GPG.

I decided to only backup a subset of my data since it didn’t seem necessary to backup Linux ISOs or the local copy of Wikipedia that I have on this drive.  So, I ended up backing up 31.2 GB of data.  Here’s the stats that duplicity produced:

--------------[ Backup Statistics ]--------------
StartTime 1244688149.16 (Wed Jun 10 22:42:29 2009)
EndTime 1244842521.31 (Fri Jun 12 17:35:21 2009)
ElapsedTime 154372.15 (42 hours 52 minutes 52.15 seconds)
SourceFiles 0
SourceFileSize 33486993762 (31.2 GB)
NewFiles 0
NewFileSize 0 (0 bytes)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 0
RawDeltaSize 33484666346 (31.2 GB)
TotalDestinationSizeChange 32831751689 (30.6 GB)
Errors 0
-------------------------------------------------

Yeah, just in case you didn’t notice.  The elapsed time was nearly 43 hours.  That’s just under two full days.  The 15/2 service that I’m getting from Cablevision was clearly the limiting factor.  But, I don’t plan on doing full backups very often, and the incrementals will be considerably smaller.

Amazon charges for S3 by the amount of storage you actually use.  They charge $.15/GB/month.  So, for 31.2 GB I’ll be paying $4.68 per month.  Not bad for a remote, redundant backup.  They also charge for bandwidth which is usually $.10/GB, but it’s currently on sale for $.03/GB.  So, the upload only cost $.93.

I recently built a birdhouse for our backyard.  I went down to the library and took out “The Backyard Birdhouse Book” by René and Christyna M. Laubach.  I was very happy with this book.  It gave a lot of good, practical details about building birdhouses that actually benefit birds.  If you want to build a birdhouse that looks like a drug store, this isn’t the book for you.  It went in to detail about what type of birds lived in my area, what they needed from a birdhouse, and then had in-depth instructions on how to build the appropriate bird house (which they call nestboxes in the book).

After deciding on the house that I was going to build I picked up some lumber and borrowed my father’s table saw to cut out all of the pieces.  I then assembled it, sanded it, and painted it.  I chose to use a “Noel Wire Racoon Guard” to protect the residents from predators like squirrels, racoons and cats.  Okay, so on to the pictures.

This is a shot of the completed birdhouse.  You can see the noel wire racoon guard, which is the green vinyl covered hardware cloth that encircles the entrance.  The ends of the guard have had the vinyl removed and been snipped on an angle to make them sharp.  This should make it very uncomfortable for predators trying to gain access to the bird house.

Birdhouse

The roof of the birdhouse can be removed so you can monitor the birds inside without disturbing them very much.

Birdhouse with roof removed

One of the sides pivots open to make it easy to clean out the previous year’s nest every Spring.

Birdhouse with side opened

Here’s a close-up of the birdhouse after it was mounted in the tree.

Birdhouse in tree close up

Finally, here’s a wide-angle shot of the birdhouse in the tree.

Birdhouse in tree

I’m really very happy with how the birdhouse came out.  I can’t wait until a bird moves in.  It is probably too late in the season to get birds moving in to it this year.  But, it will be weathered and ready for next year.  If birds do move in I’ll be sure and post some pictures.

Oh, and by the way, the title of this post is a reference to a clip from the Family Guy episode titled “I dream of Jesus” where Peter keeps singing Surfin’ Bird.  It’s one of my favorites.

Relay for Life

My family and I are going to be participating in the Relay for Life event in our town this weekend. Relay for Life is an event that raises money for the American Cancer Society. This description is from the Relay for Life website:

“Relay For Life is the American Cancer Society’s signature activity. It offers everyone in a community an opportunity to participate in the fight against cancer. Teams of people camp out at a local high school, park, or fairground and take turns walking or running around a track or path. Each team is asked to have a representative on the track at all times during the event. Relays are an overnight event, up to 24 hours in length.”

By going to my relay for life web page you can donate money to the Relay for Life cause or purchase a luminaria.  Luminaria are illuminated bags that are placed around the track to remember people who have lost their lives to cancer and honor those who have won their battle.

If you can, please go to my relay for life web page and donate some money or purchase a luminaria.  You can donate as little as $5, a Luminaria is just $10.  Every little bit helps.  Thank you.

Bonuses

So, the whole country is in an uproar over these AIG bonuses.  According to Wikipedia, AIG has received over $170 billion in bailout money.  The bonuses that have been paid out total $165 million, although some sources are reporting the total to now be closer to $218 million.

Now, assuming the worst case of $218 million dollars, only .12% of the tax-payer money went to bonuses.  I should point out that the number is not twelve percent, but point-twelve percent.  That’s just slightly more than one tenth of one percent.

Ideally, none of the tax-payer money would have gone to bonuses that were contingent on performance.  Clearly everyone with any real level of responsibility at AIG under-performed at best.  But, should this really be monopolizing the national dialogue?  Does the average news-watching American know where 99.88% of the bailout money went or what purpose it served?

This XKCD comic points out the problem I’m having with the media:

XKCD 1000 times comic

Declaring Blog Bankruptcy

There’s this idea of ‘Email Bankruptcy‘ where after being inundated with incoming email and not having the time to reply to each one you just announce that you’re deleting them all and promise to be more vigilant in the future.  Well, I’m declaring blog bankruptcy.  I haven’t posted here in over 6 months.  There have been many interesting and blog-post-worthy events in my life these past 6 months, and I just can’t seem to find the time to give them the blog posts that they deserve.  So I’m going to skim over each of them, get up-to-date, and thereby declare blog bankruptcy.

House

Jen and I bought a house.  We looked at a whole lot of houses, got the mortgage, made an offer, closed, moved in, and fixed up the joint.  We’re really happy here.  We love the neighborhood and the school district.  We’re on a dead-end street which is wonderful.  We’ve got 4 bedrooms, 1 bathroom, a basement, a garage, and a decent sized yard.  To sum it up, we’re thrilled.

Wedding

Jen and I got married.  After a year of being engaged we tied the knot.  Jen planned a beautiful wedding.  I helped where I could, but she really deserves all of the credit.  We got married at the Riviera in Massapequa.  It was wonderful.  It was great to be there with our families and friends.  It was a great ceremony, even though the officiant got my name wrong once.  :o)  Jen was so beautiful.  I mean, absolutely breath-taking.  The reception was wonderful as well.  Everyone was dancing around and just having a great time.

Honeymoon

Jen and I went on our honeymoon.  We went to St. Lucia and it was absolutely breath-taking.  We stayed in a beautiful suite right near the water.  We went swimming a lot of the time.  The food was just amazing.  We went scuba-diving which was awe-inspiring.

Alright, that’s the end of my blog bankruptcy post.  Hopefully now I’ll be able to post shorter updates for less significant events in my life.  I felt like I couldn’t post anything before posting about these major life events.  I feel bad for the atrocity that I’ve committed here – posting 1 paragraph about buying a house, our wedding, and our honeymoon.  But it had to be done.  I’m sure I’ll get pictures uploaded somewhere soon and I’ll link to them here.  Ahh – I feel like a weight has been lifted from my shoulders.

rPath, Foresight, Conary

I first started speaking about rPath Linux and their Conary package management tool a little over a year ago.  But, I just want to get it down in writing that I think this is the future of linux packaging and indeed distribution management in general.  They took traditional package management (think rpm w/ yum or dpkg w/ apt) and made it look antiquated.  That’s right, apt-get is not the end of the line for package managers.

Conary really takes the problem of managing the software that is installed on your system and fully maps it out.  I feel like they broke the problem down in to small chunks and solved each one of the problems well.  rPath’s main goal is to create tools and a distribution that can be customized and modified by down-stream distributors to create software appliances.  There are a lot of features designed for this task in Conary, but they can benefit other users who may not feel like they have their own customized distribution.  For example, do you have a farm of machines that are running the same software with your own customized configuration files?  Why not create a package that shadows the upstream apache package for example, but includes your own configuration files in the apache:conf component?

One of the other wonderful features of Conary is that the repository is kept in version control and branches are supported by all of the supporting tools.  Imagine if Debian was using a tool like Conary to track and manage their entire distribution.  All of the Debian derivative distributions would be able to have their own branches of the distribution.  This would make it a lot easier for derivative distributions to use Debian’s existing packages and for them to make contributions back to Debian.

There is one problem with the Conary story.  All of the server-side distribution management tools are closed-source.  I do see this as a show-stopper.  Well, a lot of things are using rPath and Conary, and rPath as a company seems to be doing alright, so maybe ’show-stopper’ is a bit harsh.  But, this will absolutely keep other major distributions and large chunks of the open-source community from getting on the bandwagon.  All I can say is that I hope this will eventually be remedied by the open-sourcing of the server-side components.  The technology feels like such a good thing for the community and for distributions in general, but it’s the kind of thing that can not be entrusted to one company.  Hopefully someday rPath will be willing/able to release the server-side stuff.

Regardless, the feature-set exposed by Conary is, to me, the clear future of package management.  Whether it ends up being Conary that makes it big or some other (possibly completly open source) implementation is yet to be seen.  But I wanted to get this article out there with a date on it so that when it happens I can say, “I called it!”  If you want to play around with Conary on your desktop I would highly recommend Foresight Linux.  It’s pretty bad ass.

I just upgraded my wordpress installation to 2.5.1 from 2.3.3 or something like that.  2.5 is very nice.  The upgrade was really simple and the new administrative interface is pretty slick.  I decided it was time for me to manage my wordpress installation with subversion so that later upgrades are a simple ’svn switch’ away.  The process of switching was pretty easy because I had scripts in place for backing up and restoring all of the custom parts of the site.

As part of the upgrade I had to make sure that all of the plugins I was using were moved over properly.  The one plugin that I really use is the openid delegation plugin.  This plugin lets you use your wordpress URL when logging in to a site using openid.  All it does is put extra <link> tags in the <head> of your home page that redirect the open id request to your real open id server.  I use claimid.com for this purpose.

I had made a change to the plugin at one point because it would not put the tags on my home page because my home page is a ‘page’ and not a ‘post’ in wordpress terms.  When I originally made the change I sent a patch over to the original author hoping that he would incorporate the change and put the improved version on his page.  He never did.  While moving the plugin today I almost over-wrote my changed file with his original.  Now, my changes amount to 6 or 7 lines of code, but I really didn’t want to have to go finding out which wordpress objects and hooks I had to use to recreate my change.  To avoid this problem in the future I decided to throw it in to my repository.  So, if you use wordpress and you’re looking in to open id, check out the slightly improved openid delegation plugin for wordpress.

It was again time for me to upgrade wordpress, so why not write up a post? Besides, Sid has been begging people to join planet.lilug.org. But we’ll get more posts on planet if we add more people and if those of us on planet post more often. So, here it is.

Lilug

John Kuczewski and Quarsaw brought up the idea of having Lilug tee shirts on the mailing list the other day and sufficiently guilted me in to finishing the work I had started on the Lilug cafe press store. $2 from every apparel purchase gets donated to Lilug. So go, shop, be merry. Buy yourself a Lilug tee-shirt with a Lilug button to pin to it. All of the cool kids are doing it. Lilug shirts are really ‘in’ this season.

LugRaffle

I’ve put a bunch of time in to the LugRaffle project that we’re doing at Lilug Dev-Sig. It’s coming along pretty nicely and is really fun to hack on. It’s to the point where it does ‘things’, but it still isn’t useful for holding a raffle. That’ll come soon enough. I welcome everyone to check it out and play around with it. You can browse the code via gitweb here: http://git.markdrago.com/git or clone it from here: http://git.markdrago.com/git/lugraffle .

House

So Jen and I are buying a house. It’s in Lindenhurst, at the end of a dead-end street, and in the Alleghany elementary school zone. It’s got four bedrooms, 1 newly renovated bath, a garage, a basement, and a decent sized yard. We’re pretty excited. We recently went to contract and we plan on closing in July. The current owners don’t want to move until the Summer, because they have kids in school, which is fine with us.

Typically Great Exchange

Jason and I have interesting exchanges on IRC:

 13:39 < kupo> I just registered URMOMZ.com
 13:40 < markdrago> kupo: nice
 13:40 < kupo> markdrago: I feel like a million bux
 13:40 < markdrago> yeah - hold on to that one forever.

				

Live Masonry

We had another live airing of the Free Software Round Table last Saturday night.  One of the big news items in the Linux world was that Ubuntu had released the 7.10 version of their distribution, which is code named Gutsy Gibbon.  I had been running the previous version of Ubuntu on my laptop and decided that it would be kind of fun to upgrade to the next version while we were on the air.  So, the first thing we did at the beginning of the show was talk about Gutsy and then I started upgrading.  It had to download 1.1GB of updates (which took about an hour) and then it started installing the updates (which would have taken about 2 hours).  However, after 1 hour of upgrading we were about to finish the show and head home.  So, I restarted my machine right in the middle of the update.  It rebooted, but a slew of things were broken.  It’s a little embarrassing to be live on a Linux radio show while you ‘brick‘ your Linux laptop.  However, last night I was able to run ‘dpkg –configure -a’ to finish installing the packages that it had half installed, and then run ‘aptitude dist-upgrade’ to finish the rest of the installation.  The system is now up and running just fine.  Besides the bricking incident the radio show went really well.  I don’t think we’ll be doing another 2 hour show anytime soon, but the hour that we actually had content prepared for went really well.

Next »