Designing tests that scale - or what I do all day

It has been a long time since I have posted anything here, but I feel like sharing a bit of what we have been up to.  We have been working on improving the quality of our tests, and increasing what we can measure while reducing the amount of data that we need to transfer to do the measurements.  Part of this work has been developing a custom web server written in Go that uses a highly efficient packet capture mechanism (golibpcap).  All this is wrapped up in something we call the net-score diagnostic test server (nsdt-server).

Without the nsdt-server we are limited to only looking at timings exposed at the application level (HTTP) as this is all the browser gives us.  This data is great for looking at end-to-end performance that applications can expect, because it is performance at the application.  It encompases everything between the server and the client giving a complete measurement - how long did an object take to get from server to client.  Simple and useful.  But what it does not tell us is why?

A lot can happen in the 30 ms between when the server sends the data and when it actually is received by the client.  The browser tells us that an object has finished downloading only after it has arrived in full - every last bit.  Before that last bit arrives there could have been packets that were lost, sent out of order, malformed and retransmitted.  This can mean that instead of one round trip to the server to get our object we may have had to make many smaller requests to make up for mishaps along the way.  Lucky for us TCP takes care of all the messy work but knowing what lengths TCP had to go through to get a complete copy of the data can tell us a lot about possible inefficiencies in the connection.

To learn about how TCP works hard so that applications don't have to you can use tools that capture streams of packets like wireshark.  This is great if you want to see one connection (your own), but we needed something that was going to scale to global proportions.

Part of our nsdt-server is a RESTful interface for starting packet traces.  Without going into too much detail here our server can manage many traces at once and when a trace completes it sends the results to a central control server to be processed.  Even with a highly efficient server we are going to need hundreds of these spread all over the world so we also developed a decentralized load balancer to send clients to a server that has a light load.  This decentralized load balancing server also keeps track of what servers are online and allows new servers to join without any human interaction.

I keep using the word global because even in our very quiet beginning we have data coming in from six continents!  Which is great because we are trying to build a globally representative picture of broadband access, but we still need more data.  Here is an interactive sample of some of the data we have collected so far.  You can see that western Europe and the eastern United States have the best coverage, but with more reach we hope to illuminate some of the darker corners of the global broadband market (like our tests from Fiji and Bermuda).  Here is a bigger interactive view of the map below.

The hope at this point is that you are interested in getting involved.  If you have a blog or website no matter how small or obscure (actually the more obscure the better) you can help by putting a small snippet of our code in your website/blog.  We have widgets that can be installed into Google sites and Blogger, and JavaScript that can be pasted into other sites.  We would love for you to install a visible version of the tool (like in the top right corner of this page), but if you want something low-key we have a version that runs completely in the background too.  Detailed instructions can be found here, and if they are not clear or you need help we are glad to help.

Cross posted on the net-score blog.

Cloud Storage Solutions

My belabored attempt to break down the practicality of using cloud based backup solutions.  There are probably services out there that I have not mentioned but this is just a few that come to mind.


Edited 3/7/2011 to reflect Google Storage for Developers lower prices.

A Google/Facebook blackout might not kill SOPA

Abstract: At least one person (a must read post) in Washington explicitly wants to censor the Internet and our attempts to stop it may just be giving them more motivation to do it. 

This is the kind of thing that I could do without knowing, but it kind of confirms my thoughts on the subject.  SOPA and PIPA may have started out with big money from the entertainment industry as a way to protect an aging business model, but soon Washington realized what an opportunity this was.  The first step down a slippery slope where a central body has control over what gets published on the Internet, and they get to do it under the guise of protecting an industry's "intellectual" property.  The technically literate community is irate...

Let's think about that for a minute.  Does everybody have their tinfoil hats on?  Great, let's begin.


The Internets are pissed 
(how we see things)

Who are the Internets for the purpose of this discussion?  They are the people that are active participants in the web of information exchange.  That is they use message boards, social media sites, and other means of exchanging ideas with others like them.  For examples see postings on Slashdot, reddit, Hacker News, and the list could go on and on...

Somehow (it does not really matter how at this point) the Internets got wind of these really bad pieces of legislation that were trying to get pushed through.  It was shared among us and the shared response was like something out of a scene in Braveheart.  In my mind reddit is as close to a William Wallace as we've come, but our reaction has been in general to attempt to stir up as much dissent among the Internets and hopefully the average American as possible.  Why not?  After all it has worked pretty well in the past.  

We have done a good job about voicing to the technical among us that SOPA and PIPA is bad and there is none that I know of that are in support of it (or at least that dare to mention it).  Something strange was happening though, none of the major news outlets (read - things my parents would see) were covering it at all.  It wasn't that we were not being loud enough about it; they don't want to cover it.  They want to bring as little attention to it as possible, and that means not trying to spin it as positive (a hard feat indeed) but not talking about it at all.  Recognizing this issue reddit is leading the charge with a way to make sure that we are getting everyone's attention - make people mad.  The claim is that by blacking out popular Internet services that you can illustrate to people what a censored Internet would feel like and prod them to action.  As Forbes pointed out blocking reddit wouldn't be enough and is in a sense preaching to the choir.  They claimed that Google and/or Facebook needed to join in to make this truly effective.  I thought this just might work.  

Imagine if everyone in the US got a message to call or visit their representatives when they visited Google or Facebook.  The response would be unprecedented.  It would show for the first time how powerful the Internet could be - the "nuclear option."  How could it not work right?  If most of the country is calling you really irate and threatening to throw you out of office if you vote for SOPA, are you going to take that chance?

The Internets are causing trouble again
(how they see things)

There is a certain amount of smug pride in being a member of the old guard in Washington and some representatives use the fact that they don't use the Internet and don't care to know how it works as something of a bragging right.  For most of them the Internet is something of a toy or an annoyance.  Then comes the Arab Spring.  All of a sudden the Internet becomes a force for leveling the playing field not only in business, but also in the struggle and dissemination of governing power.

Imagine that you are a representative and people are calling you left and right telling you that they don't want you to support SOPA/PIPA.  On one hand the entertainment industry has a lot of power in the US and they are stuffing your pockets with wads of cash if you are willing to play ball, but on the other hand if people are angry enough you might loose your job.  The tension really starts to heat up when you start hearing about the "nuclear option," and you realize that larger portions of your votes are at risk.  

If you vote against SOPA/PIPA then you are going to lose the support of the massive entertainment industry, and make your constituents really happy (until the next thing that comes along they don't like).  But doing this confirms for those pesky Internet folks that they have a lot of power, and sets a precedence that threatens your future income from lobbyists and their clients.  

If you vote for SOPA/PIPA then you are assuring long term support from the entertainment industry, and the framework will be in place to silence whatever critics might arise in the future.  Since the majority of your voters will move on to whatever reality TV bullshit is trending or the latest shiny toy in a few days anyways, what's to risk?  Possible short-term lost, but a very substantial long term gain.

This is just crazy talk!

By doing what we thought was best at the time - raising awareness about an important issue - I fear that we may have given the politicians more incentive to implement a framework for shutting us up.  

Think about it - the more the Internet flexes its muscles the more the politicians see it as a threat and the harder they will fight to cripple it.  They are getting closer to the point where they risk losing everything and may result to extreme measures to protect their position of power.

The truth is at this point that at least one of our leaders see this as a golden opportunity to silence a very vocal minority.  Civil liberties are rarely lost over night, but it is a gradual descent.  We will not be North Korea over night, but as Benjamin Franklin put it, "those who would give up essential liberty, to purchase a little temporary safety, deserve neither liberty nor safety."  I can't help but think about that now every time I have to fly.

I hear myself and know how crazy all of this sounds.  Hopefully this will all disappear soon, but this one didn't.

un-occupy GoDaddy

I began moving all of my domains from GoDaddy last week in response to their support of SOPA.  I think that this piece of legislation is reprehensible and sets the United States on a path down a slippery slope where censorship is the norm.  To be honest censorship is one of the scariest things that could happen in a country. 

I would like to say that the actual move of the domains was relatively painless, but it wasn't.  I had heard reports of GoDaddy doing nasty things to make the transfer of domains take longer, and given that these kind of actions can carry stiff penalties for domain registries I assumed that they were largely blown out of proportion and that it would not happen to me.  Let's just say that I should have expected nothing less than the worst from a company that has spent the last few years sucking.  They have been a terrible registrar, but the inconvenience has stopped me from moving until I heard about the SOPA support.  

I am a little ashamed of myself for not moving my domains over the elephant slaughter by their CEO, Bob Parsons.  I have also been embarrassed about doing business with a company that uses blatant low-brow and crude sexual advertising.  I love a beautiful woman (I married her), but when I hope that none of the professional women that I have worked with do a whois lookup on one of my domains revealing that I have a domain registered with GoDaddy - it can't be a good sign.  What does having a domain with GoDaddy potentially say about me?  

• I am frugal (cheap).
• I react to ads with scantily clad women.
• I like bad customer service or worse...
• I don't know what good customer service is.
• I associate with crass men who kill majestic animals for sport.
• I am lazy...

I am pretty sure that none of these are true, and I am going to be more aware of what my purchases say about me as a person.  This is important because our purchases do say a lot about who we are.  Purchasing power is (at least in the US) the most powerful weapon that we have against big-business and their pupet of a government.  Let's explore this for a minute...

Reddit users proposed a boycot of GoDaddy on the 29th of December, and due to a large number of people proactively moving their domains in the days leading up to the 29th GoDaddy did a full turnabout on their SOPA support.  I have no doubt that their public stance is different from their true intentions and their response was too little too late, but the fact is that when their users started walking away the company listened real quick.  I still moved my domains because GoDaddy has given me plenty of reasons over the years and this was just the straw that broke the camel's back, but it was still pretty evident that this type of tactic is effective.  Compare that to the occupy movement that is taking place all over the country - there has been a lot of fuss but no one is listening.  The protesters are still sleeping in the cold, and the bankers are still getting their bonuses.  But the protesters are still walking in their Keen Sandals over to the Starbucks and paying with their Visa before sitting down with their iPad.  I am not saying that any of those things alone is bad, but if you want to change big business then you have to go after their business.

The best piece of advice that I have heard came from a friend that owns a popular restaurant here in Chapel Hill, NC.  He was talking to a bunch of protesters staying at the occupy camp down the street from his business, and he told them that the best way to get a big banks is to go after their income - use cash.  The idea is that small business owners like himself pay huge sums of money to the banks in the form of fees for debit and credit card purchases, and if those fees stop rolling in that the banks will eventually become weaker over time.  I was skeptical at first, and I am still not 100% convinced that the scale of these payments are enough to be effective, but maybe it is like a death of 1000 cuts.  At any rate, even if it is not effective at sticking it to the big banks, it will be effective at helping small business owners and keeping more funds in the community and off Wall St.  This is the easiest to do since we eat/shop in these places already and all we have to do is remember to hit up the yuppie food stamp machine (ATM) before hand.

For other big businesses, the changes in our personal habits may be a little harder to absorb, but it is a necessary change if we want corporations that act with social responsibility.  We have to vote with our dollars.  For this to be truly effective we will need more transparency in campaign contributions, but if a company backs a candidate that I think is harmful for the advancement of our society then maybe cutting off their revenue stream is the most effective means of changing their mind.  The first thing that I see that may need to change is how me and my family entertain ourselves.  The entertainment industry has really gotten too big not to fail.  They have their fingers in everything and have proven that they can easily move Washington with their wallet.  Since paying cash is not going to be effective here we are going to have to start paying attention to which studios are doing good things with our hard earned money (non of the major ones).  

The biggest problem here is that I see little chance in convincing the larger population to boycot any movie studio because (i) it requires though and investigation, and (ii) there are often not convenient and equal products to substitute.  Just like the majority of people are not going to sleep on the hard cold ground to effect change in a corrupt partnership between big banks and the government, few people are going to skip watching the latest big budget film because the studio is lacking scruple.  

I am not expecting large and lasting change any more.  Maybe I am just getting syndical as I get older, but wouldn't it be nice.  Still, I am going to do my best to do my part and I hope that a growing number of people will find their own solutions.  As for GoDaddy...  I have moved all of my domains and deleted the ones I couldn't move.  

There is no shortage of articles on this topic but this one titled Stupid GoDaddy Deserves Boycott is particularly great.  "If GoDaddy were to just go away, the world and Internet would be a better place. If I had any domains there to pull, they would already be gone.