WebTrends Data Surge & Rocking the Analytics

Almost a year ago I had left WebTrends to kick off a start up.  I had always promised I’d return to WebTrends if they’d have me back.  With that context, I completed this start up kick off and then headed back to WebTrends.  I did this for a couple of reasons.  The main one was the people I get to work with.  WebTrends without doubt has one of the best crews of people out of all the software companies in Portland.  The services department to the engineering group to the sales team are all a great team of people.  They handle their jobs well, push forward with learning, and strive to make WebTrends a great place to work and make sure that the products and services we offer get better and better by the day.

Another one of the reasons that I returned to WebTrends is because of the industry itself.  Analytics, specifically web analytics, are hot.  Everything from Twitter tracking, sales trending, site analysis, and market studies, web analytics is an awesome field to be in.  WebTrends is one of the leaders in that field.

WebTrends over the years has had varying development methodologies and this brings me to my third main point of why I returned.  WebTrends is steadily getting more and more Agile.  In the methodology sense.  Anyone that has read my blog in the past knows I”m a fairly hard core agile advocate (I”ve seen more devoted like Arlo Belshee, but I”m an advocate none the less).  From the iteration based to the kanban boards to the pair or promiscuous pairing to functional teams, the agile style of team & project work is awesome.  WebTrends has started down this road heavily and that is my cup of tea.

What I’ve Been Working On

I was asked recently by some fellow .NETter geeks, “what have you been working on at WebTrends?”  To which I’ve been happy to say, “Iz Zecret!”  or the most part, what I’ve been working on has been on the down low lately, however, I?ll be blogging a LOT more about it real soon.  I’ll be providing examples here, and on a new site that isn’t published yet, about how we’ll be doing some awesome “fill in the blank” – so stay tuned for that.

Follow WebTrends

If you?d like more info on WebTrends here?s some places to check out and tune in.

Stay tuned, I’ll have more coming from the WebTrends Data Pipe soon!

REST, So Many Nuances, The Self Signed SSL Cert Override

Another thing that comes up frequently is the need to do development internally against some REST Services that have SSL as the transport.  If this is your case, in client code you’ll need to toss a delegate in to actually get past the SSL Self Signed Cert and get to the rest of your REST Services authentication or whatever else is next in the process.  Here’s a code snippet on how to toss a delegate in.

textResults.Text = DateTime.Now.ToLongTimeString() + "\n";
if (!textUriRoot.Text.EndsWith("/"))
    textUriRoot.Text += "/";
string baseUri = textUriRoot.Text + ProfileUri;
MessageBox.Show(baseUri, "Using URI", MessageBoxButton.OK, MessageBoxImage.Information, MessageBoxResult.OK);

// Deal with the self signed SSL certificate.
ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };

var webClient = new WebClient();
webClient.Credentials = new NetworkCredential(@"username", @"password");
webClient.DownloadStringCompleted += GetProfiles;
webClient.DownloadStringAsync(new Uri(baseUri));

.csharpcode, .csharpcode pre { font-size: small; color: black; font-family: consolas, “Courier New”, courier, monospace; background-color: #ffffff; /*white-space: pre;*/ } .csharpcode pre { margin: 0em; } .csharpcode .rem { color: #008000; } .csharpcode .kwrd { color: #0000ff; } .csharpcode .str { color: #006080; } .csharpcode .op { color: #0000c0; } .csharpcode .preproc { color: #cc6633; } .csharpcode .asp { background-color: #ffff00; } .csharpcode .html { color: #800000; } .csharpcode .attr { color: #ff0000; } .csharpcode .alt { background-color: #f4f4f4; width: 100%; margin: 0em; } .csharpcode .lnum { color: #606060; }

Just A Bit More REST

Just a few more examples, kind of a follow up to my Simian Satasin Asynchronous Calls Entry.  Keep in mind, I’m providing these as examples, I do NOT suggest writing tests with Thread.Wait calls in them.  Very very bad practice.

    [TestClass]
    public class GetRestServices
    {
        private XDocument doc;

        [TestMethod]
        public void TestSynchronousCallWithHttpWebRequest()
        {
            var request = (HttpWebRequest) WebRequest.Create("https://twitter.com/statuses/public_timeline.xml");
            var response = request.GetResponse();
            var responseStream = new StreamReader(response.GetResponseStream(), Encoding.UTF8);
            var doc = new XmlDocument();
            doc.Load(responseStream);
            Assert.IsNotNull(doc);
        }

        [TestMethod]
        public void TestAsynchronousCall()
        {
            const string baseUri = "http://twitter.com/statuses/public_timeline.xml";
            var svc = new WebClient();
            svc.DownloadStringCompleted += svc_DownloadStringCompleted;
            svc.DownloadStringAsync(new Uri(baseUri));

            Thread.Sleep(1000);
            Assert.IsNotNull(doc);
            var docTest = new XDocument();
            Assert.IsInstanceOfType(doc, docTest.GetType());
        }

        private void svc_DownloadStringCompleted(object sender, DownloadStringCompletedEventArgs e)
        {
            doc = XDocument.Parse(e.Result);
        }

        [TestMethod]
        public void TestAsynchronousCallWithSecurity()
        {
            const string baseUri = "http://twitter.com/statuses/friends_timeline.xml";
            var svc = new WebClient();
            svc.Credentials = new NetworkCredential("yourtuweetusername", "yourtuweetpassword");
            svc.DownloadStringCompleted += svc_DownloadStringCompleted;
            svc.DownloadStringAsync(new Uri(baseUri));

            Thread.Sleep(3000);
            Assert.IsNotNull(doc);
            var docTest = new XDocument();
            Assert.IsInstanceOfType(doc, docTest.GetType());
        }
    }

Hope those are helpful for those searching.

Simian Satasin Asynchronous Calls

All I wanted was an asynchronous WebClient call.  What do I get?  Tons of dead links in MSDN, tons of pure crap of non working code all over the net.  Then I realized I needed to search for silverlight AND asynchronous instead of just asynchronous.  Anyway I finally found some legit code snippets and put together this simple test.  The story goes like this…

Here’s a little test for getting asynchronous calls up and running, and tested.  Keep in mind long tests that take place over the network or that will introduce extensive latency should NOT be part of a CI build.  If anything set these into a separate build that can run over a period of time to assure integrity and that the services are up, but seriously, don’t add them to a CI build.  I’ve seen enough of these, and it is VERY destructive to productive development.  No really, did I point this out yet, do NOT add these to a standard CI build.  K – thx.

With that warning, here’s a way to test some services over REST.

     [TestClass]
     public class AsyncTest
     {
         private XDocument doc;

         [TestMethod]
         public void TestMethod1()
         {
             string baseUri = "<a href="http://twitter.com/statuses/public_timeline.xml">http://twitter.com/statuses/public_timeline.xml</a>&quot;;
             var svc = new WebClient();
             svc.DownloadStringCompleted += svc_DownloadStringCompleted;
             svc.DownloadStringAsync(new Uri(baseUri));

             Thread.Sleep(1000);
             Assert.IsNotNull(doc);
             XDocument docTest = new XDocument();
             Assert.IsInstanceOfType(doc, docTest.GetType());
         }

         private void svc_DownloadStringCompleted(object sender, DownloadStringCompletedEventArgs e)
         {
             doc = XDocument.Parse(e.Result);
         }
     }

Create a variable that the handler will assign a value to, it doesn’t really matter what it assigns as long as it proves the handler executed.  Of course, if you’re looking for a particular bit of code or a property you would want to test for that, but otherwise the primary concern is to test that it fired appropriately and didn’t cause an explosion of some type.

Next create the method that initiates the call and wire up the event handler to the event.  Now when you execute the method, which I’ve left as the default lame name of TestMethod1, you should get a positive test.  The big question for this test is, “Will the REST service return in less than a second?”  If it takes longer you need to add to the Thread.Sleep(X) method.

There is of course all types of issue with this particular test.  Isolation is broken, it breaches boundaries, yadda yadda yadda yadda.  The point is, you get a test that can verify that a service is up, and that it responds with something.  As an extra kicker you can even verify that the service is responding within a certain threshold of time since a Thread.Sleep method is used.  If it fails because the time isn’t enough, you would know you need to do something about the performance.

Either way, good luck on testing these, if you have any other ideas on legit ways to test services like these, please pop a comment on this entry.

PEX, Ben Hall, and Some Other Notes

While checking out more ideas for my data generation tool/app/libraries I stumbled on this video of a presentation by Ben Hall.  The presentation “DDD7 Session Video – Microsoft Pex – The Future of unit testing?” is a great introduction to the PEX framework/toolset.  If you write unit tests, or are trying to get started writing tests, this is a great video to watch.

I watched the video and have a new level of admiration for this guy, and for Red Gate.  He’s using ReSharper (+ points), steps through code smoothly and logically (+ points), and above all he knows his information.  Excellent traits.

PEX Worries?

At 40:14 in the video Ben brings up a major point of contention that I too have.  I don’t really want to use PEX with the Microsoft Test Framework.  I?d much rather use NUnit, mbUnit, or SOMETHING besides Microsoft Test.  Even though, I know, the last dozen code entries I’ve posted uses the Microsoft Test Framework I intend to change that soon.  Ben goes on to point out that almost all of the major test frameworks are now supported.  This is awesome news!  I now have full motivation and have downloaded the PEX libs.

With That Written?

I’m now off to download NUnit and get rid of my Microsoft Tests.  I will prospectively go get XUnit too and start taking a solid look at that tool.

Data Generation Tools

While viewing this video Ben also mentioned a data generation project on CodePlex.  I couldn’t help but get a bit giddy about this fact.  So I checked it out.  Red Gate Generators – Red Gate also has a professional tool that they sell.  The libraries look pretty cool and solid, but I’ll have to review the library license before I know if I can use anything there or not.  I might have to just “not see” anything there.

Fini.