Caffeinated Coder

I have not yet downloaded the new CTP, but the KC User group meeting the other night as well as recent decision that I had to make about whether or not to purchase LiteSpeed has inspired me to do some research into SQL Server 2008.

Apparently Microsoft is still a little sensitive from being 2 years late on the last version because Dave Campbell spent the first 5 minutes of his Tech Ed presentation explaining how they have re-engineered their process to ensure that the next version will be delivered on schedule. For example, they will no longer be checking in features into the CTP codebase unless they are fully completed so early CTP’s will only include a few fully functional features rather than most of the envisioned features in a half-baked state.

Well, whenever we do get the next version, here are a few features that caught my eye.

• Merge statements (aka upserts) – a nice efficient syntax for loading data into a data warehouse that will allow you to specify either an insert or update based on whether or not a record exists.

MERGE INTO FactTable F
USING TransactionTable T ON T.OrderID = F.OrderID
WHEN MATCHED THEN UPDATE
SET F.Quantity = T.Quantity

• Table Valued Parameters – allows you to pass in a result-set as a parameter to a sproc of udf.
• Resource Governor – helps prevent runaway queries that can bring down your server. This is especially helpful if you work someplace that allows ad hoc querying against transactional systems.
• Declarative Management Framework – the Tech Ed webcast showed a demo where a create table statement was prevented from running in Query Analyzer because it violated a policy that had been set up regarding ownership and schemas. Another demo showed a database in SMS with a warning icon because the shrink DB setting had been enabled that violated a configuration policy.
• Dynamic Development – a rather vague term, but one example was given that described tagging columns with metadata tags such as ’sensitive’ that would then allow you to do queries to track these categories and also apply policies such as only allowing users to query these columns through a secure connection.
• New Data Types – The filestream data type allows for a best-of-breed approach to managing blob data that would blend the efficiency of the file system with the manageability of the database (backup, restore, querying). Location and Spatial data types allow for built in support for geo-mapping features.
• Change Data Capture – a good way to appease the auditors by tracking and exposing changes though a relational interface.
• Encryption – Also an auditor pleaser. It is supposed to be configurable without any application changes required.
• Performance Enhancements and Tracking – The biggest gains appear to be in start join optimization and data compression, which takes advantage of the fact that SQL Servers are rarely CPU bound. There are also supposed to be a plethora of performance monitoring tools, which we currently rely on Quest tools to provide us.

I’m looking forward to playing with some of these features. With any luck its name will reflect reality and the upgrade path will be easier than it was with

Popularity: 5% [?]

There was some discussion about SQL Server 2008 at the KC .NET user group meeting last night and I involuntarily winced at the thought of another upgrade because we just finished an intensive migration process a few months ago.

In hindsight, the migration went fairly smoothly but it was definitely not a trivial process, nor was it something that I would want to repeat any time soon. It definitely requires some advanced planning and organized regression testing. I thought I would jot down a few lessons that we learned before I completely block this experience out of my memory.

Here are some quick background stats (mid-sized bank):

1. Apps Tested: 40
2. Technologies supported: .NET 2.0, 1.1, 1.0, Classic ASP, VB6, PowerBuilder, Access 97
3. Databases: 25
4. Total Tables: 2400
5. Total DB Objects (sprocs & udf’s): 3500
6. DTS packages: 100
7. Server implementation: Clustered Quadzilla running on 32 bit Standard Edition, separate Reporting Server
8. Total required object changes: 40-50

Lessons Learned:

1. The upgrade wizard was helpful but not full-proof. It caught about 60% of the changes we had to make. If you have separate reports or application logic that runs on specific days (i.e. end of month, start of month, end of quarter, etc), make sure you run profile traces on those days and then use the trace files as input for the upgrade adviser.
2. Script out all the sprocs and udf’s and then run the ddl scripts against an upgraded test server. We caught another 20% of the problems through the syntax errors we got when trying to recreate these objects.
3. Set up an organized regression test of your critical apps, especially if your business has a low tolerance to risk. The last 20% of our changes were only caught through testing because of dynamically generated SQL or SQL that was embedded within the applications rather than stored as objects within SQL Server.
4. Carefully review all proposed sproc changes and when possible compare before and after result sets with a tool like RedGate BeyondCompare. Most of the changes we had were syntax related and easily fixed, however we had to redo a few changed sprocs because of some misunderstandings of how the old *= join syntax worked.
5. Be on guard for a few hard to spot errors that might show up later in production. For example, this excerpt from one of our udf’s actually concatenated multiple rows together in SQL 2000, but only returned the first row in 2005. The only way we figured this out was that someone reported that a few of our app generated pdf documents were missing several paragraphs.

 

 

   1: select @para = @para + cast(dgp.Paragraph as varchar(7000)) + '

   2:

   3:             from DocumentParagraphs dgp

   4:

   5:             where dgp.ReportID = @ReportID

   6:

   7:                         and dgp.ActiveFlag = 1

   8:

   9:             order by dgp.ParagraphOrder

6. Do not rely on 8.0 compatibility mode as a safety net. Dropping back into 8.0 compatibility mode did not make the problems we found magically work.
7. If you have to move many DTS packages to a new machine, be prepared for some mind-numbing, tedious work. Each package had to be moved manually with lots of GUI clicks and all inner packages had to be re-selected due to GUID issues. Despite this headache, the DTS process was much smoother than expected and we didn’t have to make any code changes within the packages. We did experience some additional flakiness with SQLMail, but we’re not sure if that is related to SQL 2005 or moving from a single server to a cluster.
8. Beware Legacy Apps – If you have to support an old version of a defunct language like PowerBuilder, make sure that you are nice to the only guy who can support it. It turns out that we really didn’t have one of those guys anymore so a couple of us were forced to stumble around in the IDE for the first time ever in order to search for the offending embedded SQL while under deadline pressure. NOT FUN!
9. Don’t call undocumented functions! An old C++ module used by a couple of our legacy apps uses pwdcompare(). We finally got it to work but only by seriously elevating the application’s account permissions.
10. Access wasn’t so bad – Contrary to all expectations, we really didn’t have any problems with linked tables within our Access databases even though we were stuck on a really old version (Access 97). I expected this to be a nightmare, but it wasn’t.

Despite these hassles, the migration was definitely worth the effort for us. We were running up against a memory limit imposed by SQL Server Standard edition with our old server and were starting to experience some painful performance problems. SQL 2005 allowed us to upgrade to a much more powerful machine without having to pay the additional $180,000 in licensing fees that it would have cost us to upgrade all of our servers to the SQL 2000 Enterprise Edition which had support for additional memory.

Popularity: 3% [?]

If you’re crazy enough to run marathons for fun like me, you are probably familiar with a concept called cross-training. This slightly counter-intuitive approach asserts that you can actually increase your performance in your chosen sport by doing it less frequently and supplementing it instead with complementary activities. In other words, you will run faster and decrease your chance of injury if you replace a few training runs a week with weight-lifting, biking, or swimming.

I think the same principle holds true in the IT world. There are clear boundaries and activities that define what a normal developer activity or skillset is and I’ve noticed a strong tendency in both myself and other developers to stay within that comfort zone not engage in cross training by expanding our knowledge into complementary disciplines.

Developers commonly learn to write complex queries, use the ipconfig or ping utilities, and upgrade their own RAM, but usually defer to DBAs and network admins when it comes to index tuning, group policies, or rebuilding machines. It is easy for a Microsoft developer to ignore the open source community and never learn to use other platforms and languages. Corporate developers who are younger or come from a non-CS background rarely look lower on the abstraction stack than the Base Class Library to learn about assembly, core programming algorithms, the inner workings of a compiler, or the win32 API.

So what are some good steps to take in order to avoid this informavore trap and begin a technical cross training program? One activity that has worked out nicely for me so far is listening to RunAsRadio, a podcast co-hosted by Richard Campbell of dotnetrock’s fame that covers diverse IT Pro topics such as network storage technology, application virtualization, MOM, IIS7, the future of 64 bit computing.

Another activity that I plan to continue is the self-paced tutorial on Assembly that I started a few months ago with MASM. I wouldn’t want to program in Assembly for a living, but it’s not nearly as scary as people make it out to be and it has greatly helped me visualize what is actually happening at the processor level.

Finally, I am way behind the curve when it comes to learning about Ruby on Rails as well as exploring several Alt.NET open source frameworks such as Spring.NET, Windsor and MonoRail, so I plan to devote as much of my spare time as I can expanding my horizons in this area in the coming year.

Popularity: 3% [?]

Jeff Atwood recently wrote a blog post on Informavores that uses a food foraging metaphor to describe typical web usage behavior. According to this analogy, the ultra short attention span of most web users is not due to laziness or cultural ADD, but has more to do with an innate survival instinct that leads us to try to get the maximal benefit for the minimal amount of effort.

With billions of pages of content and sophisticated search algorithms that are spookily accurate, it just doesn’t make sense to spend time on pages with a ‘weak information scent’ when the information we want is likely on the next result page for a fraction of the effort. So the next time you start feeling guilty about hitting the back button within seconds of the page loading, remember that your itchy trigger finger may not stem from a character flaw but rather an appropriately unforgiving predatory instinct. Now, if only I can come up with an equally flattering metaphor for TV channel surfing…

Along the same lines, I’ve been thinking about a few common pitfalls that I’ve noticed myself and other developers fall into that I’ll call Informavore Traps. Here’s a short synopsis of a few anti-patterns of information foraging that I think tend to stunt our professional growth as developers and keep us low on the alpha-geek food chain.

• Self-Imposed Boundaries – These are unnecessary mental filters or borders that we construct in our minds that severely restrict our hunting territory.
• Indulging in Information Junk Food – This is the tendency to stop consuming knowledge on a subject once we establish a command of buzzwords and recycled blog opinions.
• Jack of All Trades, Master of None – This refers to the tendency to spend too much effort on gaining a wide breadth of knowledge and not enough time to become an expert in a particular field.

I’ll discuss my thoughts in detail on each of them during the next blog posts.

Popularity: 3% [?]

Scott Berkun has an amusing post on ADD (**shole Driven Development), a parody on TDD which occurs when the technical decision-making process in an IT Shop is hijacked by an over-bearing personality who always manages to persuade people into doing things a certain way because the cost of opposing that person is simply too high.

The comment section of this blog post is definitely worth reading and coins some other noteworthy processes that are rampant in the industry, such as BTPWAL (Blame the People Worked and Left), BDD (Buzzword Driven Development), and IAAD (It’s Almost Done Development).

This post even prompted me to respond with my own new methodology:

WTDD (Wild Theory Driven Debugging) – A methodology that enables developers to quickly resolve debugging tasks by blaming erratic, intermittent software failures on external forces that are not immediate under the developer’s control (i.e. firewall, network, test domain, server issues).

This approach is extremely lightweight in that it requires no direct physical evidence such as logs, tracing output, or KB articles to support the assertion. When used in conjunction with ADD and loosely related coincidences, it has the added benefits of allowing you to sound smart while simultaneously weaseling out of any significant debugging effort.

Popularity: 4% [?]

I’ve noticed my geek ego being aggravated a few times lately so I figured I’d better work on a more effective personal implementation of IHumbleProgrammer. My current implementation seems to be leading to errors in my thought process and even causing my learning to hang. Please excuse the VB.NET, poor formatting, and lack of a valid unit test.

Someone sent me a great quote the other day from Douglas Adams:

“I love deadlines. I especially
like the whooshing sound they
make as they go flying by.”

It brought back distant memories of a gigantic waterfall project that I survived. I think the word survived is appropriate in this case because we labored on it for over 3 years before we ever released code to production, which by that time was somewhere between 12-18 months later than our original deadline (I blame my fuzzy memory on the excessive overtime that overshadowed that part of my life). As with any good old fashioned waterfall project, the first in a long series of missed deadlines was arbitrarily generated by the business and the project was both time-boxed and feature-boxed (at least in the sense that the amount of work rarely went down).

There were many months of intense overtime, implicit threats of being fired, low morale, and constant bickering with users over whether or not a newly requested feature (or as they liked to refer to it…bug) was explicitly or implicitly included in the 500 page functional specifications. I have clear memories of pondering whether I should keep the % complete cell in the project status worksheet at 90% complete for yet another week or move it up by 1% to delay the inevitable ‘Come to Jesus’ meeting for just a little bit longer just in case a miracle happened. The strange thing is that the project was still considered tremendously successful despite all the waterfall-induced deadline madness. Ah the good old days…

A few years ago, our IT department adopted an agile approach. I’m not saying that things are perfect. We definitely had speed bumps and hurdles during the transition period and still have occasional perception problems to deal with when a team’s velocity dips too low for too long, but it just isn’t’ the same. I rarely notice any users yelling at project managers or any developers working for over 40 hours or talk of an impending deadline that everyone is worried about missing. Teams actually seem to be able to release code at the promised time.

All of this makes me feel like a rite of passage is slowly fading away and 10 years from now young developers will roll their eyes whenever I reminisce about the ‘old waterfall days’. I will say, ‘Why…back in MY DAY…we didn’t have these sissy, daily stand-up meetings…We waited until the executives were good and ornery and before we bothered scheduling a status meeting…’

By the way, there is an excellent podcast on dotnetrocks about agile methodologies. It is a live panel discussion from last year’s Tech Ed in Barcelona with Roy Osherove, Seven Forte, and Kate Gregory.

Popularity: 4% [?]

In his blog post on Old School Programming, Wesner Moise waxes nostalgic about his pre-high school experiences with writing his own disassembler and assembler for the Comodore 64. Apparently he used his homemade dissassembler to decode and rewrite the entire 8k BASIC ROM back to source and then used his assembler to add his own extensions to the BASIC language to support structured programming and better graphics. Aside from retyping a few bouncing ball programs from the owner’s manual,

I believe that the only thing I ever did with my Comodore 64 was play pirated video games with my friends. For some reason all I can remember is a break dancing video game where I felt proud of getting the little block character to spin on his head. As you can imagine, the post made me feel like I belonged somewhere between plankton and an anemic protozoa on the alpha geek food chain.

By contrast I spent most of my day today in high altitude architectural abstraction. The bank I work for has hired a consulting company to do a data strategy gap analysis and we spent a good part of the day talking about BPM, SOA, and Master Data Strategy. There were several points where I had to politely suggest that we move on to another topic because we had risen a little too high in the abstraction layer and I was sure that someone walking past might have had trouble discerning whether the topic of the conversation had to do with coding, politics, or pepperoni pizza. The next two days will be more of the same.

Does anyone know if you can get decompression sickness by ascending too quickly from the depths of creating custom interrupts on the Comodore 64 in order to implement a rasterizer that bypassed hardware limits for 8 sprites to the abstract heights of macro workflows and model driven architecture? If anyone happens to observe any of these symptoms tomorrow, please sit me down and give me a steady dose of concrete examples. Thanks in advance.

[Originally Posted Monday, June 18, 2007 11:31 PM]

Popularity: 6% [?]

A few years ago I shifted roles from a developer team lead who coded almost full time to an Architect. Although I am naturally a bit of an abstract thinking and find architectural issues interesting, I have always been hesitant about this role because of the ubiquitous Architecture Astronauts that have given the discipline a bad name. These guys have a knack for intimidating even the smartest of developers by throwing around architectural jargon that they can describe in the abstract but are completely unable to translate these ideas into concrete code.

Last night as I was lying in bed, I realized that I hadn’t written a single line of code in the last few months because of my involvement in some high level strategy initiatives, my advisory role in a number of projects, and my supervisory responsibilities over a solutions architect and a data modeler. I realized with horror that much of my time these days was spent just reading about fuzzy concepts like SOA, SaaS, BPM, EII, EAI, MDM, Data Governance, and I was sure that I had suddenly become one of those useless pieces of corporate baggage that real developers make jokes about. That thought prompted me to spend my last waking moments plotting various routes to getting back to my coding roots.

This morning I relistened to an old Hanselminutes interview with Jeffrey Snover, the Powershell Architect, to help prepare for an upcoming .NET User group talk that I will be giving on Powershell. It was an excellent podcast, but what caught my attention the most was when Jeffrey mentioned that in the beginning of the project he had locked himself in a room for a month and pounded out a 15,000 lines of code as a proof of concept that he then used to convince people to get the project started.

THAT is the kind of architect that I want to be when I grow up! I need to figure out a way to get myself out of all these meetings about process improvement, data strategy, and regulatory compliance and start focusing on some nice juicy proof of concept work.

[Originally Posted Friday, June 15, 2007 12:10 PM]

Popularity: 6% [?]

I listened to an interesting prodcast on Channel 9 on my way to work this morning. I never had an inclination to listen to the MicroISV show before, but the title intrigued me: CrazyBusy – Overstretched, Overbooked and About to Snap! Strategies for Coping in a World Gone ADD“.

Multi-tasking has never been a strength of mine so I may be biased, but I definitely agree with his premise of the show that the frenetic pace that most of us sustain these days due to new technologies such as cell phone, PDA’s, and the internet is actually causing us to get less done. I view it as similar to the issues faced when doing multi-threading in programming. When used judiciously, multi-threading can dramatically increased the perceived speed and responsiveness of the application. However, if you go crazy with the number of threads that you use, the app will actually slow down to a crawl because it is spending all of its allotted slice of time just to reload the required data in memory and by the time that happens there is barely any time left to do any work before having to release control to another thread. It’s the same way with life, it may look like someone is getting a lot done when they are juggling dozens of projects at once, but it has been my experience that in reality the overall speed and quality decreases dramatically.

So how do you stop the madness? As for myself, I think I’m going to start with my email habits. This tends to be the single biggest interrupter during my day since I usually drop everything whenever I notice an email pop-up notification, even if it is just to read and delete it. I think I’m going to force myself to actually close Outlook and only check it on an hourly basis instead. The vast majority of time someone will talk to me in person or call me if they need something answered right away, so I might as well take advantage of the asynchronous nature of email and just ignore them until I am at a natural stopping point in my work. I’m also going to try to decrease the number of active projects that I have going on at any point in time as well as the number of technical topics that I explore. I’ve had a nagging feeling for a while that my understanding of topics is becoming more and more superficial and now I feel certain of it. Perhaps, I’ll even try to narrow the number of blogs that I track…nah….

[Wednesday, June 13, 2007 5:20 PM]

Popularity: 4% [?]