Goshido

Web Performance Tuning Pattern: Database Sharding

Posted on 13. Aug, 2013 by ger in Performance Tuning Patterns

I was recently talking to an engineer at another company and he was asking about my opinions on database sharding.

I’ve worked on a number of web-applications including Goshido and a customer’s application that I’ve recently helped to scale. We successfully used a number of scaling techniques but so far we haven’t had to go as far as database sharding.

In the past, I wrote a pattern language for performance tuning embedded applications [1]. Patterns are a simple literary mechanism to share experience and impart solutions to commonly occurring problems.

So out of curiosity, I decided to do some research on sharding and write it up as a pattern for web application tuning.

Context

• Scaling a web application with a backend database
• Database write performance or capacity is a confirmed bottleneck
• Database configuration tuning has already been implemented
• It is no longer feasible to move the database to a larger server

Problem

Eventually you can arrive at a situation where database write performance or database capacity is the limiting factor for your system scalability.

Solution

Distribute some of your application data across a number of database shards. The application or database implements a translation from an object id to shard database. Essentially your application now has one logical database layered above a number of physical databases.

For example, imagine a micro-blogging application where users are frequently submitting posts. You could use a modulus of the “post id” to distribute each post to one of many database shards. Later when someone wants to read a post you would translate the “post id” to a shard id before retrieving the post from the appropriate database.

This example demonstrates a key-based partitioning strategy. Other strategies include:

• Vertical partitioning: where different types of data are stored on different databases (profiles, posts, users)
• Directory-based partitioning: a lookup table that translates object id to shard

For detailed description of other strategies (like date-time and master indexes) see Max Indelicato’s blog post below [2]. Eric Ries outlines a URL based sharding strategy that enables you to incrementally shard a system over time [3]. He recommends quickly building non-scalable solutions then when the product gets traction, start sharding by identifying “highly-trafficked tables that are rarely joined with any others”.

Forces

• Some NoSQL (NoJoin) databases (i.e. MongoDB) have integrated support for sharding and shard balancing.
• If you are using a traditional relational database (i.e. MySQL) you probably need to implement application level sharding (although some databases support forms of sharding like federations in Windows Azure SQL Database).
• Application level sharding adds complexity.
• Changing sharding strategy or number of shards can be complicated and may involve moving data from one shard to another increasing deployment complexity/time.
• Sharding can impact other system workloads. In the example above, if we wanted to show multiple posts for a specific user we would need to issue read requests to multiple database shards.
• Sharding limits your capability to join across tables. To get around this you may need to modify your application to decompose joins, denormalize the data or copy some data to the shards [6]. This will increase complexity.
• Directory based partitioning has a single point of failure.

Further reading

1. “Tricks and techniques for performance tuning your embedded system using patterns”: a series of articles I co-authored with Peter Barry for Embedded Magazine a few years ago part 1, part 2, part 3.
2. Max Indelicato’s Primer on Database Sharding has more details on implementing partitioning strategies
3. Sharding for Startups by Eric Ries
4. Book – Cloud Architecture Patterns by Bill Wilder. While the sub-title mentions Azure, this book is general enough to be applied to other cloud platforms.
5. Book – Scalability Rules: 50 Principles for Scaling Web Sites by Martin L. Abbott; Michael T. Fisher.
6. Book – Professional Website Performance: Optimizing the Front-End and Back-End by Peter Smith

Photo Credit: Winter Ice Mosaic by Ctd 2005.

Scrum and Agile for devops

Posted on 16. Jul, 2013 by ger in Agile, Guides, Scrum

Devops engineers were part of two recent Agile and Scrum training workshops I ran in Dublin. Here’s some resources that you might find useful if you’re coming to agile from a devops perspective.

Blogs and blog posts

Two of the best blogs I found are the Agile admin and Agile Web Operations. The Agile admin has a good post describing Scrum from an ops perspective. Some devops teams combine Scrum for longer term projects and Kanban for shorter term work. This blog post I wrote recently summarizes a great article by Jim Coplien about the linkages between Kanban and Scrum.

Books

These books include sections on agile devops

• Scott W. Ambler; Mark Lines, Disciplined Agile Delivery.
• Paul Swartout, Continuous Delivery and DevOps
• Michael Hüttermann, DevOps for Developers
• Jez Humble; David Farley, Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation

Learn more

Try Goshido, our collaboration & project management platform. Goshido is so flexible it can be used for Scrum, Lean and projects with traditional work-breakdown plans. It’s great for both long and short term devops projects.

Photo Credit

Internet Splat Map by Steve Jurvetson

User Stories for Agile Requirements Management

Posted on 26. Mar, 2013 by ger in Agile, Guides, Scrum

We use “user stories” to document and manage requirements. Recently I provided training for one of our customers. Here’s some interesting resources we recommend.

What are user stories?

User stories are a short placeholder for a detailed conversation about requirements – deferring detail until needed. Stories represent customer value and are written in the customers’ terminology.

The general formula for a user story is: As a <type of user>, I want <some activity> so that <some reason>.

For example: “as a registered user, I want the system to warn if my password is easy to guess, so my account is harder to hack.”

Why use them?

I’ve seen studies which state incorrect requirements are the cause of 80% of software project failures. User stories are a simple, low-overhead way of documenting and managing requirements.

Resources

One of the trickiest parts of using user stories is breaking them down to the right size. Richard Lawrence has written an excellent pattern language to help you in this process.

There’s been an interesting debate on the value of acceptance testing.

We recommend the following books:

• User Stories Applied: Mike Cohn
• Agile Software Requirements: Dean Leffingwell
• Specification by Example: Gojko Adzic
• The Art of Agile Development: James Shore

Learn more

Try Goshido, our collaboration & project management platform. Goshido is so flexible it can be used for Scrum, Lean and projects with traditional work-breakdown plans. It’s great for managing product roadmaps and requirements documented as user-stories.

Project Collaboration Overview & 7 Great Books : Siemens Talk

Posted on 12. Mar, 2013 by ger in Agile, Events, Guides, Leadership, New Ways to Work, blog

Last night in Dublin I spoke to a group leaders from Siemens at an event coordinated by WDHB.

The key points of my talk were:

• Existing tools still don’t solve the problems of communication within large organizations
• Businesses need a flexible system that can scale from gigantic to many small-informal projects
• Culture is even more important than tools. Agile ways of working & leading with short iterations and empowered teams are crucial.

I recommended a number of books that explore the last point:

• Radical Management: Stephen Denning
• Succeeding with Agile: Mike Cohn
• The Future of Management: Gary Hamel
• Management 3.0: Jurgen Appelo
• Getting Things Done: David Allen
• The Way We’re Working isn’t Working: Tony Schwartz
• What Were They Thinking?: Jeffrey Pfeffer

Other speakers at the event were Marie Wallace of IBM and Polly Sumner of Salesforce. Polly recommended a really interesting followup action at end of her talk. She suggested everyone in the room try to buy or get customer support for their own product.

By coincidence I read the following quote in the Siemens “Pictures of the Future” magazine this morning.

“Modern information and communication media promote a short attention span and a superficial understanding of the world.” — David Gelernter.

By another coincidence I read this in the Chess Media newsletter this morning.

“Communication barriers cost large companies an average of $26,000 per employee per year due to productivity losses.” — Jacob Morgan

Learn more

Try Goshido, our collaboration & project management platform. Goshido is so flexible it can be used for project collaboration in any kind of business: product development, energy, healthcare & consulting.

You can download my slides.

Dispelling common misconceptions in Lean, Kanban and Scrum

Posted on 17. Nov, 2012 by ger in Agile, New Ways to Work, Scrum

Happiness Door Scrum Training Dublin

They say teaching is the best way to learn something. Recently, I delivered a Scrum training course for a multinational company in Dublin. Afterwards I did some reading to followup on some of the questions that came up. This reading lead me to some interesting new thinking on Scrum, Lean, Agile and even running training courses. It looks like many people have common misconceptions about how Kanban and Scrum fit together.

Some background

Scrum is one of the agile project management techniques that can make your team far more productive. Most of the time I’m CTO of Goshido (cloud software for project communication). Goshido is inspired in part by some of the concepts in Scrum. Some of our customers ask for training in agile project management and Scrum in particular. Our Scrum training is unique because I am still a practicing software developer and project manager, not a Scrum-trainer who delivers training every week.

Lean is a process improvement discipline that software development and startup companies.

Kanban is one of the Lean tools, originally a card or a box used in manufacturing to signal depletion of product, parts, or inventory. It is a way to “pull” work through a system.

Overlapping phases & organizing by project

The company I was training has structured the organization based on roles and skills rather than projects. The analysts are in one team, the UX people another, the developers and testers yet another, and operations another. During the project there’s a handover from one team to another. One of the principles of Scrum is to organize teams around the work that needs to be done for a customer, to mix the disciplines together with overlapping phases. Dustin Curtis tells a great story about the magic of Penn and Teller to illustrate the point that:

Assembly lines work fantastically well for raw assembly, but they suck the life out of creative people.

Getting feedback during training

During each training day I asked people to provide anonymous feedback both half way through and at the end of the day. I applied Jurgen Appelo’s happiness door technique. I got great feedback from the attendees and was able to make adjustments both during the day and for subsequent days to make the training more relevant and valuable to the people attending.

Kaizen, retrospectives and continuous improvement

One of the technical leads was concerned about the business focus of the product backlog. How would he get technical tasks like a transition to github into the backlog? He knew this would boost the productivity of the software team but what if the product owner didn’t feel the same way?

Conversely, I’ve seen situations in other companies where technical leads have refactored parts of a system without concern for the business priorities. This in turn lead to a surprise then an overreaction by senior management who as a consequence put unnecessary controls in place to enforce visibility.

In my experience, Scrum retrospectives pay big dividends at the end of each sprint but it’s easy to let them slide. In the post “Getting Lean with Scrum”, Christine Hegarty describes how you can add a simple process to retrospectives.

She suggests capturing a single kaizen in each retrospective to be added to the next Sprint backlog as a task. I think it’s a great idea, and could balance the needs of continuous improvement and business priorities while making retrospectives more actionable.

Misconceptions about Lean and Kanban

While reading Christine’s post I also spotted a guest post by Jim Coplien describing Kanban, Lean and Scrum. I remember Jim from the early days of the software patterns movement. He wrote a pattern language for software organizations. Jim is a co-author of the recent book Lean Software Architecture. In the blog post, he clarifies some of the many misconceptions of Kanban and it’s goals.

The challenge is to develop a learning organization that will find ways to reduce the number of kanban and thereby reduce and finally eliminate the inventory buffer. Remember: the kanban is an organized system of inventory buffers and, according to Ohno, inventory is waste…

He’s also concerned it’s been missappropriated in software development.

Kanban applies to repetitive work — building the same item again and again… Not everything can be replenished based on a pull system; some things must be scheduled… That is what software is like: We rarely build the same thing again and again.

The ideal Scrum team “does a little analysis, a little design, a little building, and a little testing all at once in very short cycles”. While Kanban can be a good fit for teams that view work as firefighting…

A good Scrum team automatically enjoys the provisions of kanban when practicing one-piece continuous flow. It is a built-in way to limit work in progress while encouraging teamwork. It gives team members time-boxed autonomy to carry out their plans while enabling them to better meet their planned forecast. Such maturity may be a foundation for later addition of techniques like kanban…

Learn more

Try Goshido, our collaboration & project management platform. Goshido is so flexible it can be used for Scrum, Lean and projects with traditional work-breakdown plans. Some of our customers do all three together. They map out a long term plan using Gannt charts then each team runs sub-projects as Scrum-Sprints.

The Business Value in Becoming an Agile Enterprise

Posted on 29. Feb, 2012 by ger in Leadership, New Ways to Work

In the last few weeks I’ve been asked the same two questions a number of times:

• Does this agile stuff really make a difference in financial terms?
• We don’t do software development, does agile really apply to us?

In this post, I’ll answer these questions (the short answers are yes and yes). I’ll tell you three short stories. Two stories quantify the benefits of agile. The final story is about a team who uses agile to run marketing campaigns.

What’s in it for us?

Michael Mah in “How Agile Projects Measure Up, and What This Means to You” analyzes two software projects. He compares their performance to projects of similar sizes from a large project-performance database. The projects measured lower costs, shorter time-to-market, and increased quality.

Follett Software used a technique called eXtreme Programming and achieved:

• Dramatically lower costs ($2.2m versus an average of $3.5m for similar sized products)
• Shorter time-to-market (7.8 months versus an average of 12.6 months)
• Less defects (121 versus 242)

BMC Software used Distributed Scrum. They added more staff (92 versus 40 average) to optimize time-to-market. They achieved:

• Dramatically shorter time-to-market (6.3 months versus an average of 15 months)
• Slightly lower costs ($5.2m versus an average of $5.5m for similar sized products). The extra cost of staff was offset by much shorter project duration.
• Slightly less defects (635 versus 713)

These are just two specific examples. David Rico completed a much larger meta-study looking at a large number of projects (which referenced a paper published in the EJIS that I co-authored with Brian Fitzgerald and Kieran Conboy).

Agile beyond software development: a retail marketing campaign

One of our customers decided to apply agile on retail marketing campaigns across 13 European countries.

Before agile they organized everything quarterly. Every quarter their campaigns had a number of open issues that spilled over and took time to resolve the following quarter. The backlog of open issues was ever increasing and pressure was mounting every quarter.

When they applied agile they split the teams into smaller units of 5 people or less. They ran projects in shorter iterations (monthly sprints instead of three month iterations). They observed a number of key benefits:

• Very few issues spilled over from one quarter to the next
• They were able to eliminate the historical backlog
• Team morale improved dramatically

Again this is one specific example. Stephen Denning has written an excellent book “The Leaders Guide to Radical Management” about applying agile principles to management in all kinds of businesses.

Learn more

If you’re a time-pressed business person, read Israel Gat’s short book “The Concise Executive Guide to Agile”. He gives a good overview of agile that focuses on the numbers and how to introduce agile in your enterprise.

If you’re based in the UK, “The Agile Leader” is a team of consultants and coaches who deliver Agile transformation to organisations.

Try Goshido, our collaboration & project management platform. Goshido is so flexible it can be used for projects in any part of your organization.

Photo Credit

whoalse by Allen

New features: multi-project dashboard & calendar

Posted on 10. Jan, 2012 by ger in Product

We’ve built some new features into Goshido that help you get better visibility of your projects.

Multi-project dashboard

The project list now includes features to tell you the status of all your projects at a glance. For each project you can see:

• Automatically updated %-complete
• Automatic countdown of time-remaining
• Color coded status indicators for actions/projects

When someone completes actions deep within a project, the %-complete and time-remaining is updated automatically at higher levels of the project.

This is reflected on the multi-project dashboard.

Calendar View

The new calendar view shows you actions you’re involved in which have deadlines which are due soon.

Learn more

Thank you for reading. Please try Goshido, our collaboration & project management platform. Goshido can can give you great visibility of all of your projects. We’d love to know what you think of these new features.

Five principles to improve productivity, reduce churn & increase profits

Posted on 18. Nov, 2011 by ger in Leadership, New Ways to Work

Five books describing a new approach to business

There’s a revolution happening in the way businesses are being run. This post will distill five key principles of this transition from five “new approach” business books. These principles will help you increase productivity, reduce staff turnover and increase profits.

Many books and blog posts have been written about this new approach to business. Many present case studies from companies like Apple, Best Buy, Enterprise, Semco, Salesforce, and WL Gore. The Whole Foods Market story is profiled in Gary Hamel’s “The Future of Management”.

The Whole Foods Story

Whole Foods Market is made up of many empowered teams. Each team:

• Has autonomy and is in essence a small business inside the store
• Has freedom but is held accountable
• Transparently publishes their performance (profit/hour)
• Can veto new hires
• Can decide what to stock

Each store is benchmarked 10 times a year. The pressure to perform comes from peers not managers. This non-hierarchical structure means decision making is distributed and small problems don’t fester before being noticed and addressed.

While this might sound new-age and chaotic, Whole Foods Market is the most profitable food retailer in the US (per sq foot). Whole Foods Market rallies around a clear purpose. “We want to improve the health and well-being of everyone on the planet through higher quality foods and better nutrition. We can’t fulfill this mission unless we are highly profitable.”

Other Success Stories

Other companies have made similar radical transitions and seen performance improvements:

• Salesforce switched their product development to agile and increased productivity by 38% and doubled their revenues over a two year period
• WL Gore has $2B in revenues and has been run as an innovation democracy for 40 years
• Best Buy started a “results only work environment” and increased productivity by 41% and reduced staff turnover costs by 90%
• Thogus Products a small manufacturing company increased output 67%

These revolutionary techniques come labelled in many ways: radical management, future management, agile businesses and protean organizations. Whichever label you choose, companies that use these techniques are surviving and prospering. Companies who stick to the old techniques of cost cutting, salary reductions and layoffs lose productivity, customers and frequently enter a downward spiral from which recovery is extremely difficult.

The key principles

Empowered teams of engaged individuals

Scott Page studied groups solving complex problems and found a cognitively diverse group of people outperforms a group of like minded experts. Teams which are given autonomy and control perform better. When people at the grassroots of the organization have a clear line of sight to customers they can see how they are contributing to the organizations goals.

Delivering true customer value

Unhappy clients can damage a brand. People with a line of sight to the customer feel more motivated. The meaning of their work is not the toy they’re assembling or the profit the company will make, but the delight on the face of a child. Companies like Enterprise have used net promotor scores to delight customers and turnaround their business. In contrast, David Carroll’s YouTube video “United breaks Guitars” has been watched over 11 million times (at the time of writing).

Using short iterations value adaptability over predictability

Projects with long timelines and complex Gantt charts repeatedly miss budgets and deadlines. The iterative approach was used to great success on Polaris submarine program in the 1950’s and 1960’s. By reducing the amount of work in progress and breaking large projects down into four weeks long or smaller iterations, teams can become far more effective.

Information radiators

Lack of management transparency has resulted in a number of disasters. Problems are brushed under the carpets. Well-meaning questioning is rooted out as dissent. When teams create dashboards to show progress there’s no need for status reports. Anyone can see information about the project. The best performing organizations have universal accountability.

Introspection and action

It’s hard to imagine how stopping a production line for a defect could be a good idea, but it is. The team must first recognize reality and the issues that exist but that’s less than half the battle. Taking remedial action is usually the hard part. Changing people’s behavior to fix systemic issues is even harder.

“Implementing continuous self-improvement requires a fundamentally different kind of mind-set from traditional management. It involves creating an environment in which the organization draws on the full talents and capacities of the people who work there… It’s about powering up the internal energy of teams so that they transcend their limitations and create products or services that generate client delight.” – Stephen Denning

The time to act is now

Maybe you’re thinking “yes this is important, we’ll do something about it soon.” Maybe you’re too busy. If your business is not improving it’s standing still. Companies who delay will: miss opportunities, lose customers, and be outmaneuvered by nimble competitors.

Learn more

Thank you for reading. We hope you found something useful. Please try Goshido, our collaboration & project management platform. Goshido can help you and your teams to apply new business principles to improve productivity.

If your company is an Enterprise Ireland client you can avail of training and support to introduce Lean and Agile techniques to your business.

Some books we recommend

Here are links and summaries to some great books on the “new approach” to business.

Stephen Denning’s book The Leader’s Guide to Radical Management applies agile techniques to the organization as a whole, not just a single team or the product development group. Denning translates Agile and Scrum from software development into general business terms.

The Way We’re Working Isn’t Working by Tony Schwartz, Jean Gomes and Catherine McCarthy creates a road map for a new way of working. At the individual level, they explain how we can build specific rituals into our daily schedules. At the organizational level, they outline new policies and practices that energize great performance.

In Succeeding with Agile, Mike Cohn describes success factors in applying an agile technique called Scrum in your organisation. It is mainly written from a software development perspective but there’s some valuable suggestions on team dynamics and the need for enlightened leadership. Chapters 10 and 12 describe the mind shift needed to become a “servant leader” of self-organizing teams.

Gary Hamel in The Future of Management makes the case that management innovation fuels long-term business success. He profiles a number of companies who have successfully reinvented management for their organizations.

In What Were They Thinking?: Unconventional Wisdom About Management, Jeffrey Pfeffer describes conventional business wisdom and the problems it can cause. He focuses on three common themes: understanding feedback effects, self-managing teams, and avoiding overcomplication.

Slides from Agile Tour 2011 – Principles of Scrum

Posted on 07. Nov, 2011 by ger in Events, New Ways to Work

A couple of weeks ago I spoke at AgileTour Dublin 2011. AgileTour aims to communicate the benefits of Agile for businesses and create leaders in Agile in all regions of the world.

I spoke about getting started and improving your organization using Scrum. You can get my slides from our website.

Other speakers included Colm O’hEocha from AgileInnovation, Fran O’Hara from Inspire Quality Services, Robert McGarry from Ignition Team, Richard Bowden from Cloud Consulting, and Alan Spencer from D&B. You’ll find details of the other talks on the Agile Tour website. Alan was speaking at the same time as me, so I missed his talk. From his slides it looks like he was discussing the benefits of Agile for business in general, a subject that’s close to our hearts too.

Thank you for reading

We hope you found something useful. Please try Goshido, our collaboration & project management platform. Goshido applies new principles for how work can be organized; the perfect blend of Agile, Lean, Productivity and Attention Management.

Cloud Computing Demystified

Posted on 15. Sep, 2011 by ger in Guides, New Ways to Work

Cloud computing can have huge benefits for a business of any size. If you’re running a small business you can share information and use world class software without up front costs. If you’re a large business you can consolidate complex applications onto less hardware, run them on someone else’s computers and access them over the internet.

I am one of the panelists at the Irish Executives conference in Galway Ireland on 15-Sep-2011. Here are some resources (articles, videos and books) that might help you learn more about the cloud software and how it can benefit your business.

Thank you for reading

We hope you found something useful. Please try Goshido, our collaboration & project management platform. Everyone is a project manager these days. Goshido helps teams of all shapes and sizes to get things done.

If you trial Goshido using the link above, you’ll get a 40% discount for you and your team if you decide to purchase.

Photo by barto, available under a Creative Commons attribution license