Why Software Systems Get Complex?

Complexity is one of the most talked about problem in Software Systems. The software that starts with simple design, code and release process eventually becomes complex and it goes to an extent when there is only one answer to all the problems — COMPLEXITY.

The purpose of this article to talk about few points that would help us in understanding the reason behind complexity.

Let’s look into the journey of software evolution.

1st Release : We did It !!

The first release is very special. All the stakeholders works super hard, application goes live and everyone is super excited about it.

The only issue is that 1st release isn’t perfect as team had to introduce few tech debts to meet timelines and on top of it, few live issues gets reported as customers start using the software.

The following diagram summarizes the expected vs actual state of software.

The ideal state of software system can be achieved by fixing all the live issues and tech debts but these are part and parcel of the software development and has to be accepted as reality. The part that can’t and shouldn’t be accepted is “Losing Control of them”.

Next Release : Hurray !!!

After 1st release, business wants to push next set of features soonest to meet immediate business goals. The tech debts from previous release are added to the backlog and team start development work again. Team works hard, develops the feature and roll it out to Production. Hurray !!

The only problem is few more tech debts gets introduced as everyone’s hands were tight due to critical timelines.

This cycle continues and soon team starts giving higher effort estimates for similar feature and at the same time, the number of live issues starts going up and then team is asked to find out WHY this is happening.

Next Release : Lets Do It !!!

Management and team member realizes that it is time to take action and fix critical tech debts. Team does the analysis and found that there are debts in multiple areas:

• Architectural Tech Debts
• API Tech Debts
• Code Tech Debts
• Test Tech Debts
• Documentation Tech Debts

Team spent time in prioritizing the tech debts and start the development works with low hanging fruits such as increasing the test coverage, refactor the bad code, adding documentation etc but as team starts looking into significant tech debts it imposes major challenges - once a simple tech debt now starts to look monstrous as many features are using that specific part of code and refactoring it could lead to cascading effect.

Examples:

• API contract changes — Multiple clients have to upgrade.
• Libraries — Multiple applications have to change logic.

This limits the team to do complete refactoring and they park few items in backlog for future.

This cycle of releases, fixes continues and if team is able to tackle the significant tech debts over time then they tame the complexity otherwise complexity starts taming the team.

The Bigger Picture

The software systems are designed and developed to tackle specific business problems but can we say that if there is no change in functional requirement then no change will be required to software systems?

The simple answer is NO and the main reason for this is that there are many external factors that leads to changes in the application. Let’s look into the following application landscape diagram:

Enterprise Landscape

The changes to an application and the corresponding tech debts can occur due to changes at various level in technical and non technical areas.

1. Enterprise Level Changes
2. Domain / Org Level Changes
3. Dependencies Changes
4. People Changes

Enterprise Level Changes

Enterprise level technology changes happens time to time that have unavoidable impact and requires changes in the applications. Enterprise architects might decide to :

• Move away from legacy technologies such as C, C++.
• Adopt new tools such as Splunk, Elastic Search, Kafka for observability.
• Introduce New CI/CD tools, Source Code Management Software etc.
• Introduce New Software and Frameworks for Security.

Domain / Org Level Changes

Domain / Org level changes can also indirectly affect the application.

• Adoption of new third party vendor tools for Rules Authoring.
• New In House Tools for doing domain level monitoring, audit etc.

Dependencies Changes

Downstream Dependencies

This is one of the most common cause of the changes in application.

• Downstream deprecates old API and introduces new one (REST to GraphQL)
• Downstream moves from AMQ to KAFKA events.
• Downstream Tech Debts?

Technologies and Framework Dependencies

Application is built using many framework dependencies and often these are required to upgraded due to various reasons.

• Java, Javascript Version Upgrade and Support.
• Log4J upgrade due to security reasons.
• No community support for open source frameworks.

Infra Dependencies

The infra level changes are rare but sometimes leads to changes in the application.

• QA, Dev and Prod configuration management tools changes.
• Multi Data Center (Active — Active)

People Changes

The people change can lead to positive / negative affect when it comes to software development.

• If the team members who developed the core features leaves the org then it can lead to the knowledge gap if new team members are not able to fully comprehend the solution.
• New architect/manager/ product owner has different priorities and vision.

So considering the big picture, the tech debts need not arise only from feature build outs as changes to the software are driven by many factors.

Conclusion

The software systems gets complex over the period of time so in order to keep it in desired state, the main goal of team should be to “NOT Lose Control Over Tech Debts”.

You can refer to next article for Tackling the tech debts !!

A software system is bound to get complex when rate of introducing the significant tech debts is higher than rate of fixing them.