Open source tools, code-based solutions and cloud technology have changed the working methods of data teams. For many years, data processing was done on-premise, and with proprietary and “closed” solutions that all had their own ways of development and use. All this brought significant challenges in scaling, developing new features, and time to market.

But today, as data engineers utilise new technologies, they must also develop a fresh mix of competences and develop the right set of working techniques as they shift towards more agile development methods.

Data teams using tried and tested software development methods find success

For starters, on top of agile development methods, DevOps thinking is becoming increasingly important, meaning many data teams now maintain solutions while simultaneously developing new things. All these factors have brought data teams into a new world, which requires a new kind of knowledge and a broader range of skills centered around software development topics and methods such as infrastructure as code, CI/CD pipelines and use of version control.

Adapting to this new way of working may be challenging at first, but data teams don’t have to reinvent the wheel by developing new methods. Working with software developers, who already have these skills, is a simple way to learn tried and tested software development methods in very little time.

Hands-on-work works

Our client's data team told us that deploying new features, which was done in a central location, was just too time-consuming. They asked for our help, and to help them out, we sent Juho Hautala, a Kodan software developer, to do hands-on work with the team. By implementing the best practices and conventions of software development, Juho helped improve the data team's processes, workflows and project times.

We delivered concrete results, making the project a success by enhancing performance, introducing automation, and building better quality systems in less time. Here's how it all happened.

Correct use of version control

Version control is a key part of a software developer's tool kit, and developers naturally know how to get the best out of it. During the project, we were happy to share the benefits of our knowledge of branching, merging and rebasing with our client.

As software developers, we know correct use of version control is key to avoiding merge conflicts, lost updates and unintentional overwriting. We shared our vast experience of version control to help the client define a clear strategy of the Git workflow, resulting in higher-quality code and speedier development.

The importance of high-quality code

Data teams are no strangers to writing code, but independently working on complex problems is a sure way to miss the simple stuff like adding comments, often resulting in messy codes that can only be understood by the creator, or code that proves difficult to maintain over time.

On the other hand, software developers have a deep understanding and talent for coding and employ best practices such as code coherence, automated testing and code reviews. These team-centric specific code writing methods produce clear work that is easy to read and edit.

To tackle shallow code reviews, we applied a hands-on approach and worked together with the data team. We made it easy to peer review work and share knowledge to enhance quality. Correctness, readability, comprehensibility and maintainability made their code more effective and easy to understand, resulting in shorter work times and fewer mistakes.

Results: shorter meetings, smoother workflows and effective teamwork

The software developer and the data team got on like a house on fire, and the data team quickly became comfortable with the code review process and proper use of version control. This advanced the project in several ways:

• Reduced number of meetings needed.

• No more long hours of code changes and manual code intervention.

• Duration of deployment meetings dropped from two hours to thirty minutes.

• Knowledge was promptly and effectively shared through discussions, chats, face-to-face meetings and weekly "show & tell" meetings, which saved time and created a fun learning environment.

• Developing and testing new ideas has become easier and more regular.

• A custom monitoring system has been taken into use.

Finding the right software developer

Of course, excellent results like this aren't achieved with best practices alone. Data teams must work with software developers who can teach the team new methods. It's not enough for software developers to stand on the sidelines and "coach" data teams. They need to be on the front line, working as team members to solve problems. They don't need to be data experts either, as even working with software developers with little knowledge of data science still brings impressive results.

Juho makes it sound simple: “It's impossible for an outsider to tell what "should" be done. You got to be part of the team to fully understand how it works and what kind of obstacles it encounters. Only that way you can help the team to develop its working processes and create positive change.”

Here at Kodan, we're highly experienced in building and maintaining APIs and infrastructures, and we've turned countless concepts into working products by innovating and experimenting. But that's not all. At Kodan, we're as passionate about teamwork as we are about tech, and have seen how super useful it can be to mix data teams and software developers. So if your data team is looking to enhance results and save hours of time, don't reinvent the wheel. Get in touch with the Kodan team, and see what we can do for you.

--

Now that you are here, see our other cases.