Things on the edge
Thoughts on technology and business
As far back as 2014, ThoughtWorks proposed the concept of an Inverse Conway Maneuver as a way to use team structure to drive the adoption of a desired architectural state. As discussed above, small independent development teams go hand in hand with the Microservices Architecture pattern.
Like application developers went from front end, back end, UX, testing, and production support roles to DevOps and full-stack developer roles, the traditional data roles of business analyst, ETL developer, data modeler, and production support can be transitioned into data engineering and analytics engineering roles.
This happens in concert with a move to transition the traditional business intelligence stack into the equivalent of a microservices stack. The transition to a microservices-based analytics platform requires reorganizing the team into more general-purpose roles. Both have a shared focus on data, but because the implementation is different, it will be helpful to initially identify two different roles.
The first role is Data Engineer. This is a software engineer role focused on building the data infrastructure and moving the data from sources to targets. Data infrastructure includes all the application infrastructure required to support the team’s analytics requirements.
In smaller companies, this role has a lot of infrastructure requirements, and it requires recruiting someone with server management experience. If this isn't possible, keeping the infrastructure requirements to a bare minimum will be a necessary. Fortunately, if done properly, this minimum viable infrastructure can meet a surprising range of requirements. In larger companies with more established IT departments, much of the existing infrastructure can be leveraged, be it local or provided as a service. In either case, data engineers work toward building an automation framework that can be shared across multiple analytics products.
The data engineer is also responsible for moving data from various sources to targets as part of product development. This is traditionally the extract and load portion of the ETL flow. This may include performing basic data profiling and implementing the storage of historical data sets in preparation for later analytics. This ensures that the data engineer understands, and has visibility to the data, up to and including the base data sets used by the analytics engineer.
Analytics Engineering combines the traditional BI roles of analyst, ETL developer, DBA, dashboard/report developer, and web developer. The analytics engineer works closely with the data engineer and customer to establish the data and analytic features required for a product. While the data engineering role acquires (extract and load) data from the source systems and people, the analytics engineer manipulates the data into the required form. This includes data modeling, transformation, and building basic analytic models.
The joint responsibilities of these roles include performance monitoring and tuning, as well as developing data services that other products consume. It is important to understand that these roles are fluid and are intended to help structure and expose the types of work being done by the team.
 Inverse Conway Maneuver (https://www.thoughtworks.com/radar/techniques/inverse-conway-maneuver).