Django microservice architecture - evolution from monolithic to distributed system | Daoman PythonAI#

#django microservice architecture - evolution from monolithic to distributed system

📂 Stage: Part 3 - Advanced Topics 🎯 Difficulty Level: Expert ⏰ Estimated study time: 8-10 hours 🎒 Prerequisite knowledge: 部署最佳实践, 性能优化

#Table of contents

• 微服务架构概述
• 服务拆分策略
• API网关设计
• 服务通信机制
• 分布式数据管理
• 迁移策略
• 本章小结

#Microservice architecture overview

#What is microservices

Microservices is an architectural style that splits a huge monolithic application into a set of small services that run independently. Each service is built around specific business capabilities and collaborates with each other through lightweight communication protocols (such as HTTP/REST or gRPC). Its core concept is "divide and conquer" to make complex systems manageable.

Different from the traditional layered architecture (where all logic is stuffed into a code warehouse), microservices emphasize:

• Organize teams by business: No longer divide teams according to technical lines such as front-end/back-end/database, but let one team be responsible for a complete business area (such as user team, order team) end-to-end, and can make independent decisions and quickly iterate.
• Independent deployment and expansion: A problem with one service will not bring down the entire system; whichever service is under great pressure, only add machines to that service, resulting in higher resource utilization.
• Freedom of Technology Heterogeneity: Service languages, frameworks, and databases can all be selected on demand and are no longer hijacked by a single technology stack. For example, the order service can use django + PostgreSQL, but the recommendation service uses Python + Redis.

#When are microservices needed?

Not all projects are suitable for microservices. The value of microservices will only be reflected when your system has the following characteristics:

• Large and complex businesses: such as e-commerce platforms and SaaS products, with many functional modules and high coupling.
• Multi-team parallel development: Teams in different cities or even different time zones need to develop independently and do not want to wait for each other.
• Fast-changing business needs: A certain function needs to be frequently modified and launched without affecting the overall stability.
• Single application left over from history: The code has become so bloated that it is difficult to maintain and needs to be gradually refactored.

#Realistic challenges that need to be faced

Microservices are not a silver bullet and introduce additional complexity:

• Difficulty in distributed debugging: A request may pass through five or six services, and locating the problem requires complete link tracking.
• Data consistency problem: It is difficult to rely on transactions in one database to ensure strong consistency when writing across services.
• Surge in operation and maintenance costs: Logging, monitoring, health checks, and configuration management all require specialized infrastructure, which places higher demands on the team.

These challenges require us to plan response strategies at the beginning of the design instead of patching them later.

#Service split strategy

#Splitting principle

How to dismantle it is a good question. It is recommended to refer to the bounded context idea in Domain Driven Design (DDD), combined with the following principles:

1. Bounded Context: Define clear business boundaries for each service. The service only handles matters within its own domain and does not cross the boundaries.
2. Independent data sovereignty: Each service has its own database and can only expose data to the outside world through API. Direct cross-database query in the code is never allowed.
3. Single Responsibility: A service only solves one core business problem to avoid becoming a "mini monomer".
4. High cohesion, low coupling: The internal logic of services is closely related, and the dependencies between services are as small as possible and one-way.

#Example of e-commerce system splitting

Let’s use the most common e-commerce platform as an example of splitting. Several core services can be separated out:

• User Service: registration/login, rights management, personal information maintenance
• Commodity Services: Product information management, classification, inventory control
• Order Service: Order generation, status transfer
• Payment Service: third-party payment and refund processing
• Notification Service: Email, SMS, and site message sending

Each service is an independent django project with its own database and deployment unit.

In the order service model, we will not create a pointer to the user table or product table.ForeignKey, but only saves the ID association, because those tables are in other databases and cannot be connected at all:

# order_service/models.py
from django.db import models

class Order(models.Model):
    STATUS_CHOICES = [
        ('pending', '待支付'),
        ('paid', '已支付'),
        ('shipped', '已发货'),
    ]
    # 只保存外部服务的ID，不做数据库层面的关联
    user_id = models.IntegerField()
    total_amount = models.DecimalField(max_digits=10, decimal_places=2)
    status = models.CharField(max_length=20, choices=STATUS_CHOICES, default='pending')
    created_at = models.DateTimeField(auto_now_add=True)

class OrderItem(models.Model):
    order = models.ForeignKey(Order, on_delete=models.CASCADE, related_name='items')
    product_id = models.IntegerField()
    quantity = models.PositiveIntegerField()
    unit_price = models.DecimalField(max_digits=10, decimal_places=2)

This approach forces us to obtain user or product details through API calls. Although there is an extra step of remote calling, it results in complete decoupling of the service and data autonomy.

#API gateway design

#Why do I need a gateway?

After microservices are deployed, the number of services increases and addresses change frequently (especially when using containers). If the client directly interacts with each service, it will face a lot of trouble: it needs to remember a bunch of addresses, deal with differences in authentication methods, and write complex fault-tolerant logic. API Gateway is the unified entrance for each service. It is like a reverse proxy and is responsible for:

• Request Routing: Forward the request to the corresponding backend service based on the URL prefix, such as/api/users/Go to User Services.
• Unified Authentication: Complete JWT or OAuth2 verification at the gateway layer, and then put the parsed user information in the request header and transparently transmit it to the internal service, so that the backend does not need to repeat verification.
• Traffic Control: Current limiting, circuit breaker, and downgrade to protect the backend from being overwhelmed by sudden traffic.
• Protocol conversion: Provide RESTful interface externally, and call services internally through gRPC or other protocols.

#Implement a simple gateway in django

We can use django REST Framework to quickly build a lightweight gateway. The core logic is to maintain a routing mapping table and forward requests.

# gateway_app/views.py
from rest_framework.views import APIView
from rest_framework.response import Response
import requests
from django.conf import settings

class GatewayView(APIView):
    # 路由映射表，生产环境中可以存入数据库或注册中心（如 Consul）
    ROUTE_MAP = {
        '/api/users/': 'http://user-service:8000',
        '/api/products/': 'http://product-service:8000',
        '/api/orders/': 'http://order-service:8000',
    }

    def dispatch(self, request, *args, **kwargs):
        # 1. 匹配目标服务
        target_url = None
        for prefix, base in self.ROUTE_MAP.items():
            if request.path.startswith(prefix):
                target_url = f"{base}{request.path}"
                break
        if not target_url:
            return Response({'error': 'Route not found'}, status=404)

        # 2. 过滤并透传请求头
        headers = self._filter_headers(request.META)

        # 3. 发起转发请求，做好超时和exception-handling
        try:
            resp = requests.request(
                method=request.method,
                url=target_url,
                headers=headers,
                params=request.query_params,
                data=request.body,
                timeout=30
            )
            return Response(
                resp.json() if resp.content else None,
                status=resp.status_code
            )
        except requests.exceptions.Timeout:
            return Response({'error': 'Service timeout'}, status=504)
        except Exception as e:
            return Response({'error': 'Service unavailable'}, status=503)

    def _filter_headers(self, meta):
        """只保留业务相关的HTTP头，去掉django内部元信息"""
        headers = {}
        for k, v in meta.items():
            if k.startswith('HTTP_'):
                # 转换为标准的头部格式，如 HTTP_X_USER_ID -> X-User-Id
                key = k[5:].replace('_', '-').title()
                headers[key] = v
        return headers

The actual production environment will mostly use mature solutions (Nginx, Kong, Envoy, etc.), but building it yourself with django can give you maximum control over the gateway behavior, and it also facilitates quick verification of the architecture in the early stages.

#Service communication mechanism

Microservices need to "talk" between each other, and there are two common modes: synchronous and asynchronous.

#Synchronous communication: REST/gRPC

Suitable for scenarios where immediate results are required. For example, before a user places an order, the order service needs to query the user's balance or coupons in real time. At this time, it must wait synchronously.

Here is a simple client-side wrapper for synchronous calls:

# user_service_client.py
import requests
from django.conf import settings

class UserServiceClient:
    BASE_URL = settings.USER_SERVICE_URL

    @classmethod
    def get_user(cls, user_id):
        try:
            resp = requests.get(
                f"{cls.BASE_URL}/api/users/{user_id}/",
                timeout=5
            )
            resp.raise_for_status()
            return resp.json()
        except Exception as e:
            # 实际项目中可以添加降级逻辑（如返回缓存数据或默认值）
            return None

Synchronous calling is intuitive and simple, but it creates strong dependencies on the caller and can easily cause cascading failures (service A times out, causing service B to also hang up). So be sure to set a timeout and use it with a fuse.

#Asynchronous communication: message queue

Suitable for scenarios such as decoupling, improving throughput, and achieving eventual consistency. For example, after an order is generated, the user needs to be notified and inventory deducted. These operations do not have to block the order creation process and can be triggered asynchronously through events.

Based on Redis's publish/subscribe and list, a lightweight message queue can be built:

# message_queue.py
import redis
import json
import uuid
from datetime import datetime
from django.conf import settings

r = redis.from_url(settings.REDIS_URL)

class MessageQueue:
    @staticmethod
    def publish(event_type, data):
        """发布事件到频道和持久化队列"""
        event = {
            'id': str(uuid.uuid4()),
            'type': event_type,
            'data': data,
            'timestamp': datetime.utcnow().isoformat()
        }
        # Pub/Sub 用于即时通知
        r.publish('events', json.dumps(event))
        # 同时推入队列列表，保证消息不丢失（消费者轮询处理）
        r.lpush(f"queue:{event_type}", json.dumps(event))

    @staticmethod
    def subscribe(event_type, handler):
        """订阅事件（简化版，生产建议用 Celery 或 Kafka 消费者）"""
        pubsub = r.pubsub()
        pubsub.subscribe('events')
        for msg in pubsub.listen():
            if msg['type'] == 'message':
                event = json.loads(msg['data'])
                if event['type'] == event_type:
                    handler(event)

After the order service creates an order, it publishes anorder_createdEvents, notification services and inventory services can consume it asynchronously:

# order_service/views.py
from rest_framework.views import APIView
from rest_framework.response import Response
from .models import Order
from .message_queue import MessageQueue

class OrderCreateView(APIView):
    def post(self, request):
        # 1. 本地事务：创建订单
        order = Order.objects.create(
            user_id=request.user.id,
            total_amount=request.data['total_amount'],
            status='pending'
        )

        # 2. 发布事件，触发后续异步流程
        MessageQueue.publish('order_created', {
            'order_id': order.id,
            'user_id': order.user_id,
            'items': request.data['items']
        })
        return Response({'order_id': order.id}, status=201)

#Distributed data management

#Selection of consistency model

In a monolithic system, database transactions can easily ensure that data and business status are consistent. When it comes to distributed systems, according to the CAP theorem, we must make a choice between consistency (C), availability (A), and partition fault tolerance (P). Most Internet businesses will choose eventual consistency (BASE theory), which allows short-term inconsistencies, but all node data will reach consensus after a certain period of time.

#Saga pattern: a powerful tool for achieving eventual consistency

Saga is the mainstream pattern for handling cross-service distributed transactions in microservices. It splits a large transaction into a series of local atomic transactions. After each transaction is completed, the next transaction is triggered through a message; if any step fails, a series of compensation operations are performed to roll back the previous operation.

Take the e-commerce order process as an example:

• Step 1: Withholding inventory of goods and services
• Step 2: User service deduction balance
• Step 3: The order service changes the order status to "Paid"

If the deduction of the balance fails, compensation is required: the withheld inventory is released and the order is closed.

Here is a simplified Saga implementation:

# order_service/saga.py
from .models import Order
from .user_service_client import UserServiceClient
from .product_service_client import ProductServiceClient

class OrderSaga:
    @staticmethod
    def execute(order_id):
        order = Order.objects.get(id=order_id)
        try:
            # 步骤1：预扣库存（内部由商品服务本地事务保证）
            ProductServiceClient.reserve_inventory(order.items)

            # 步骤2：扣减用户余额
            UserServiceClient.deduct_balance(order.user_id, order.total_amount)

            # 步骤3：更新订单状态
            order.status = 'paid'
            order.save()

        except Exception as e:
            # 任何一步失败，执行补偿
            OrderSaga.compensate(order_id)
            raise e

    @staticmethod
    def compensate(order_id):
        order = Order.objects.get(id=order_id)
        # 释放库存
        ProductServiceClient.release_inventory(order.items)
        # 如果余额已扣减，发起退款
        if order.status == 'paid':
            UserServiceClient.refund_balance(order.user_id, order.total_amount)
        # 最终关闭订单
        order.status = 'cancelled'
        order.save()

Note: The internal operations of each service here (such as deducting inventory and deducting balances) are still their own local database transactions. They are strung together through the coordinator (Saga), and global final consistency is guaranteed through compensation. This model must handle issues such as idempotence and null compensation. In practice, it is often paired with event sourcing or distributed transaction frameworks (such as Seata) to reduce complexity.

#Migration strategy

To smoothly migrate a running single application to microservices, you cannot “replace it and start over”. Strangler Mode is recognized as the safest method in the industry.

#The core idea of ​​the Strangler mode

• New functions are directly uploaded to microservices: All new functions are implemented with independent microservices and mounted next to the existing system through the API gateway.
• Gradual replacement of old functions: Start with peripheral, low-risk functions (such as notifications, evaluation systems), and gradually advance to core modules (products, orders). Only move a small part at a time and verify that it is stable before continuing.
• Unified entrance traffic diversion: Through the gateway, traffic is dynamically switched to new microservices. Once a problem occurs, it can be quickly switched back to the single entity.

#Phased implementation recommendations

1. Check out the existing system: Figure out the boundaries, data dependencies and call links of each module in the monomer.
2. Build infrastructure: First prepare the bases such as API gateway, service registration and discovery, centralized logging and monitoring.
3. Split edge services: Choose a module that is least coupled with the core business to practice, such as notification services, and accumulate experience in microservice development and operation and maintenance.
4. Gradually split the core: Core services such as products and orders will be migrated step by step, and data will be double-written or synchronized throughout the process to ensure that the data in the old and new systems are consistent.
5. Monolith offline: When all traffic is cut to the microservices and runs stably for a period of time, the monolithic application is officially retired.

#Summary of this chapter

In this chapter, we systematically sort out the key knowledge points of Django microservice architecture:

• Service Splitting: Based on the bounded context of domain-driven design, each service is only responsible for a piece of business and has independent data storage.
• API Gateway: Provides a unified entrance for all services, implements routing, authentication, current limiting and other functions, and isolates the client from internal details.
• Service communication: REST or gRPC is used for synchronous calls, and message queue is used for asynchronous calls, to cope with immediate needs and eventual consistency scenarios respectively.
• Data Consistency: Give up strong consistency, embrace eventual consistency, and achieve reliable distributed transaction compensation through Saga mode.
• Migration Strategy: Adopt the strangler model and gradually replace from edge to core to minimize risks.

💡 Core Point: Microservices are a means to solve complex system management problems, not a panacea. It is a reasonable choice only when the size of the team, business complexity, and operation and maintenance capabilities are sufficient to support its costs. Don't use microservices for the sake of microservices - starting from a singleton and evolving when you really need it is often the most pragmatic path.