System design with ChatGPT?
A guide to using AI for high-level software decisions
If youāre reading this, youāve probably already used LLMs to write code.
When I first started using AI to help with my work, I was convinced that it would never be able to help me make high-level software architecture decisions. I was wrong about that.
In this guide, weāll go over using ChatGPT for architecture brainstorming, evaluating tradeoffs, and making informed decisions. The approach Iāll lay out here is practical, with actionable steps, a case study, and best practices.
With this sort of thing I usually get best results using a reasoning model. Thereās a reason OpenAI calls them āthe plannersā! If you can, use o1, o1-pro, o3-mini, etc.
Understanding system design with ChatGPT
System design is usually a collaborative process. ChatGPT can act as a creative thought partner during this process, helping you explore ideas that you might not consider alone. An LLM like ChatGPT can suggest design options, recall common architecture patterns, and explain trade-offs between different approachesā.
Itās like having an staff engineer available on demand to bounce ideas off of. Thereās certainly shortcomings, but letting those scare you off is myopic.
Using ChatGPT in system design discussions can augment you in a few notable ways:
Brainstorming: ChatGPT is excellent at generating a range of possibilities. For example, if you describe a problem, it can propose multiple architectural styles or components that might fit (layered architecture, microservices, event-driven design, etc.). You wonāt like many of the ideas, and thatās okay!
Trade-off Analysis: The model can discuss the pros and cons of design decisions. You can ask it about scalability vs. consistency, or SQL vs. NoSQL, and it will articulate the benefits and drawbacks of each choice. LLMs can explain these trade-offs in detailā with increasing accuracy.
Knowledge Base: ChatGPT has been trained on vast amounts of technical content. It can recall principles from well-known system design resources (e.g. principles of CAP theorem, REST vs. gRPC, caching strategies) and best practices (like using CDN for static assets, or sharding a database for scale). Essentially, it can surface relevant knowledge on demand, which might save time searching through documentation.
Architecture Documentation: You can even use ChatGPT to help draft design docs once youāre done iterating. For instance, after making a decision, you could ask it to write a brief summary of why that choice was made over alternatives. This first draft can be a time-saver (and forces you to clearly describe requirements to the AI)ā
Itās important to note that ChatGPT will not replace your expertise or critical thinking. Think of it as a smart assistant that can amplify your architectural thinking. You remain the decision-maker who must validate which suggestions make sense to implement.
Using ChatGPT as a thought partner
To get the most value, you should treat ChatGPT as an interactive thought partner. This means crafting your prompts and questions in a way that guides the AI to produce useful, relevant insights. Here are some strategies for using ChatGPT effectively during system design discussions:
Set the stage in your prompt: Provide context about the problem youāre trying to solve. Include the domain (e.g. fintech, e-commerce), high-level requirements, and any specific constraints youāre aware of. For example:
Iām designing a real-time chat application for 1 million users. I need low latency messaging and high availability. As my thought partner, what high-level architecture might suit this, and what should I consider?
Ask for alternatives and trade-offs: Instead of asking a very broad question like āHow should I design X?ā, break it down. For instance:
What are a few different architecture approaches for X, and the trade-offs of each?
Can you discuss the pros and cons of a microservices architecture for this use case versus a monolith?
By explicitly requesting a comparison or multiple options, you avoid getting a one-sided answer. ChatGPT can enumerate options and even create comparison tables or lists of pros/cons for each approach.
Be iterative: Use a step-by-step dialog. Start with high-level ideas, then drill down. You might begin with, āWhat components would I need for an online marketplace system?ā Once you get an answer, follow up with more specific questions:
How could I scale the payment component if traffic increases 10x?
What database would be appropriate if I need strong consistency for transactions, and why?
This iterative querying lets you progressively refine the design with ChatGPTās input at each layer of detail.
Role-play scenarios: You can ask ChatGPT to adopt a role or perspective. For example,
Act as a Chief Architect reviewing my design. These are my main components (describe them)... What questions or concerns might you raise?
This often prompts the AI to generate critical questions or point out potential issues, similar to an expert peer review.
Request Best Practices and Pitfalls: Leverage ChatGPTās knowledge of industry best practices. Ask things like,
What are best practices for ensuring data consistency in a distributed system?
What common mistakes should I avoid when designing a rate-limiting mechanism?
This can surface well-known pitfalls (e.g. not paging large database queries, or ignoring network partition handling) so you can address them early. Itās like having a checklist generated for you.
By treating the conversation as a brainstorming session, where you continuously refine prompts and dig deeper based on previous answers, you keep ChatGPT focused and useful. Remember that clarity and specificity in your prompts usually yield better answers. If an answer is too generic, ask follow-up questions with more details or say āplease elaborate on Xā. If an answer seems off-track, you can correct the course by providing the correct information and asking it to consider that.
A Step-by-step system design process with ChatGPT
Letās break down a typical high-level system design workflow into concrete steps and see how ChatGPT can assist at each stage.
Step 1: Clarifying Requirements
This step being necessary is why I think software engineering isnāt going away as a profession. See the below issue for more on that š
Itās up to you to actually be able to articulate what you want your final product to do, including the less-obvious requirements (scale, performance, reliability, security, etc.)
If youāre working at a medium-large business, youāll have to work with your stakeholders to understand this. If youāre a smaller team, it might be up to you and your AI companions :)
Use ChatGPT to ensure you have a comprehensive understanding of the problem before diving into solutions. For example, describe the system context to ChatGPT and ask a critical question:
What questions should I ask about requirements for this system?
The AI might respond with a list of key questions or assumptions that can reveal requirements you forgot to consider.
You can also have ChatGPT brainstorm the requirements themselves. Provide whatever you know, then ask:
Given this scenario, what do you think the core functional requirements are? And what about important non-functional requirements?
ChatGPTās ability to simulate a conversation with stakeholders can validate your understanding. The AI might surface edge cases (āWhat happens if a payment fails?ā) or constraints (āDoes the system need to comply with GDPR?ā) early on. By the end of this step, you should have a refined list of what the system must do and the key constraints it must satisfy.
Step 2: Identifying key components
With clear requirements, the next step is to sketch out some high-level architecture ā the major components or subsystems of the system and how they interact. Essentially, you break the system into building blocks.
So how can ChatGPT help?
ChatGPT can assist by suggesting a decomposition of the system. You can prompt it with something like,
Given these requirements (list them), what would a high-level system architecture look like? What are the main components or services I should have?
For example, if designing an online store, ChatGPT might list components: a frontend UI, an authentication service, product catalog service, order service, payment service, a database for orders and products, a caching layer for fast product lookup, etc. It could also suggest using a message queue between services (for tasks like sending order confirmation emails asynchronously) if itās a well-understood pattern for that domain.
Youāre unlikely to get super creative answers, and thatās okay. Do you really want your system to be super unique? Me neither š¬
At this stage, you can engage in a back-and-forth with the AI to refine the component list:
āDo I need a separate service for X, or can it be part of the main application?ā (ChatGPT might explain the trade-offs of splitting it out, e.g. for independent scalability or team ownership.)
āWhat type of database would each service use?ā (It might propose relational for transactional data, NoSQL for something like caching or analytics, etc.)
āShould I consider an event-driven architecture here?ā (It might respond with scenarios where an event bus would help decouple components, versus when it might be overkill.)
Always tailor the components to your specific requirements ā if ChatGPT suggests a component that doesnāt make sense for your use case, youāre not stuck with it. LLMs donāt get offended if you ignore a suggestion.
Step 3: Diagramming and trade-offs
For even a moderately complex system itās helpful to create a visual representation of the system ā a high-level architecture diagram. This could be a simple boxes-and-arrows diagram showing clients, services, databases, and integrations and how data flows between them. I like to use Excalidraw for this sort of thing.
Diagrams help communicate the design to others and ensure you have a correct mental model of the component interactions.
ChatGPT wonāt draw images directly in the chat, it can help you turn your ideas into a diagram in a few ways:
Structured Descriptions: You can ask ChatGPT to describe the architecture in a structured way thatās easy to translate into a diagram. Iāve had a ton of success with this in drawing graphs to explain technical concepts in my educational writing.
UML or Pseudo-diagrams:You can ask for an ASCII art diagram or a description of relationships. Sometimes, a simple list of components and arrows between them in text form (as ChatGPT outputs) is enough to then manually create a neat diagram in a tool like draw.io or Lucidchart.
Once you have your diagram, itās a good idea to evaluate tradeoffs.
For each major decision, you can explicitly ask ChatGPT to weigh options.
By systematically querying ChatGPT on each axis of concern (scalability, consistency, availability, latency, cost, complexity), you ensure youāre considering the implications of your design choices. The model can sometimes articulate known principles or point out where a certain choice might violate a requirement. It can also remind you of alternatives: for example, if you ask about scaling a database, it might mention techniques like read-replicas, caching, or sharding, each with their trade-offs.
Example case study: designing a pizza ordering system (with ChatGPT)
To make the above process more concrete, letās walk through a simplified example. Weāll design a pizza ordering system with a real-time order tracker (I used to work at a Dominoās so Iām partial to the Dominoās Pizza Tracker ). At each step, Iāll show you how to use ChatGPT as part of the workflow.