CPython Pipeline: Tokenization, PEG Parser, AST & Bytecode Compilation

1Concept

CPython compiles .py source text through 4 distinct pipeline stages: 1) Lexer/Tokenizer produces token streams; 2) Parser generates Concrete Parse Tree & Abstract Syntax Tree (AST); 3) Compiler transforms AST into Python bytecode (.pyc); 4) CPython Evaluation Loop (the main _PyEval_EvalFrameDefault C switch-loop) executes instructions onto the evaluation stack.

2Architecture Diagram

[ Python Source (.py) ]
          |
          v
   [ Lexer / Tokenizer ] ---> Generates Token Stream (NAME, NUMBER, OP)
          |
          v
   [ Parser (PEG) ]      ---> Generates Abstract Syntax Tree (AST)
          |
          v
   [ Bytecode Compiler ] ---> Generates Code Objects (.pyc / __pycache__)
          |
          v
   [ CPython Eval Loop ] ---> Executes opcodes in _PyEval_EvalFrameDefault()

3Code Example

import dis
import ast

source_code = '''
def calculate_tax(salary: float, rate: float = 0.2) -> float:
    total_tax = salary * rate
    return total_tax
'''

parsed_ast = ast.parse(source_code)
print("=== AST Node Hierarchy ===")
print(ast.dump(parsed_ast, indent=2)[:260] + "...")

code_obj = compile(source_code, filename="<tax_calc>", mode="exec")
print("\n=== Bytecode Opcodes Disassembly ===")
dis.dis(code_obj)

4Expected Output

=== AST Node Hierarchy ===
Module(
  body=[
    FunctionDef(
      name='calculate_tax',
      args=arguments(
        posonlyargs=[],
        args=[
          arg(arg='salary', annotation=Name(id='float', ctx=Load())),
          arg(arg='rate', annotation=Name(id='float', ctx=Load()))],...

=== Bytecode Opcodes Disassembly ===
  0           0 RESUME                   0
  2           2 LOAD_CONST               0 (<code object calculate_tax>)
              4 MAKE_FUNCTION
              6 STORE_NAME               0 (calculate_tax)
              8 LOAD_CONST               1 (None)
             10 RETURN_VALUE

5Key Takeaways

• ✓CPython compiles source code to AST before producing bytecode instructions.
• ✓Use `dis.dis()` to inspect CPython stack-based bytecode opcodes.
• ✓__pycache__ contains precompiled bytecode (.pyc) to speed up module load times.