Skip to content

README.md

Latest commit

 

History

History
274 lines (211 loc) · 9.67 KB

README.md

File metadata and controls

274 lines (211 loc) · 9.67 KB

Distributed File System (DFS)

A simplified Distributed File System inspired by GFS and HDFS. Features a master node for metadata management and multiple chunk servers for distributed data storage with automatic replication.

Quick Start

# Build all binaries
go build -o bin/dfs-master ./cmd/master
go build -o bin/dfs-chunkserver ./cmd/chunkserver
go build -o bin/dfs-client ./cmd/client

# Start the master server
./bin/dfs-master

# Start chunk servers (in separate terminals)
./bin/dfs-chunkserver -port :9001 -master localhost:8000 -dir ./data/cs1
./bin/dfs-chunkserver -port :9002 -master localhost:8000 -dir ./data/cs2

# Upload a file
./bin/dfs-client put /path/to/local/file.txt remote-name.txt

# Download a file
./bin/dfs-client get remote-name.txt /path/to/download/dir

Features

✅ Implemented

Feature Description
File Upload Chunked upload with automatic allocation
File Download Reassembly from distributed chunks
Chunk Replication Configurable replication factor (default: 2)
Heartbeat System Bidirectional streaming for health monitoring
Dead Server Detection Automatic detection and replication triggering
Replica Placement Storage-aware server selection
Structured Logging slog-based logging with levels
Client SDK High-level API for file operations
CLI Tool Command-line interface for put/get operations

🔜 Planned

  • Write pipeline (chain replication)
  • Lease management for primary writes
  • Checksum verification
  • Graceful shutdown handling

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                           Client                                 │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │  DFSClient SDK                                           │    │
│  │  ┌───────────┐  ┌────────────┐  ┌────────────────────┐  │    │
│  │  │ Uploader  │  │ Downloader │  │ MasterClient       │  │    │
│  │  └───────────┘  └────────────┘  └────────────────────┘  │    │
│  └─────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────┘
                              │
                              │ gRPC
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                        Master Server                             │
│  ┌─────────────┐  ┌──────────────┐  ┌────────────────────────┐  │
│  │ Metadata    │  │ Heartbeat    │  │ Chunk Placement        │  │
│  │ Manager     │  │ Manager      │  │ Manager                │  │
│  └─────────────┘  └──────────────┘  └────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘
                              │
                              │ gRPC (Heartbeat + Tasks)
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                       Chunk Servers                              │
│  ┌───────────────┐  ┌───────────────┐  ┌───────────────┐        │
│  │ ChunkServer 1 │  │ ChunkServer 2 │  │ ChunkServer N │        │
│  │ :9001         │  │ :9002         │  │ :900N         │        │
│  └───────────────┘  └───────────────┘  └───────────────┘        │
└─────────────────────────────────────────────────────────────────┘

Component Diagram

flowchart TB
    classDef client fill:#fef08a,stroke:#eab308,color:#1c1917
    classDef master fill:#c4b5fd,stroke:#8b5cf6,color:#1c1917
    classDef chunk fill:#93c5fd,stroke:#3b82f6,color:#1c1917

    Client["Client<br/><br/>CLI / SDK<br/>File operations"]:::client --> |gRPC| Master

    subgraph Master["Master Server"]
        MM["Metadata Manager<br/>File → Chunk mapping"]:::master
        HM["Heartbeat Manager<br/>Monitor ChunkServers"]:::master
        CM["Chunk Placement<br/>Replica selection"]:::master
    end

    subgraph CSGroup["ChunkServers"]
        CS1["ChunkServer 1<br/>:9001"]:::chunk
        CS2["ChunkServer 2<br/>:9002"]:::chunk
        CSN["ChunkServer N<br/>:900N"]:::chunk
    end

    Master --> |Heartbeat| CS1
    Master --> |Heartbeat| CS2
    Master --> |Heartbeat| CSN

    Client --> |Read/Write| CS1
    Client --> |Read/Write| CS2
    Client --> |Read/Write| CSN
Loading

Write Sequence

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#c4b5fd', 'secondaryColor': '#93c5fd', 'tertiaryColor': '#fef08a', 'primaryTextColor': '#1c1917', 'lineColor': '#8b5cf6'}}}%%
sequenceDiagram
    participant C as Client
    participant M as Master
    participant CS1 as ChunkServer 1
    participant CS2 as ChunkServer 2

    C->>M: AllocateChunk(filename, index)
    M-->>C: ChunkID + ReplicaServers
    C->>CS1: UploadChunk(stream)
    CS1-->>C: Success
    
    Note over M,CS2: Replication via heartbeat
    M->>CS1: ReplicationTask
    CS1->>CS2: ReplicateChunk
    CS2-->>CS1: Ack
Loading

Read Sequence

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#c4b5fd', 'secondaryColor': '#93c5fd', 'tertiaryColor': '#fef08a', 'primaryTextColor': '#1c1917', 'lineColor': '#8b5cf6'}}}%%
sequenceDiagram
    participant C as Client
    participant M as Master
    participant CS as ChunkServer

    C->>M: GetFileInfo(filename)
    M-->>C: ChunkIDs + Locations
    C->>CS: DownloadChunk(chunkID)
    CS-->>C: ChunkData stream
Loading

Project Structure

dfs/
├── cmd/                    # Entry points
│   ├── master/main.go      # Master server binary
│   ├── chunkserver/main.go # Chunk server binary
│   └── client/main.go      # CLI client binary
│
├── internal/               # Private implementation
│   ├── master/             # Master server logic
│   ├── chunkserver/        # Chunk server logic
│   └── client/             # Client SDK
│       ├── dfsclient.go    # High-level SDK
│       ├── uploader/       # Upload handling
│       ├── downloader/     # Download handling
│       └── masterclient/   # Master communication
│
├── pkg/                    # Shared packages
│   └── logger/             # Structured logging
│
├── dfs/                    # Generated protobuf code
│   ├── masterpb/
│   └── chunkpb/
│
└── proto/                  # Protobuf definitions

CLI Usage

# Upload a file
./bin/dfs-client put <local-file> <remote-name>

# Download a file
./bin/dfs-client get <remote-name> <local-directory>

# Specify custom master address
./bin/dfs-client -master localhost:9000 put file.txt myfile.txt

Chunk Server Flags

./bin/dfs-chunkserver [flags]

Flags:
  -port string    Chunk server address (default ":9001")
  -master string  Master server address (default ":8000")
  -dir string     Storage directory (default "./data")

Configuration

Parameter Default Description
REPLICATION_FACTOR 2 Number of replicas per chunk
CHUNK_SIZE 64 MB Size of each chunk
LIVE_THRESHOLD 30s Server considered dead after this
Heartbeat Interval 5s Chunk server heartbeat frequency

Data Flow

Write Path

  1. Client requests chunk allocation from Master
  2. Master returns ChunkID and replica servers
  3. Client uploads directly to chunk server
  4. Master triggers replication via heartbeat

Read Path

  1. Client requests file info from Master
  2. Master returns chunk locations
  3. Client downloads chunks directly from chunk servers
  4. Client reassembles file from chunks

Heartbeat System

Chunk servers maintain a bidirectional gRPC stream with the master:

ChunkServer → Master:

  • Server address
  • Free storage (MB)
  • List of stored chunks

Master → ChunkServer:

  • Replication tasks
  • Delete tasks

Development

# Build all
go build ./...

# Run tests
go test ./...

# Lint
go vet ./...

License

MIT