Skip to content

Built-in Components ​

Firehose ships with reusable building blocks for common event processing patterns.

Conditions (condition) ​

Expression-Based Filtering ​

go
import "github.com/emad-elsaid/firehose/condition"

// Simple comparison
Where: condition.Cond[OrderEvent]("amount > 1000")

// Boolean logic
Where: condition.Cond[OrderEvent]("premium = true and amount > 500")

// String operations
Where: condition.Cond[OrderEvent](`country = "US" or country = "CA"`)

Function Adapter ​

go
Where: condition.Func[OrderEvent](func(ctx context.Context, evt OrderEvent, syms boolexpr.Symbols) (bool, error) {
    return evt.Amount > 1000, nil
})

Rate Limiting ​

go
Where: &condition.RateLimit[OrderEvent]{
    Limit: rate.Every(time.Second),
    Burst: 10,
}

Deduplication ​

go
Where: &condition.Once[OrderEvent]{
    Duration: 5 * time.Minute,
    Cache:    cache.NewMemory[string](10*time.Minute, time.Minute),
}

Struct Validation ​

go
Where: &condition.Valid[OrderEvent]{}

Validates event fields using go-playground/validator tags. Returns an error if validation fails.

Multiple Conditions ​

go
Where: condition.Conditions[OrderEvent]{
    condition.Cond[OrderEvent]("amount > 100"),
    &condition.RateLimit[OrderEvent]{Limit: rate.Every(time.Second), Burst: 5},
    condition.Func[OrderEvent](customCheck),
}

Actions (actions) ​

Function Adapter ​

go
import "github.com/emad-elsaid/firehose/actions"

Select: actions.Func[HTTPRequest, User](func(
    ctx context.Context,
    req HTTPRequest,
    syms boolexpr.Symbols,
) (User, error) {
    user := extractUserFromRequest(req)
    return user, nil
})

Caching ​

go
Select: &actions.Cache[OrderEvent, ProcessedOrder]{
    Action: ProcessOrder{},
    Cache:  cache.NewMemory[ProcessedOrder](10*time.Minute, time.Minute),
    TTL:    5 * time.Minute,
}

Uses EventID to generate cache keys. Requires a CacheStorage backend.

Action Composition ​

Chain multiple transformations:

go
// Two actions: I → M → O
Select: actions.Chain[HTTPRequest, ParsedRequest, User]{
    First:  ParseRequest{},
    Second: ExtractUser{},
}

// Three actions: I → A → B → O
Select: actions.Chain3[HTTPRequest, ParsedRequest, ValidatedRequest, User]{
    First:  ParseRequest{},
    Second: ValidateRequest{},
    Third:  ExtractUser{},
}

// Chain4 and Chain5 also available

Load Balancing ​

Round-robin distribution:

go
Select: &actions.RoundRobin[OrderEvent, ProcessedOrder]{
    Actions: []fh.Action[OrderEvent, ProcessedOrder]{
        ProcessWithServiceA{},
        ProcessWithServiceB{},
        ProcessWithServiceC{},
    },
}

Random distribution:

go
Select: &actions.Random[OrderEvent, ProcessedOrder]{
    Actions: []fh.Action[OrderEvent, ProcessedOrder]{
        ProcessWithServiceA{},
        ProcessWithServiceB{},
    },
}

Destinations (destinations) ​

Function Adapter ​

go
import "github.com/emad-elsaid/firehose/destinations"

Into: destinations.Func[User](func(ctx context.Context, user User) error {
    err := saveToDatabase(user)
    return err
})

Accumulator ​

Collect events in memory (useful for testing):

go
accumulator := &destinations.Accumulator[User]{}

Into: accumulator

// Later, retrieve collected items
users := accumulator.Items()

Fanout ​

Send to all destinations:

go
Into: destinations.Fanout[User]{
    Destinations: []fh.Destination[User]{
        UserDatabase{},
        EmailService{},
        AnalyticsService{},
    },
}

Load Balancing ​

Round-robin:

go
Into: &destinations.RoundRobin[User]{
    Destinations: []fh.Destination[User]{
        DatabaseShard1{},
        DatabaseShard2{},
        DatabaseShard3{},
    },
}

Random:

go
Into: &destinations.Random[User]{
    Destinations: []fh.Destination[User]{
        Server1{},
        Server2{},
    },
}

Channel Adapters ​

go
// Consume from channel, forward each item
Into: destinations.FromChan[User]{
    Into: UserDatabase{},
}

// Wrap event in single-item channel
Into: destinations.ToChan[User]{
    Into: ChannelConsumer{},
}

Slice Adapters ​

go
// Consume from slice, forward each item
Into: destinations.FromSlice[User]{
    Into: UserDatabase{},
}

// Wrap event in single-item slice
Into: destinations.ToSlice[User]{
    Into: BatchProcessor{},
}

Sources (sources) ​

Function Adapter ​

go
import "github.com/emad-elsaid/firehose/sources"

From: sources.Func[HTTPRequest](func(ctx context.Context, cb fh.Callback[HTTPRequest]) (<-chan struct{}, error) {
    server := &http.Server{Addr: ":8080"}

    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        event := HTTPRequest{Method: r.Method, Path: r.URL.Path}
        cb(r.Context(), event, func(err error) {
            // Handle error
        })
    })

    go server.ListenAndServe()
    return ctx.Done(), nil
})

Manual Source ​

Emit events manually (useful for testing):

go
manual := &sources.Manual[OrderEvent]{}

From: manual

// Later, emit events
manual.Emit(ctx, OrderEvent{OrderID: "123", Amount: 99.99})

Cache Storage (cache) ​

In-memory cache backend:

go
import "github.com/emad-elsaid/firehose/cache"

cache := cache.NewMemory[ProcessedOrder](
    10*time.Minute,  // default TTL
    time.Minute,     // cleanup interval
)

// Use with actions.Cache or condition.Once
Select: &actions.Cache[OrderEvent, ProcessedOrder]{
    Action: ProcessOrder{},
    Cache:  cache,
    TTL:    5 * time.Minute,
}

Implements the CacheStorage interface with Get, Set, and GetOrSet methods.

Middlewares (middlewares) ​

Panic Recovery ​

go
import "github.com/emad-elsaid/firehose/middlewares"

Middlewares: []fh.Middleware[I, O]{
    &middlewares.Panic[I, O]{},
}

Structured Logging ​

go
Middlewares: []fh.Middleware[I, O]{
    &middlewares.Slog[I, O]{},
}

Parallel Execution ​

Run same-source rules in parallel:

go
import "github.com/emad-elsaid/firehose/runner"

Middlewares: []fh.Middleware[I, O]{
    &middlewares.Parallel[I, O]{
        Runner: runner.Basic{},
    },
}

Next Steps ​

Released under the MIT License.