Built-in Components
Firehose ships with reusable building blocks for common event processing patterns.
Conditions (condition)
Expression-Based Filtering
go
import "github.com/emad-elsaid/firehose/condition"
// Simple comparison
Where: condition.Cond[OrderEvent]("amount > 1000")
// Boolean logic
Where: condition.Cond[OrderEvent]("premium = true and amount > 500")
// String operations
Where: condition.Cond[OrderEvent](`country = "US" or country = "CA"`)Function Adapter
go
Where: condition.Func[OrderEvent](func(ctx context.Context, evt OrderEvent, syms boolexpr.Symbols) (bool, error) {
return evt.Amount > 1000, nil
})Rate Limiting
go
Where: &condition.RateLimit[OrderEvent]{
Limit: rate.Every(time.Second),
Burst: 10,
}Deduplication
go
Where: &condition.Once[OrderEvent]{
Duration: 5 * time.Minute,
Cache: cache.NewMemory[string](10*time.Minute, time.Minute),
}Struct Validation
go
Where: &condition.Valid[OrderEvent]{}Validates event fields using go-playground/validator tags. Returns an error if validation fails.
Multiple Conditions
go
Where: condition.Conditions[OrderEvent]{
condition.Cond[OrderEvent]("amount > 100"),
&condition.RateLimit[OrderEvent]{Limit: rate.Every(time.Second), Burst: 5},
condition.Func[OrderEvent](customCheck),
}Actions (actions)
Function Adapter
go
import "github.com/emad-elsaid/firehose/actions"
Select: actions.Func[HTTPRequest, User](func(
ctx context.Context,
req HTTPRequest,
syms boolexpr.Symbols,
) (User, error) {
user := extractUserFromRequest(req)
return user, nil
})Caching
go
Select: &actions.Cache[OrderEvent, ProcessedOrder]{
Action: ProcessOrder{},
Cache: cache.NewMemory[ProcessedOrder](10*time.Minute, time.Minute),
TTL: 5 * time.Minute,
}Uses EventID to generate cache keys. Requires a CacheStorage backend.
Action Composition
Chain multiple transformations:
go
// Two actions: I → M → O
Select: actions.Chain[HTTPRequest, ParsedRequest, User]{
First: ParseRequest{},
Second: ExtractUser{},
}
// Three actions: I → A → B → O
Select: actions.Chain3[HTTPRequest, ParsedRequest, ValidatedRequest, User]{
First: ParseRequest{},
Second: ValidateRequest{},
Third: ExtractUser{},
}
// Chain4 and Chain5 also availableLoad Balancing
Round-robin distribution:
go
Select: &actions.RoundRobin[OrderEvent, ProcessedOrder]{
Actions: []fh.Action[OrderEvent, ProcessedOrder]{
ProcessWithServiceA{},
ProcessWithServiceB{},
ProcessWithServiceC{},
},
}Random distribution:
go
Select: &actions.Random[OrderEvent, ProcessedOrder]{
Actions: []fh.Action[OrderEvent, ProcessedOrder]{
ProcessWithServiceA{},
ProcessWithServiceB{},
},
}Destinations (destinations)
Function Adapter
go
import "github.com/emad-elsaid/firehose/destinations"
Into: destinations.Func[User](func(ctx context.Context, user User) error {
err := saveToDatabase(user)
return err
})Accumulator
Collect events in memory (useful for testing):
go
accumulator := &destinations.Accumulator[User]{}
Into: accumulator
// Later, retrieve collected items
users := accumulator.Items()Fanout
Send to all destinations:
go
Into: destinations.Fanout[User]{
Destinations: []fh.Destination[User]{
UserDatabase{},
EmailService{},
AnalyticsService{},
},
}Load Balancing
Round-robin:
go
Into: &destinations.RoundRobin[User]{
Destinations: []fh.Destination[User]{
DatabaseShard1{},
DatabaseShard2{},
DatabaseShard3{},
},
}Random:
go
Into: &destinations.Random[User]{
Destinations: []fh.Destination[User]{
Server1{},
Server2{},
},
}Channel Adapters
go
// Consume from channel, forward each item
Into: destinations.FromChan[User]{
Into: UserDatabase{},
}
// Wrap event in single-item channel
Into: destinations.ToChan[User]{
Into: ChannelConsumer{},
}Slice Adapters
go
// Consume from slice, forward each item
Into: destinations.FromSlice[User]{
Into: UserDatabase{},
}
// Wrap event in single-item slice
Into: destinations.ToSlice[User]{
Into: BatchProcessor{},
}Sources (sources)
Function Adapter
go
import "github.com/emad-elsaid/firehose/sources"
From: sources.Func[HTTPRequest](func(ctx context.Context, cb fh.Callback[HTTPRequest]) (<-chan struct{}, error) {
server := &http.Server{Addr: ":8080"}
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
event := HTTPRequest{Method: r.Method, Path: r.URL.Path}
cb(r.Context(), event, func(err error) {
// Handle error
})
})
go server.ListenAndServe()
return ctx.Done(), nil
})Manual Source
Emit events manually (useful for testing):
go
manual := &sources.Manual[OrderEvent]{}
From: manual
// Later, emit events
manual.Emit(ctx, OrderEvent{OrderID: "123", Amount: 99.99})Cache Storage (cache)
In-memory cache backend:
go
import "github.com/emad-elsaid/firehose/cache"
cache := cache.NewMemory[ProcessedOrder](
10*time.Minute, // default TTL
time.Minute, // cleanup interval
)
// Use with actions.Cache or condition.Once
Select: &actions.Cache[OrderEvent, ProcessedOrder]{
Action: ProcessOrder{},
Cache: cache,
TTL: 5 * time.Minute,
}Implements the CacheStorage interface with Get, Set, and GetOrSet methods.
Middlewares (middlewares)
Panic Recovery
go
import "github.com/emad-elsaid/firehose/middlewares"
Middlewares: []fh.Middleware[I, O]{
&middlewares.Panic[I, O]{},
}Structured Logging
go
Middlewares: []fh.Middleware[I, O]{
&middlewares.Slog[I, O]{},
}Parallel Execution
Run same-source rules in parallel:
go
import "github.com/emad-elsaid/firehose/runner"
Middlewares: []fh.Middleware[I, O]{
&middlewares.Parallel[I, O]{
Runner: runner.Basic{},
},
}Next Steps
- Learn about Middleware
- See Real-World Examples
- Read the API Reference