lakeFS
FreeGit-for-data on object storage
FreeFree tier
Inputs: file, apiOutputs: file, api
About lakeFS
lakeFS is an open-source data version control platform that applies Git-like operations to object storage, enabling users to manage data lakes with the same principles used for code management. It supports AWS S3, Azure Blob Storage, and Google Cloud Storage as underlying storage, and is API-compatible with S3, allowing seamless integration with modern data frameworks such as Spark, Hive, AWS Athena, DuckDB, and Presto. The tool facilitates repeatable, atomic, and versioned data lake operations for ETL jobs, data science, and analytics. lakeFS is developed by Treeverse and is available under an open-source license, with a sandbox quickstart option via pip or Docker for local experimentation.
Key Features
Git-like version control for data lakes on object storage
Supports AWS S3, Azure Blob Storage, and Google Cloud Storage
S3 API compatibility for integration with data frameworks
Atomic and repeatable data operations (e.g., commits, branches, merges)
Open-source with community contributions on GitHub
Quickstart sandbox via pip or Docker for local testing
Pros & Cons
Pros
- Open-source and free to use, with no licensing costs
- Brings familiar Git-like workflows to data management
- Works with major cloud object storage providers
- S3 API compatibility simplifies integration with existing tools
- Active community and GitHub repository with documentation
Cons
- Requires setup and configuration, not a fully managed service
- Learning curve for users unfamiliar with Git concepts applied to data
- Performance may depend on underlying object storage and network
- Free tier is open-source; enterprise support or hosted options may have separate pricing
Best For
Versioning and managing data lakes for analytics and data scienceEnabling reproducible ETL pipelines with atomic commitsCollaborating on data projects with branching and merging workflowsIntegrating with Spark, Hive, Athena, DuckDB, or Presto for data processingTesting data changes in isolated branches before production deployment
FAQ
What is lakeFS?
lakeFS is an open-source data version control tool that brings Git-like operations to object storage, allowing users to manage data lakes with branches, commits, and merges.
Which storage backends does lakeFS support?
Based on available information, lakeFS supports AWS S3, Azure Blob Storage, and Google Cloud Storage.
Is lakeFS free to use?
lakeFS is open-source and appears to be free to use. Users should verify licensing details and any enterprise options on the project's official site.
How can I get started with lakeFS?
You can start with a sandbox instance using pip install lakefs and running python -m lakefs.quickstart, or via Docker with docker run. Detailed instructions are in the documentation.
Does lakeFS integrate with data processing frameworks?
Yes, lakeFS is S3 API-compatible and works with frameworks like Spark, Hive, AWS Athena, DuckDB, and Presto, as indicated in the project's README.
Where can I find the lakeFS source code?
The source code is hosted on GitHub at https://github.com/treeverse/lakeFS.