msaFilesystem - Agnostic Abstract Filesystem API which allows to use S3, GCS, Azure Datalake, your local FS, Youtube etc Optimized for use with FastAPI/Pydantic.

Overview

msaFilesystem Logo


msaFilesystem - Agnostic Abstract Filesystem API which allows to use S3, GCS, Azure Datalake, your local FS, Youtube etc
Optimized for use with FastAPI/Pydantic.
Package version Supported Python versions


Alpha/EarlyStage !!!

Documentation: msaFilesystem Documentation (https://msaFilesystem.u2d.ai/)


Features

  • Abstract Filesystem: Agnostic Abstract Filesystem API which allows to use S3, GCS, Azure Datalake, your local FS, Youtube etc. Based on: PyFilesystem2
  • App Filesystems: Manage filesystems in platform-specific application directories. These classes abstract away the different requirements for user data across platforms.
  • FTP Filesystem: A FTP (File Transport Protocol) Filesystem. Optionally, the connection can be made securely via TLS. This is known as FTPS, or FTP Secure.
  • Memory Filesystem: A filesystem that stored in memory. Memory filesystems are useful for caches, temporary data stores, unit testing, etc.
  • Mount Filesystem: A Mount FS is a virtual filesystem which can seamlessly map sub-directories on to other filesystems.
  • Multi Filesystem: A MultiFS is a filesystem composed of a sequence of other filesystems, where the directory structure of each overlays the previous filesystem in the sequence.
  • OS Filesystem: Manage the filesystem provided by your OS. In essence, an OSFS is a thin layer over the io and os modules of the Python standard library.
  • Sub Filesystem: Manage a directory in a parent filesystem. A SubFS is a filesystem object that maps to a sub-directory of another filesystem.
  • Tar Filesystem: Read and write tar files.
  • Temporary Filesystem: Manage filesystems in temporary locations. A temporary filesytem is stored in a location defined by your OS (/tmp on linux). The contents are deleted when the filesystem is closed.
  • Zip Filesystem: Read and write zip files.
  • SMB Filesystem: A filesystem over SMB.
  • WebDAV Filesystem: A filesystem for WebDAV.
  • Azure Datalake & S3FS Filesystem: A filesystem for Azure Datalake storage & S3FS.
  • Google Cloud Storage (GCS) Filesystem: A filesystem for Google Cloud Storage (GCS). With GCSFS, you can interact with Google Cloud Storage as if it was a regular filesystem.
  • Google Drive Filesystem: A filesystem for Google Drive. Interact with Google Drive as if it was a regular filesystem.
  • Dropbox Filesystem: A filesystem for Dropbox.
  • OneDrive Filesystem: A filesystem for Dropbox.
  • YouTube Videos and Playlists Filesystem: A filesystem for YouTube Videos and Playlists.
  • External Filesystems: See the following wiki page for a list of filesystems not in the core library, and community contributed filesystems.

Main Dependencies

  • fs~=2.4.16: Module that provides a common interface to any filesystem
  • six~=1.16.0: Python 2 and 3 compatibility utilities
  • pysmb~=1.2.8: SMB/CIFS library
  • fs.webdavfs~=0.4.2: WebDAV support
  • fs-dlk~=0.1.3: Azure Datalake support
  • fs-s3fs~=1.1.1: Amazon S3 filesystem support
  • fs-gcsfs~=1.5.1: Google Cloud Storage (GCS) support
  • fs.googledrivefs~=2.3.0: Google Drive support
  • fs.dropboxfs~=0.2.2.post2: Dropbox support
  • fs.onedrivefs~=1.1.1: OneDrive support
  • fs.youtube~=0.3.1: Youtube support
  • fs.smbfs~=1.0.5: SMB support

License Agreement

  • msaFilesystem is based on MIT open source and free to use, it is free for commercial use, but please show/list the copyright information about msaFilesystem somewhere.

How to create the documentation

We use mkdocs and mkdocsstring. The code reference and nav entry get's created virtually by the triggered python script /docs/gen_ref_pages.py while mkdocs serve or build is executed.

Requirements Install for the PDF creation option:

PDF Export is using mainly weasyprint, if you get some errors here pls. check there documentation. Installation is part of the msaFilesystem, so this should be fine.

We can now test and view our documentation using:

mkdocs serve

Build static Site:

mkdocs build

Build and Publish

Build:

python setup.py sdist

Publish to pypi:

twine upload dist/*
You might also like...

API development and Heroku deploy using FastAPI framework and data extraction from SQLite local hosted database.

fastapi-sqlite-heroku API development and Heroku deploy using FastAPI framework and data extraction from SQLite local hosted database. General Steps E

Oct 13, 2022

Speak devs! The Turing Machine is an abstract model of a computer, created by the so-called father of computing Alan Turing

Speak devs! The Turing Machine is an abstract model of a computer, created by the so-called father of computing Alan Turing

Speak devs! The Turing Machine is an abstract model of a computer, created by the so-called father of computing Alan Turing. The Turing machine is restricted only to the logical aspects of its operation (memory, states and transitions), and not its physical implementation. In a Turing machine, any digital computer can be modeled, from the old ones to the most current ones. So let's see this in practice, or better in code with a turing complete language!

May 15, 2022

Multi-label Classification Model for English Scientific Literature: develop a multi-label classification model for scientific research literature based on the given metadata (title and abstract) of scientific research literature and corresponding hierarchical labels from a domain-specific topic taxonomy.

Multi-label Classification Model for English Scientific Literature: develop a multi-label classification model for scientific research literature based on the given metadata (title and abstract) of scientific research literature and corresponding hierarchical labels from a domain-specific topic taxonomy.

Jul 14, 2022

Instructions With Holes: A library for matching instructions to more abstract schemes that can be instantiated with other operands.

IWHO - Instructions With HOles IWHO is a python library for representing machine instructions. Its focus is on enabling easy instantiations of instruc

Sep 13, 2022

Grep Python Abstract Syntax Trees (AST) using XPath

pyastgrep A command-line utility for grepping Python files using XPath syntax against the Python AST. TODO Change examples to match current output Cle

Nov 28, 2022

A utility that allows you to use DI in fastapi without Depends()

fastapi-better-di What is this ? fastapi-better-di is a utility that allows you to use DI in fastapi without Depends() Installation pip install fastap

May 24, 2022

Library to implement various quantum tools like state discrimination, non local games etc

qtools Library to implement various quantum tools like state discrimination, non local games etc My first addition is NPA hierarchy unlike available l

Jun 29, 2022

A tiny discord api for language agnostic testing.

pocketcord A tiny discord api for language agnostic testing. Pocketcord essentially is a discord api that is meant to run fully offline and can send f

Oct 16, 2022

TinyFlux is the tiny time series database optimized for your happiness :)

TinyFlux is the tiny time series database optimized for your happiness :)

Quick Links Example Code Snippets Full Example Notebooks and Scrips Documentation Changelog Contributing Installation $ pip install tinyflux Or, downl

Nov 15, 2022
Releases(0.0.2)
Owner
Stefan Welcker
Stefan Welcker
msaServer - Helper & Wrapper around Uvicorn/Gunicorn for FastAPI based apps Optimized for use with FastAPI/Pydantic.

msaServer - Helper & Wrapper around Uvicorn/Gunicorn for FastAPI based apps Optimized for use with FastAPI/Pydantic. Documentation: msaServer Document

Stefan Welcker 3 Nov 1, 2022
Fluxos de captura e subida de dados no datalake da Prefeitura do Rio de Janeiro.

Pipelines Este repositório contém fluxos de captura e subida de dados no datalake da Prefeitura do Rio de Janeiro. O repositório é gerido pelo Escritó

Prefeitura do Rio de Janeiro 18 Nov 3, 2022
My first attempt at a rough ETL pipeline; technologies include spark, GCS, prefect orchestration, and terraform

Data Engineering Project #1 : Fitpipe, a hourly max heart rate ETL pipeline My first attempt at a rough ETL pipeline; technologies include spark, GCS,

null 9 Nov 17, 2022
🪛 A simple pydantic to Form FastAPI model converter.

pyfa-converter Makes it pretty easy to create a model based on Field [pydantic] and use the model for www-form-data. How to simplify your life? What d

null 17 Nov 17, 2022
FastAPI-User-Auth is a simple and powerful FastAPI user RBAC authentication and authorization library. Based on FastAPI-Amis-Admin and provides a freely extensible visual management interface.

项目介绍 FastAPI-User-Auth FastAPI-User-Auth是一个简单而强大的FastAPI用户RBAC认证与授权库. 基于FastAPI-Amis-Admin并提供可自由拓展的可视化管理界面. 源码 · 在线演示 · 文档 · 文档打不开? FastAPI-User-Auth是

AmisAdmin 151 Nov 23, 2022
Python utility to generate filesystem content for Obsidian.

Security Vault Generator Quickly parse, format, and output common frameworks/content for Obsidian.md. There is a strong focus on MITRE ATT&CK because

Justin Angel 72 Nov 11, 2022
Use pydantic schemas to define your query parameters, with extra sugar in them ✨

FastAPI Query Parameters Empower your query parameters setup on FastAPI through Pydantic Installation poetry add fastapi-qp Usage You can just define

Clicampo 36 Nov 21, 2022
The project will help you scrape image from a website and will send it directly (with a local path) to a Azure Blob Storage

Python Image Scrapper to Azure Blob Storage (with local path) The project will help you scrape image from a website and will send it directly (with a

Antoine Smet 5 Nov 7, 2022
A password manager that works in harmony with the browser, allows you to create strong passwords, encrypt your passwords and store them in your own local or cloud storage.

Password Manager A password manager that works in harmony with the browser, allows you to create strong passwords, encrypt your passwords and store th

Metin Ilgar 1 May 23, 2022
🚀This is a repository that was created to play and enjoy testing different things with localstack, which is a local AWS deployment that allows to interact with multiples AWS services locally. ☁☁

CLOUD STACK DEPLOY This repository has some projects using AWS services like Glue, Lambda and many others. This repository uses a simple localstack in

Elkin Javier Guerra Galeano 2 Sep 26, 2022