Data Ingestion with Python Cookbook: A practical guide to ingesting, monitoring,

The Nile on eBay   Data Ingestion with Python Cookbook by Gláucia Esppenchutz

Deploy your data ingestion pipeline, orchestrate, and monitor efficiently to prevent loss of data and qualityKey FeaturesHarness best practices to create a Python and PySpark data ingestion pipelineSeamlessly automate and orchestrate your data pipelines using Apache AirflowBuild a monitoring framework by integrating the concept of data observability into your pipelinesBook DescriptionData Ingestion with Python Cookbook offers a practical approach to designing and implementing data ingestion pipelines. It presents real-world examples with the most widely recognized open source tools on the market to answer commonly asked questions and overcome challenges.You'll be introduced to designing and working with or without data schemas, as well as creating monitored pipelines with Airflow and data observability principles, all while following industry best practices. The book also addresses challenges associated with reading different data sources and data formats. As you progress through the book, you'll gain a broader understanding of error logging best practices, troubleshooting techniques, data orchestration, monitoring, and storing logs for further consultation.By the end of the book, you'll have a fully automated set that enables you to start ingesting and monitoring your data pipeline effortlessly, facilitating seamless integration with subsequent stages of the ETL process.What you will learnImplement data observability using monitoring toolsAutomate your data ingestion pipelineRead analytical and partitioned data, whether schema or non-schema basedDebug and prevent data loss through efficient data monitoring and loggingEstablish data access policies using a data governance frameworkConstruct a data orchestration framework to improve data qualityWho this book is forThis book is for data engineers and data enthusiasts seeking a comprehensive understanding of the data ingestion process using popular tools in the open source community. For more advanced learners, this book takes on the theoretical pillars of data governance while providing practical examples of real-world scenarios commonly encountered by data engineers.

FORMATPaperback CONDITIONBrand New Author Biography

Gláucia Esppenchutz is a data engineer with expertise in managing data pipelines and vast amounts of data using cloud and on-premises technologies. She worked in companies such as Globo, BMW Group, and Cloudera. Currently, she works at AiFi, specializing in the field of data operations for autonomous systems.She comes from the biomedical field and shifted her career ten years ago to chase the dream of working closely with technology and data. She is in constant contact with the open source community, mentoring people and helping to manage projects, and has collaborated with the Apache, PyLadies group, FreeCodeCamp, Udacity, and MentorColor communities.

Table of Contents

Table of ContentsIntroduction to Data IngestionPrincipals of Data Access – Accessing your DataData Discovery – Understanding Our Data Before Ingesting ItReading CSV and JSON Files and Solving ProblemsIngesting Data from Structured and Unstructured DatabasesUsing PySpark with Dened and Non-Dened SchemasIngesting Analytical DataDesigning Monitored Data WorkflowsPutting Everything Together with AirowLogging and Monitoring Your Data Ingest in AirflowAutomating Your Data Ingestion PipelinesUsing Data Observability for Debugging, Error Handling, and Preventing Downtime

Details ISBN183763260X Pages 414 Publisher Packt Publishing Limited Year 2023 ISBN-13 9781837632602 Format Paperback Publication Date 2023-05-31 Imprint Packt Publishing Limited Subtitle A practical guide to ingesting, monitoring, and identifying errors in the data ingestion process Place of Publication Birmingham Country of Publication United Kingdom AU Release Date 2023-05-31 NZ Release Date 2023-05-31 UK Release Date 2023-05-31 Author Gláucia Esppenchutz DEWEY 005.74 Audience General We've got this

At The Nile, if you're looking for it, we've got it.With fast shipping, low prices, friendly service and well over a million items - you're bound to find what you want, at a price you'll love!

TheNile_Item_ID:159459615;