how to update mass data using Python Pandas
Part 14/20
Effortlessly Updating Mass Columns in Multiple Files
with Custom Functions using Pandas, PyArrow and Polars
---
Notebook link
https://github.com/AmitXShukla/RPA/bl...
YouTube Playlist
• Python Automation Scripts
GitHub Repo
https://github.com/AmitXShukla/RPA
This tutorial will guide you through the process of writing code to update mass columns across CSV or data files.
While this initial notebook focuses on simple calculations,
the Automation series will gradually introduce more complex RPA projects that rely on similar code snippets for mass updates.
By learning to code a basic calculator, you'll have a foundation for more advanced techniques.
Throughout the notebook, you'll define functions for simple arithmetic calculations and algebraic operations passed as parameters.
These concepts will later be applied to perform mass calculations across multiple files and directories.
Keep this notebook handy as a reference.
define functions for simple arithmetic calculations
define functions to pass arithmetic/algebraic operations as parameters
performing mass calculations across files and directories
Imagine yourself as an employee of a bank, tasked with the responsibility of recalculating or verifying the calculations in the aforementioned spreadsheet.
Although it may appear straightforward initially, the task becomes incredibly challenging when you consider that you need to carry it out for
a total of 350 banks
each with a minimum of 1+ million daily transactions
over a period of 365 days
and for the past 5 years
Here are few options
using pandas to update columns with calculation using functions
using pyarrow to update columns with calculation using functions
using polars to update columns with calculation using functions
using polars and transform columns inline (with out calling functions)
mass calculations
Step 1: OOPs - classes, data structures, custom transformation methods
Step 2: data transformation using Pandas
Step 3: data transformation using PyArrow
Step 4: data transformation using Polars