Context
Since July 2025, I have worked through Apside within CNAV’s Data Factory on data processing, integration and migration.
- Business data from multiple sources
- Data to pseudonymise and reconcile
- Public data collected automatically
Challenge
Migrating from Cloudera to Teradata requires adapting processes to the target environment while implementing business rules.
Objectives
- Develop PySpark processing on Cloudera.
- Adapt processing for Teradata with SQL and Shell.
- Integrate data from multiple sources and develop reconciliation processes with Python.
Contribution
I translate business rules into integration and migration processes. My work also includes a pseudonymisation and reconciliation process, and a tool for automated public data collection.
- Developing and updating PySpark processing on Cloudera.
- Translating business rules and adapting processing for Teradata with SQL and Shell.
- Developing a Python process for data pseudonymisation and reconciliation.
Work carried out
The assignment is ongoing. Work to date covers PySpark updates, adaptation for Teradata and Python tools for data pseudonymisation, reconciliation and collection.
Technologies
- Python
- PySpark
- SQL
- Shell
- Cloudera
- Teradata
- Jenkins
- GitLab
A similar challenge?
Facing a similar challenge? Let’s talk.
Describe your data, processes and objectives to discuss an engagement suited to your needs.