All projects

Migrating processing from Cloudera to Teradata

Adapting PySpark processing for Teradata and integrating data from multiple sources.

Professional experienceData integration and migration

An assignment from my professional background. No internal or confidential data is disclosed.

Abstract illustration of the main data flow — no internal or confidential data is shown.

Context

Since July 2025, I have worked through Apside within CNAV’s Data Factory on data processing, integration and migration.

  • Business data from multiple sources
  • Data to pseudonymise and reconcile
  • Public data collected automatically

Challenge

Migrating from Cloudera to Teradata requires adapting processes to the target environment while implementing business rules.

Objectives

  • Develop PySpark processing on Cloudera.
  • Adapt processing for Teradata with SQL and Shell.
  • Integrate data from multiple sources and develop reconciliation processes with Python.

Contribution

I translate business rules into integration and migration processes. My work also includes a pseudonymisation and reconciliation process, and a tool for automated public data collection.

  • Developing and updating PySpark processing on Cloudera.
  • Translating business rules and adapting processing for Teradata with SQL and Shell.
  • Developing a Python process for data pseudonymisation and reconciliation.

Work carried out

The assignment is ongoing. Work to date covers PySpark updates, adaptation for Teradata and Python tools for data pseudonymisation, reconciliation and collection.

Technologies

  • Python
  • PySpark
  • SQL
  • Shell
  • Cloudera
  • Teradata
  • Jenkins
  • GitLab

A similar challenge?

Facing a similar challenge? Let’s talk.

Describe your data, processes and objectives to discuss an engagement suited to your needs.

Discuss your project