Migrating 500GB from OneDrive to an External Drive on macOS

OneDrive to Seagate Migration: Complete Technical Guide

This document outlines the exact architecture, scripts, and sequence of operations used to safely force-download and migrate a 500GB+ OneDrive repository to an external Seagate drive, overcoming macOS FileProvider limitations and Microsoft API throttling.

---

The Pre-Flight Setup & Permissions

Before running any scripts, the environment must be prepared to prevent Operation not permitted errors.

• External Drive: Mounted using Mounty to ensure write access. Path: /Users/michael/.mounty/Seagate Backup Plus Drive
• Permissions: Full Disk Access granted to /bin/bash in System Settings > Privacy & Security so background scripts can read inside OneDrive folders.
• Power Settings: disksleep set to 0 (sudo pmset -c disksleep 0) to prevent the external drive from spinning down during the massive transfer.

---

Phase 1: The Great Purge (Library Cleanup)

To prevent the backup from timing out on hundreds of thousands of tiny library files, we purged them directly from the cloud.

• Script: ~/Library/CloudStorage/OneDrive-Personal/cleanup_node_modules_venv.sh
• Action 1 (node_modules): Used fd with 24 parallel threads to instantly delete all node_modules folders. (Status: COMPLETED 100%)
• Action 2 (venvs): Attempted to run pip freeze to save a requirements_auto_backup.txt before deleting Python .venv folders.
• Venv Status Update: Most succeeded, but 13 broken venvs remain. These 13 folders were so corrupted or “dataless” that pip freeze crashed (Abort trap: 6). Per our strict “Safe-by-Default” rules, the script refused to delete them. They were left in place to be slowly dragged across by the main backup.

---

Phase 2: The Core Backup Architecture

The migration uses a dual-script system running simultaneously in the background via macOS launchd.

The “Bulldozer” (Main Backup Script)

• Script: full_onedrive_backup.sh (Version 5)
• Function: Iterates through a static queue (.backup_queue), processing one top-level folder at a time. It uses cp to force macOS to download the dataless file, then copies it to the Seagate.
• Features: Runs 5 parallel threads. Includes a “Size-Aware Redundancy Check” (skips files if they already exist on Seagate AND are > 0 bytes).
• Current Status: AUTO-PAUSED at Item 11 (Documents/2022). It successfully completed all Priority Items (1–10: Journaling, Ideas, Todo, and all Project folders). It is waiting for the Sweeper to finish before tackling the bulk archives.

The “Sweeper” (Quarantine Catch-Up Script)

• Script: onedrive_quarantine_retry.sh (Version 2)
• Function: Whenever the Bulldozer hits a timeout or “Resource deadlock avoided” error, it logs the file path to .quarantine_queue. The Sweeper runs continuously in the background, making 5 parallel retry attempts with extended 10-minute timeouts to rescue these stubborn files.
• Current Status: ACTIVELY RUNNING. It is over 90% through its current cycle, with roughly ~174,000 files remaining in the Priority Queue. (A secondary Bulk Queue of ~274,000 non-priority files is waiting on standby).

The “Orchestrator” (Launchd Wrapper)

• Script: backup_wrapper.sh
• LaunchAgent: ~/Library/LaunchAgents/com.user.onedrivebackup.plist
• Function: Triggered every 7 minutes by launchd. It checks if OneDrive is open (and automatically opens it if it’s 1–6 AM). If everything is healthy, it triggers the Bulldozer. It contains the logic that auto-pauses the Bulldozer at Item 11 if the Sweeper still has priority files to finish.

---

User Controls (Desktop Scripts)

To safely interrupt this massive background operation without corrupting data:

• ~/Desktop/Pause_Backup.command: Instantly kills the wrapper, bulldozer, and sweeper processes. Allows the user to safely quit the OneDrive app to free up system resources.
• ~/Desktop/Resume_Backup.command: Reopens the OneDrive app, restarts the Sweeper in the background, and allows the 7-minute launchd cycle to resume the Bulldozer naturally.

---

Phase 3: The Final Verification

Once the Bulldozer finishes Item 27 and the Quarantine Queue hits 0, the migration is complete. We will then run a mathematical audit.

• Script: verify_backup.sh
• Tool: rclone (API connection: onedrive_remote)
• Function: Compares the cloud hashes of the files on Microsoft’s servers directly against the physical files on the Seagate drive. This proves byte-for-byte perfection without needing to download the 500GB of data back to the Mac’s internal SSD.

---