I recently tracked down a Python packaging issue on macOS that may also explain several development problems I had written off in the past as mistakes in my own setup.
The surprising part was that the problem was not really Python, pip, the virtual environment, or my pyproject.toml.
It was iCloud Drive.
My development projects had been living under ~/Documents. Because macOS was synchronizing Desktop and Documents through iCloud Drive, that also meant my Git repositories, source files, package metadata and Python virtual environments were sitting inside an iCloud managed directory.
That sounds harmless enough. Source code is mostly text, after all. In practice, though, an active development environment expects filesystem behavior that is much more predictable than a document synchronization service necessarily provides.
When pip install -e succeeds but the package still does not work
The issue appeared while using an editable Python installation:
pip install -e .
Or, with development dependencies:
pip install -e '.[dev]'
pip completed the installation normally. There was no obvious error suggesting that anything had gone wrong.
And yet Python could not reliably discover the package.
That makes this kind of failure particularly irritating. The virtual environment exists, package metadata is generated, the installer reports success, but imports or command line entry points still do not behave as expected.
The clue was the macOS UF_HIDDEN flag
While inspecting the virtual environment, I found that macOS had applied the UF_HIDDEN filesystem flag to files involved in the editable installation.
Some of those files were part of the mechanism Python uses to locate an editable package.
You can inspect macOS filesystem flags with:
ls -lO
Removing the flag manually fixed the issue:
chflags nohidden <file>
But the fix was temporary.
The flag returned.
At that point, the interesting question was no longer “what is wrong with my Python package?” but “what is changing these files underneath Python?”
Why iCloud Drive is different from a plain local directory
It is easy to think of iCloud Drive as a normal folder with a background process that copies files to the cloud.
On current versions of macOS, there is more going on than that.
iCloud Drive is integrated with Apple’s File Provider infrastructure and can manage synchronization state, file availability, metadata and other filesystem behavior behind the scenes.
For normal user documents, that is exactly what you want. PDFs, presentations, photos and office files usually benefit from this abstraction.
Development tools are a different case.
A typical Python project may contain:
- a
.gitrepository - a
.venvPython virtual environment .pthfiles- editable installation metadata
pyproject.toml- build directories
- caches
- lock files
- symlinks
- large numbers of small dependency files
Python packaging tools expect these files to behave like ordinary local filesystem objects.
If iCloud Drive is also managing metadata and file state in the same directory tree, the result can be surprisingly difficult to diagnose.
The fix was simply to move the repository
Rather than keep fighting macOS with filesystem workarounds, I moved the project out of my iCloud synchronized Documents directory.
Instead of keeping it here:
~/Documents/Projects/my-python-project
I moved it here:
~/dev/my-python-project
Then I recreated the virtual environment and installed the project again:
cd ~/dev/my-python-project
python -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'
It worked immediately.
No chflags workaround was required. I did not change the package layout, the Python code, pip, or the packaging configuration.
The editable installation behaved normally, the command line entry point worked outside the repository, and the full test suite passed.
Recreate the virtual environment after moving a Python project
If you relocate a Python repository, it is better to rebuild its virtual environment instead of moving the existing .venv directory with it.
Virtual environments can contain absolute paths that refer to their original location.
So after moving a project out of iCloud Drive, create the environment again:
rm -rf .venv
python -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'
That removes another potential source of confusing path related behavior.
This may explain older Python packaging problems I had on macOS
The more interesting realization came afterwards.
This was not the first time I had run into strange behavior while trying to make a Python project properly installable.
In older projects, editable installs, package discovery and virtual environments had occasionally behaved in ways I could not explain. Eventually I assumed that I had misunderstood something about Python packaging.
Those projects had one important detail in common: they were also developed inside my iCloud synchronized Documents directory.
I cannot prove that iCloud Drive caused every one of those earlier failures. Python packaging has plenty of ways to go wrong without any help from macOS.
But seeing the same repository start working immediately after moving it outside iCloud Drive makes me considerably less suspicious of those old setup.py and pyproject.toml files.
Where I now keep source code on macOS
My separation is now very simple:
~/dev Source code and active software development
~/Documents Documents, presentations, PDFs and other user files
You could just as easily use:
~/Projects
~/src
~/code
The folder name is not important.
The important part is keeping active Git repositories, virtual environments and build environments outside directories managed by iCloud Drive.
Git already provides the right model for synchronizing and versioning source code. GitHub, GitLab, Bitbucket or another remote repository can handle backup and collaboration without adding another filesystem management layer to the development directory itself.
Symptoms that are worth checking if your Python project is in iCloud Drive
If you develop Python software on macOS and something feels inexplicably inconsistent, first check where the repository actually lives.
pip install -e .succeeds, but the package cannot be imported- editable Python installs behave inconsistently
UF_HIDDENfilesystem flags appear unexpectedly- Python virtual environments behave differently after synchronization
.pthfiles are not processed as expected- package discovery fails on macOS
- files unexpectedly change state
- development tools behave differently inside
~/Documentsor~/Desktop
Before spending hours debugging pip, setuptools, virtualenv, package imports or pyproject.toml, try moving the repository into a normal local development directory and recreating the virtual environment.
Sometimes the most reliable development environment is simply the one with the least interesting filesystem.
TL;DR
If a Python project stored in an iCloud Drive synchronized folder on macOS shows strange problems with pip install -e, editable installs, virtual environments or package discovery, move it outside iCloud Drive and rebuild the virtual environment.
mkdir -p ~/dev
mv ~/Documents/Projects/my-python-project ~/dev/
cd ~/dev/my-python-project
rm -rf .venv
python -m venv .venv
source .venv/bin/activate
pip install -e '.[dev]'
In my case, that was the complete fix.