* Clean warnings for each file in comemod cli
* Fix ZeroDivisionError: float division by zero
When codemodding too fast
* Recreate CodemodContext for each file
Keep only context.metadata_manager
Remove wrapper from context defaults on each file
#453 fixed scratch leaking between files by setting it to empty, but that drops all the scratch space that was set up before the codemod runs (e.g. in the transformer's constructor)
This PR improves the fix by preserving the initial scratch.
* Enumeration members are singletons. Copying on them would be no-op
* Avoid generating unnecessary `pass` statement
* Several trivial refactor
* Avoid building unnecessary intermediate lists, which are mere slight waste of time and space
* Remove unused import, an overlook from commit 8e6bf9e9
* `collections.abc.Mapping.get()` defaults to return `None` when key doesn't exist
* Just use unittest's `assertRaises` to specify expected exception types, instead of catching every possible `Exception`s, which could suppress legitimate errors and hide bugs
* We know for sure that the body of `CSTTypedTransformerFunctions` won't be empty, so don't bother with complex formal completeness
* Always compute a module and package name
* Update name_provider to correctly support __main__ (also updated the tests to use data_provider)
* Update name_provider to correctly handle relative imports and package name
* Update relative module resolution to work on package names
* Use full_package_name in libcst.codemod.visitors.GatherImportsVisitor
* Use full_package_name in libcst.codemod.visitors.RemovedNodeVisitor
* Use full_package_name in libcst.codemod.visitors.AddImportsVisitor
* Fix failing test
* Fix typo in variable name
* PR feedback
* Force rebuild
* Support module and package names in the codemod context
* PR feedback
* Reorganize module name and relative name logic to libcst.helpers.module
* Force rebuild
which ensures we won't have inconsistent black-vs-isort errors
going forward. We can always format by running `ufmt format .`
at the root, and check with `ufmt check .` in our CI actions.
Instead of always launching the specified number of jobs or cpu_count, we should take into account how many files need to be processed and how many files are delivered at once to each process.
Note that this is similar to what fixit does, although they don't specify a chunk size.
* Read install requirements from requirements.txt
* read extras_require from requirements-dev.txt
* add requirements-dev.txt to MANIFEST.in
* apply fixes for new version of Black and Flake8
* don't upgrade Pyre
* re-format
It appears we're running out of file handles when running on large repo on OSX due to waiting until all files have been
processed to join/close the subprocesses.
This PR joins/closes them as they finish, resolving the issue.
```
$ python3 -m libcst.tool codemod {codemod_name} {direectory}
Calculating full-repo metadata...
Executing codemod...
Codemodding {file}
Traceback (most recent call last):
File "/Users/rwilliams/src/go/src/github.com/lyft/python-lyft-ingest/venv/lib/python3.6/site-packages/libcst/codemod/_cli.py", line 253, in _parallel_exec_process_stub
OSError: [Errno 24] Too many open files: '{file}'
Failed to codemod {file}
19.80s 24% complete, 01m 02s estimated for 741 files to go...Traceback (most recent call last):
File "/opt/lyft/brew/Cellar/python36/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/opt/lyft/brew/Cellar/python36/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/Users/rwilliams/src/go/src/github.com/lyft/python-lyft-ingest/venv/lib/python3.6/site-packages/libcst/tool.py", line 833, in <module>
main(os.environ.get("LIBCST_TOOL_COMMAND_NAME", "libcst.tool"), sys.argv[1:])
File "/Users/rwilliams/src/go/src/github.com/lyft/python-lyft-ingest/venv/lib/python3.6/site-packages/libcst/tool.py", line 828, in main
return lookup.get(args.action or None, _invalid_command)(proc_name, command_args)
File "/Users/rwilliams/src/go/src/github.com/lyft/python-lyft-ingest/venv/lib/python3.6/site-packages/libcst/tool.py", line 581, in _codemod_impl
repo_root=config["repo_root"],
File "/Users/rwilliams/src/go/src/github.com/lyft/python-lyft-ingest/venv/lib/python3.6/site-packages/libcst/codemod/_cli.py", line 720, in parallel_exec_transform_with_prettyprint
process.start()
File "/opt/lyft/brew/Cellar/python36/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/process.py", line 105, in start
self._popen = self._Popen(self)
File "/opt/lyft/brew/Cellar/python36/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/opt/lyft/brew/Cellar/python36/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/context.py", line 277, in _Popen
return Popen(process_obj)
File "/opt/lyft/brew/Cellar/python36/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/opt/lyft/brew/Cellar/python36/3.6.5_1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/popen_fork.py", line 65, in _launch
parent_r, child_w = os.pipe()
OSError: [Errno 24] Too many open files
```
