Remove NPM as a dependency, use the NPM installed on users' machine

[ErjanGavalji]

I am copying here the discussion from #1740, for it deserves a separate thread...

[ErjanGavalji]

By @enchev:
Hey @rosen-vladimirov,

Can you post here more info why we have our own nmp and why we do not use the instilled one. What problems we solved?

Thanks

[ErjanGavalji]

By @rosen-vladimirov:
Hi @enchev ,
As you are asking, using npm as dependency resolves exactly this issue. In case we were relying on user's npm, you wouldn't have any workaround for this particular problem and in fact nativescript-cli would have been unusable with npm 3. And btw this is not "our" npm, it's the real npm, we are just using it as dependency.

How does npm work

NativeScript projects rely on npm and npm's structure. This means that each project has package.json with dependencies and devDependencies. When you call npm install all of your dependencies and devDependencies will be installed and placed in your node_modules directory. Different npm versions place the dependencies in a different way inside node_modules dir.
npm versions 2.x.x will install each dependency in it's own directory and each of it's dependencies will be installed inside it's node_modules directory. All dependencies of devDependencies will be installed in the respective node_modules dir.
npm versions 3.x.x tries to flatten the dependencies and install everything at your project's node_modules directory. So for example in case devDependency A has dependency B, which has dependency C, at the root node_modules you'll receive A, B and C directories.

How does NativeScript CLI handle dependencies

Due to known behavior of NativeScript's require, all dependencies used in a project must be flattened when copied to platforms/<platform>/.../tns_modules directory. So when the project is prepared, CLI tries to flatten the dependencies from node_modules dir, filters the one which are devDependencies or dependencies of devDependencies or dependency of dependency of dependency ... of devDependencies and process them to tns_modules dir inside platforms.

The problem

When npm 3.x.x is installed and user executes npm install, the flattening of modules makes it really hard for NativeScript CLI to understand from where is each package inside node_modules dir. It would require parsing all package.json files inside node_modules and it's subdirectories, checking each package and is it a dependency or devDependency, etc.

The workaround

NativeScript CLI has npm 2.x.x as dependency and executes npm install with it. This allows us to have the "old" way of installing the dependencies, where it is much easier to filter the devDependencies.

Why npm as dependency

In case we rely on user's npm version we have to:

• use child_process methods for spawning npm. Spawning new process is slower than simple require
• Parse stdout, stderr outputs of the newly spawned process. When we use the dependency, we work with callbacks directly where we have the result and error in a JSON objects.
• Your code should be usable with all possible versions of npm. However npm has breaking changes between different versions, for example some methods accept additional arguments in new versions. How would you handle this in your code?

[ErjanGavalji]

by @hdeshev:

Dragging our own pinned version of npm is a workaround that seems to work for many cases, but it's nothing more than a workaround. I believe we have to get rid of it -- the breakage caused by a command such as npm install (which is perfectly fine in any node project) is a real show stopper.

I'm probably oversimplifying everything, but can we rely on the user running npm install and use different module resolution strategies to identify modules we need to copy/update to <platform dir>/app/tns_modules?

[ErjanGavalji]

by @rosen-vladimirov:

Please excuse me, but relying on a specific version of a library is not a workaround for me. In the same manner we can drop all of our dependencies and ask the users to install them globally.

I agree that npm install is normal process and shouldn't break the project. We've already discussed this with @ErjanGavalji and he is working on a new approach that should resolve the issue with detecting which modules should be moved to tns_modules.

Also we cannot rely on user's calling of npm install - the idea was to have an easy way to do:

$ tns create app1
$ cd app1
$ tns run android

That's why CLI calls npm install on it's own. Also when preparing the project, we want to be sure everyhing is installed, so the user should not take care of this.
In case you decide that you want the users to call npm install, all of this will stop working.

[ErjanGavalji]

I am fine having a dependency to npm for the pure reason of not having to execute a side process.

However, having a strict dependency makes us lazy. The CLI does not evolve with the rest of the framework because we do not dedicate enough resources on keeping it modern by updating its dependencies. This is especially valid for complex dependencies such as npm, which is a giant tool by itself.

While it is possible that the teams specify a recurring task for updating the dependencies, I think it would be better that the necessity of an update come naturally.

Thus, I vote that we call the npm instance that is installed on customer machine. I'd add additional verification via builds (Travis-CI?) that will produce a list of npm/nodejs versions NativeScript CLI works with properly.

[hdeshev]

The point I tried to make with my previous post is that we have no choice but to support the scenario where the end user runs an npm install command by him/herself. This is the way npm projects work, and people are used to doing that.

So, here we are having to support npm 2 and npm 3 anyway (since we don't want to force users into using a specific version of npm). Why bundle npm 2 then? I'm afraid it could mask issues that would normally be discovered earlier during internal testing.