feat(wasm-builder): New approach to build demos by ark0f · Pull Request #3602 · gear-tech/gear

ark0f · 2023-12-17T20:20:43Z

This is a new approach to build Gear programs which can replace gear-wasm-builder.

Schematic architecture:

Pipeline:

Purpose of program build script is just write features to lock file and don't use any cargo:rerun-if instructions so default cargo heuristics are enabled.

Binaries crate build script list its build dependencies and corresponding reads lock files to obtain program features then it runs cargo build -p program1 -p program2 so cargo builds WASM binaries effectively and concurrently. Then it writes to lock files to skip cargo invocation in next or in another binaries crate to avoid few seconds cache verification

Pros:

Advanced feature tracking.
Features are tracked automatically so we don't need to re-run build script each time like in gear-wasm-builder (can be mitigated by cargo-metadata and Cargo.toml tracking)
Faster build.
gear-wasm-builder builds program via new cargo command in the same target directory so each cargo process blocks other. wasm-dep-builder collects list of programs so only one cargo process runs.

Cons:

Create and import binaries crate.
It is important to create binaries create that will list and build programs

Current problem:

When crate includes itself as dev dependency (any integration tests) with different feature set then cargo build and runs 2 different versions of build script - for workspace and for tests.

gear-wasm-builder case is that cargo builds and runs script just once with merged feature set.

So, how to resolve these features? Should we deliver different WASM binaries? Or just merge feature sets?
I vote for merging but we need some mechanism to obtain all of feature sets cargo resolved for each build script invocation of program before build script invocation of binaries crate if we don't want to waste time on WASM build with incomplete feature set

clearloop · 2023-12-19T13:48:58Z

plz check this #3588 (comment) btw

gshep

Good job! 🥇

gshep · 2024-02-02T06:46:18Z

@@ -0,0 +1,21 @@
+// This file is part of Gear.
+//
+// Copyright (C) 2023 Gear Technologies Inc.


I suggest removing the year number from the copyright since it is not consistent across the source code and we will not update it later

I see most of files have year when just a few don't

StackOverflowExcept1on · 2024-02-05T19:34:18Z


 //! Module for memory pages.

+#![allow(clippy::assertions_on_constants)]


merge master and remove this

StackOverflowExcept1on · 2024-02-05T20:08:23Z

@@ -0,0 +1 @@
+


why examples/signal-wait/** is empty?

DennisInSky · 2024-02-06T10:41:47Z

 gear-stack-buffer = { path = "stack-buffer" }
 gear-utils = { path = "utils/utils" }
 gear-wasm-builder = { path = "utils/wasm-builder", default-features = false }
+wasm-dep-builder = { path = "utils/wasm-dep-builder" }


Do we want to prefix it with gear as for the others?

DennisInSky · 2024-02-06T10:43:48Z

+[dependencies]
+gcore.workspace = true
+parity-scale-codec.workspace = true
+gstd = { workspace = true }


Suggested change

gstd = { workspace = true }

gstd.workspace = true

DennisInSky · 2024-02-06T11:02:51Z

+
+[dependencies]


DennisInSky · 2024-02-06T11:09:56Z

+
+# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html


DennisInSky · 2024-02-06T11:24:17Z

+
+# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html


DennisInSky · 2024-02-06T11:43:52Z

 gmeta = { workspace = true }
 gstd = { workspace = true, features = ["debug"] }
 demo-wat.workspace = true
+gclient-binaries = { path = "binaries" }


Would it be better to call it demo-binaries?

DennisInSky · 2024-02-06T11:54:07Z


 [dev-dependencies]
 gsdk = { path = ".", features = ["testing"] }
+gsdk-binaries = { path = "binaries" }


-> demo-binaries?

DennisInSky · 2024-02-06T11:55:09Z

-demo-ping.workspace = true
-demo-futures-unordered.workspace = true
 demo-meta-io.workspace = true
+gtest-binaries = { path = "binaries" }


-> demo-binaries?

DennisInSky · 2024-02-06T11:56:21Z

 prometheus-endpoint.workspace = true

 [dev-dependencies]
+gear-authorship-binaries = { path = "binaries" }


-> demo-binaries?

DennisInSky · 2024-02-06T12:05:27Z

+repository.workspace = true
+
+[dependencies]
+cargo_metadata = "0.18.1"


Do you want to move them into the workspace deps?

ark0f · 2024-02-07T14:58:34Z

cargo check may be failed because of missing .lock. Investigating...

NikVolf · 2024-02-25T16:17:18Z

Can you describe the approach in the PR outline?

NikVolf · 2024-02-26T07:57:28Z

Can we maybe put "dependency builder" inside gear-wasm-builder itself?

breathx · 2024-02-26T16:12:51Z

Can we maybe put "dependency builder" inside gear-wasm-builder itself?

It will replace gear_wasm_builder some day, so is there any reason to merge them now?

Just for notice: as it was mentioned, there are a few problems with these approach, and as far as we discussed with @ark0f they're already solved, but not pushed

ark0f · 2024-03-12T18:14:09Z

Can you describe the approach in the PR outline?

Done

breathx · 2024-05-06T08:59:02Z

Temporary closing in favour of opened referencing issue as won't be implemented

ark0f added 7 commits December 13, 2023 03:16

Initial commit

576b56f

Add tracing based on cargo and lock file

b9e51b9

Use wasm-dep-builder in pallet-gear

45c2d40

Use wasm-dep-builder in all pallet-gear demos

5b4b12f

Build packages in workspace

53c2744

Fix cargo config is in wrong folder

1526bf1

Optimize binaries

df03809

ark0f added A1-inprogress labels Dec 17, 2023

Merge origin/master

2384b97

ark0f added 19 commits December 25, 2023 00:34

Merge remote-tracking branch 'origin/master' into al/no-wasm-builder

c7e9636

Fix missing wasm_binary.rs file error appears sometime

84de8f3

Fix some demos are not rebuilt

c13b7f6

Make demo imports like before

ce5d39d

Make pallet-gear-binaries crate optional

fb310d6

Follow new approach in test-syscalls

2313d74

Make build command persistent

6dfbc83

Rename WASM binary consts

03ffa6e

Remove test-wasm-dep

6f3a50c

Introduce debug feature in binaries crate

fb5f2e9

Advanced feature tracking

0a2e664

Introduce LockFile

297513c

Introduce from_value constructors

96882c9

Remove obsolete environment variable

78f0d44

Make features tracking persistent

58f6b47

Refactor builder

4e98b52

Fix clippy warnings

0c850e6

Revert benchmarks.rs in fungible-token

3ee9e29

Use wasm-dep-builder in gtest

c3e0a2f

Merge remote-tracking branch 'origin/master' into al/no-wasm-builder

1c2dde2

gshep approved these changes Feb 2, 2024

View reviewed changes

ark0f added 7 commits February 5, 2024 18:57

Merge remote-tracking branch 'origin/master' into al/no-wasm-builder

42a7a33

Build in its crate's own directory

cda85fb

Rename read -> read_any for BinariesLockFileConfig

9f45efd

Remove debug warnings

54fc69a

Copy binaries to default directory

ae89a05

Fix unused imports warning

9e59065

Update copyright

943d9e4

ark0f requested review from DennisInSky, StackOverflowExcept1on, breathx and clearloop February 5, 2024 16:37

StackOverflowExcept1on reviewed Feb 5, 2024

View reviewed changes

Comment thread examples/signal-wait/src/lib.rs

@@ -0,0 +1 @@

StackOverflowExcept1on Feb 5, 2024

Copy link
Copy Markdown

Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why examples/signal-wait/** is empty?

StackOverflowExcept1on approved these changes Feb 5, 2024

View reviewed changes

DennisInSky approved these changes Feb 6, 2024

View reviewed changes

ark0f added A5-dontmerge and removed A5-dontmerge labels Feb 7, 2024

breathx added the A5-dontmerge label Feb 26, 2024

ark0f mentioned this pull request Mar 25, 2024

feat(wasm-builder): Another approach to build demos #3837

Closed

breathx mentioned this pull request May 6, 2024

Consider options of redesigning building approach (deprecate gear-wasm-builder) #3945

Closed

breathx closed this May 6, 2024


		//! Module for memory pages.

		#![allow(clippy::assertions_on_constants)]


		# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html


		[dependencies]

Conversation

ark0f commented Dec 17, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clearloop commented Dec 19, 2023

Uh oh!

gshep left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ark0f commented Feb 7, 2024

Uh oh!

NikVolf commented Feb 25, 2024

Uh oh!

NikVolf commented Feb 26, 2024

Uh oh!

breathx commented Feb 26, 2024

Uh oh!

ark0f commented Mar 12, 2024

Uh oh!

breathx commented May 6, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

ark0f commented Dec 17, 2023 •

edited

Loading