Unable to parse Devfile with parent on GitHub: `url path to directory or file should contain 'tree' or 'blob'` if parent is a GitHub release artifact, or `failed to switch repo to revision` if parent is a `blob` or `raw` URL

[rm3l]

Which area this feature is related to?

/kind bug

Which area this bug is related to?

/area library

What versions of software are you using?

Go project

Operating System and version:
Fedora 38

Go Pkg Version:
Latest commit on the main branch: devfile/library@f041d79

Bug Summary

Describe the bug:

In redhat-developer/odo#6768, we have updated the Devfile library to the latest commit. And now, some of our tests are failing due to Devfile parsing issues.

To Reproduce:
I've narrowed down this issue to the following reproduction steps:

1. Sample Devfile

$ cat << EOF > /tmp/devfile-issue-with-parent-uri.yaml
schemaVersion: 2.1.0
metadata:
  name: new-ol-version
  description: Child devfile to test OL image updating
parent:
  uri: https://github.com/OpenLiberty/application-stack/releases/download/maven-0.7.0/devfile.yaml
  components:
    - name: dev
      container:
        image: maven:3.6-adoptopenjdk-11-openj9
EOF

2. Parser is now failing (I'm running this from the library repo: https://github.com/devfile/library/blob/main/main.go)

$ go run .  /tmp/devfile-issue-with-parent-uri.yaml        

parsing devfile from /tmp/devfile-issue-with-parent-uri.yaml
failed to populateAndParseDevfile: url path to directory or file should contain 'tree' or 'blob'

NOTES:

• I've tried to replace the parent URL with the blob one (https://github.com/OpenLiberty/devfile-stack/blob/maven-0.7.0/templates/devfiles/maven/devfile.yaml), but the parser still fails:

$ go run .  /tmp/devfile-issue-with-parent-uri.yaml
parsing devfile from /tmp/devfile-issue-with-parent-uri.yaml
failed to populateAndParseDevfile: 1 error occurred:
        * failed to switch repo to revision. repo dir: /tmp/git-resources2839940370, revision: maven-0.7.0

• I've also tried to replace the parent URL with the raw one (https://raw.githubusercontent.com/OpenLiberty/devfile-stack/maven-0.7.0/templates/devfiles/maven/devfile.yaml), but the parser still fails:

$ go run .  /tmp/devfile-issue-with-parent-uri.yaml
parsing devfile from /tmp/devfile-issue-with-parent-uri.yaml
failed to populateAndParseDevfile: 1 error occurred:
        * failed to switch repo to revision. repo dir: /tmp/git-resources3340282075, revision: maven-0.7.0

Expected behavior

• Is it intentional that we cannot reference a parent from a GitHub release artifact?
• I guess however that using the raw or blob URLs should have worked correctly.

Suggestion on how to fix the bug

[mike-hoang]

It seems like using git switch is a little too limiting in that it only works well when switching branches. Will open a PR to change to using git checkout instead.

devfile/library#173

[rm3l]

It seems like using git switch is a little too limiting in that it only works well when switching branches. Will open a PR to change to using git checkout instead.

devfile/library#173

Thanks, @mike-hoang !!
Indeed, git switch expects a branch, but I think it can also work by detaching the working tree at the tag/branch commit; so a command like git switch --detach <branch_or_tag> should also work.
From your diff in https://github.com/devfile/library/pull/173/files, I think the issue is because of the origin/ prefix. There is only one remote after the repo was cloned, so Git should find the right remote branch/tag.

Another point about error handling: would it be possible to wrap the original error of the git commands executed by the parser, to help debug such errors in the future?

[rm3l]

Is it intentional that we cannot reference a parent from a GitHub release artifact?

Back to the original issue, the Devfile in our tests contains a parent referenced via a GitHub release artifact URL.
Is this not supposed to work anymore?

[mike-hoang]

so a command like git switch --detach <branch_or_tag> should also work.

When testing, I ran into some issues where checkout would still work:

➜  devfile-stack git:(fcee161) git switch --detach master
fatal: '--detach' cannot be used with '-b/-B/--orphan'

➜  devfile-stack git:(fcee161) git checkout master
Previous HEAD position was fcee161 Merge pull request #166 from mezarin/portDevfileRegDevfileUpdatesToStack
branch 'master' set up to track 'origin/master'.

So I think checkout would be a better option here.

Another point about error handling: would it be possible to wrap the original error of the git commands executed by the parser, to help debug such errors in the future?

Yup I can wrap the git error

Back to the original issue, the Devfile in our tests contains a parent referenced via a GitHub release artifact URL.
Is this not supposed to work anymore?

Yes, with the recent changes this will not work anymore. I wasn't aware that release artifact urls were being used, maybe this can be raised in the community call?

[rm3l]

Back to the original issue, the Devfile in our tests contains a parent referenced via a GitHub release artifact URL.
Is this not supposed to work anymore?

Yes, with the recent changes this will not work anymore. I wasn't aware that release artifact urls were being used, maybe this can be raised in the community call?

Yes, why not! I'll add it to the agenda.

I wasn't aware of that fact either, until I stumbled upon this in our tests :-)

At first, I thought we were not using the uri field correctly, but the spec and doc say:

So, unless I'm missing something from the docs/spec, it looks like it should be possible to use any hosted devfile.yaml as parent; and if the URI points to a supported Git repo, the library will clone it.

I tried to workaround this issue by serving the Devfile from a test HTTP server, but the parser was expecting a GitHub, Gitlab or BitBucket host.

# With this URI: http://localhost:5000/devfile.yaml

$ go run .  /tmp/devfile-issue-with-parent-uri.yaml 
parsing devfile from /tmp/devfile-issue-with-parent-uri.yaml
failed to populateAndParseDevfile: url host should be a valid GitHub, GitLab, or Bitbucket host; received: localhost:5000

[mike-hoang]

So, unless I'm missing something from the docs/spec, it looks like it should be possible to use any hosted devfile.yaml as parent; and if the URI points to a supported Git repo, the library will clone it.

Yes, this should be the correct behaviour 😄
Updated the pr to clone only when it is a git repo.