Add --json/jsonl and --params options by ninoseki · Pull Request #120 · urlscan/urlscan-cli

ninoseki · 2026-03-09T07:35:52Z

This PR adds --json/jsonl (JSON payload) and --params (query parameters) for making this CLI is more agent frendly. (ref. https://justin.poehnelt.com/posts/rewrite-your-cli-for-ai-agents/)

I think this is good for human too. A user can send an arbitary request even if we forget adding a new flag so.

How It Works

--json/--params is mapped as extra field in a struct
Use extra field value to orverride data to be sent (JSON body, query parameters)

wip wip

fw42 · 2026-03-09T15:24:55Z

Hmm there are 12 identical implementations of MarshalJSON in this PR. Can we not pull that into a helper function somehow?

fw42

Hmm I don't know if this is really an improvement. What is the motivation for this PR? Did anyone request this feature? I feel like if you're using --json, you might as well just use curl. This seems to add some extra complexity and quite a bit of extra code (this PR is over 1000 lines of new code) for a small edge case.

fw42 · 2026-03-09T15:29:21Z

+	}
+}
+
+func TestChannelOptionsMarshalJSON_WithoutExtra(t *testing.T) {


This test seems a bit unnecessary. It's already tested implicitly through the other one, isn't it?

fw42 · 2026-03-09T15:30:27Z


 	// size (number of results per batch) is "limit" in this API endpoint
+	// size is an important parameter for iteration
+	// so pick it up from extra if it's given


Hm why is that needed if we already have size and limit?

size (limit in the hostname API) is an important param for iterating :

t.HasMore = len(r.Results) >= it.size

If it's given from --params, the iterator needs to pick it up from there.

What I meant is: Why do we have to be able to extract size from the new extra field if we already have a --size parameter that could be used? I thought this was all to give access to fields that we haven't implemented yet (but size is already implemented, so why do we need to be able to also use it via the extra thing)?

fw42 · 2026-03-09T15:31:40Z

-	OverrideSafety *string   `json:"overrideSafety,omitempty"`
-	Country        *string   `json:"country,omitempty"`
+	URL            string         `json:"url"`
+	CustomAgent    string         `json:"customagent,omitempty"`


Why all the changes from *string to string?

Oops, I have to comment that.
I noticed that using pointer against strings in the struct is not meaningful.

type A struct { foo *string `json:"foo,omitempty"` } type B struct { foo string `json:"foo,omitempty"` }

Both produce {} in Go so.

Also using pointer in the struct is inconsistent since other JSON related structs don't use pointer.

fw42 · 2026-03-09T15:33:25Z

+}
+
+func AddParamsFlag(cmd *cobra.Command) {
+	cmd.Flags().String("params", "", "Query string parameters as JSON (e.g. '{\"key\":\"value\"}')")


Why as JSON? Seems a bit unintuitive. Why not -params "foo=bar&bla=blabla" 🤷‍♂️

I referenced gws and it has --params <JSON> URL/Query parameters as JSON flag. Also I've used a use the same way like this and don't think it's so strange.

fw42 · 2026-03-09T15:34:19Z

+}
+
+func AddJSONLFlag(cmd *cobra.Command) {
+	cmd.Flags().String("jsonl", "", "JSONL payload to send as request bodies (one JSON payload per line)")


Hm how do I use this command? Do I have to pass a string with multiple lines as parameter? like -jsonl "{ "foo": "bar" }\n{"bla": "bla"}\n"?

Yes that's right.

Yes that's right.

I'm confused. In the docs below it sounds like the JSONL payload comes from either a file that's being read or from STDIN. But here you just said that the payload is being given as a command line argument. Which one is it?

fw42 · 2026-03-09T16:35:45Z


-var createCmdExample = `  urlscan pro channel create -n <name>`
+var createCmdExample = `  urlscan pro channel create -n <name>
+  urlscan pro channel create --json '{"channel":{"name":"...","type":"webhook","webhookURL":"https://..."}}'`


Hm adding it to the example like this kind of makes it look like that this is the recommended way that we want people to use this tool. Whereas it's more meant like a last resort in case we forget to implement something, right?

Co-Authored-By: Claude <noreply@anthropic.com>

fw42

I'm not convinced this PR is an improvement. Seems like a lot of extra code and additional complexity. I'll leave it up to you to decide whether to merge it though.

fw42 · 2026-03-18T14:41:17Z


 	// size (number of results per batch) is "limit" in this API endpoint
+	// size is an important parameter for iteration
+	// so pick it up from extra if it's given


What I meant is: Why do we have to be able to extract size from the new extra field if we already have a --size parameter that could be used? I thought this was all to give access to fields that we haven't implemented yet (but size is already implemented, so why do we need to be able to also use it via the extra thing)?

fw42 · 2026-03-18T14:56:49Z

 		}
+		if json != nil {
+			opts = append(opts, api.WithIncidentExtra(json))
+		} else {


Am I reading this right that the new --json command and the existing command flags can't be combined? If there is a json then all the other commands will just be ignored?

fw42 · 2026-03-18T15:01:49Z

+}
+
+func AddJSONLFlag(cmd *cobra.Command) {
+	cmd.Flags().String("jsonl", "", "JSONL payload to send as request bodies (one JSON payload per line)")


Yes that's right.

I'm confused. In the docs below it sounds like the JSONL payload comes from either a file that's being read or from STDIN. But here you just said that the payload is being given as a command line argument. Which one is it?

fw42 · 2026-03-18T15:02:11Z

-  urlscan scan bulk-submit list_of_urls.txt <url>`
+  urlscan scan bulk-submit list_of_urls.txt <url>
+  # submit with JSONL file where each line is a JSON payload
+  urlscan scan bulk-submit --jsonl payloads.jsonl


Where is this loading for reading .jsonl files? I don't see it

ninoseki added 7 commits March 9, 2026 15:48

feat: support --json/jsonl && --params

326519f

wip wip

refactor: show usage if there is no input

47cb2e9

refactor: remove meaningless error

984b21c

fix: handle url correctly

8d8e9e3

fix: fix wrong description

61d48dd

docs: sync docs

360f7a6

fix: remove wrongly added func

b0b70e2

ninoseki requested review from cdnsyseng and fw42 March 9, 2026 07:35

fw42 reviewed Mar 9, 2026

View reviewed changes

refactor: marshalJSONWithExtra

79ea626

Co-Authored-By: Claude <noreply@anthropic.com>

ninoseki requested a review from fw42 March 17, 2026 02:35

fw42 reviewed Mar 18, 2026

View reviewed changes

ninoseki marked this pull request as draft April 10, 2026 04:52

Conversation

ninoseki commented Mar 9, 2026

How It Works

Uh oh!

fw42 commented Mar 9, 2026

Uh oh!

fw42 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fw42 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants