Curl with Partial Files

73 points by ingve 11 days ago | 19 comments

oneeyedpigeon 11 days ago |
This is powerful stuff, but I can't help feeling like it's adding unnecessary bloat to an already-enormously complicated command. What is the advantage of:
curl --variable "pwd@secret;[0-31]" \ --expand-user daniel:{{pwd}} \ https://example.com/
over, e.g.
curl --user daniel:`command-to-fetch-password` \ https://example.com/
? That command may be as simple as an existing one, like cut.
rubenv 11 days ago |
The latter one shows your password in the process listing.
LetMeLogin 11 days ago |
First one will show how to get it :D
skywhopper 11 days ago |
… which is not accessible to as many users as the process list is.
bbarnett 8 days ago |
And can often be passed in bash via a filehandle, without a file.
bandie91 8 days ago |
`mount /proc -o hidepid=1` recommended :)
skywhopper 11 days ago |
Not all places you might run curl allow for shell tricks like process substitution or perhaps for subprocesses at all.
djbsbdb 11 days ago |
The main benefit seems to be context dependent quoting which is tricky to get right and in most cases (like in your example) omitted completely
mdaniel 11 days ago |
I'm sure this solves someone's problem, but reading https://everything.curl.dev/cmdline/variables.html makes me wonder whether curl really needs this added complexity
cobbzilla 7 days ago |
Folks using range requests will certainly welcome these as a great improvement. Without them, you need some ugly shell around your curl command to do it right, sending headers manually and slicing files in the shell.
And, if you don’t need them, don’t use them! Have you seen how many options curl has? But never good ones to support range requests until now.
mdaniel 7 days ago |
I'm _pretty sure_ you're referring to `-C - -o $filename` and not this behavior. This one allows slurping ranges of local files and sending those in their entirety. I am open to maybe one day it does something crazypants with `-X PATCH --variable "fred@file[867-5309]"` or such but I doubt gravely that it does any such thing today
As shown here:
$ python3 -m http.server 9090 & # that doesn't matter, just return _something_ on :9090 $ ls -l .DS_Store -rw-r--r--@ 1 mdaniel staff 18436 Dec 30 11:30 .DS_Store $ curl -vo .DS_Store -C - http://127.0.0.1:9090/.DS_Store ** Resuming transfer from byte position 18436 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 127.0.0.1:9090... * Connected to 127.0.0.1 (127.0.0.1) port 9090 > GET /.DS_Store HTTP/1.1 > Host: 127.0.0.1:9090 > Range: bytes=18436- > User-Agent: curl/8.7.1 > Accept: */*
One can see where curl helpfully loaded the local file's 18436 size, set up the Range header, and included it on the first-contact with the http server.
While digging into this, there's actually one for its modification time, too. Curl knows all:
$ curl -v --time-cond .DS_Store -o .DS_Store -C - http://127.0.0.1:9090/.DS_Store > If-Modified-Since: Mon, 30 Dec 2024 19:30:50 GMT
As for your "well, just don't use it then" stance, it has been my experience in building software that the Law of Unintended Consequences is brutal. The first bug report filed where someone discovers that a `--header` argument cannot contain {{ anymore due to a fat-fingered function change will let me put another quarter in the I Told You So jar
JohnMakin 7 days ago |
> Without them, you need some ugly shell around your curl command to do it right, sending headers manually and slicing files in the shell.
This seems much more readable than the stuff introduced here, and I would still heavily prefer to do it this way.
RestartKernel 11 days ago |
What is the use-case for this feature? I get specifying byte ranges for parsing headers and whatnot, but I don't know a curl workflow that benefits from this.
trws 11 days ago |
The first one that comes to mind is something I haven’t done lately but used to do constantly, enough so that I wrote a custom fuse filesystem to combine isofuse with http handling, and that’s parsing an index from something with a range request then fetching only the part or parts I want. Think fetching only the trailing index from a zip archive and then fetching the specific file you want out of a 10tb zip rather than downloading the whole thing. Now, how often would I do this on the command line? Not sure, but I might not have bothered with fuse if I’d had the option.
mdaniel 11 days ago |
Unless I'm gravely misunderstand this feature currently, it won't help that use case as the variable assignment only supports string literals, env cars, or existing files which wouldn't help your range request for the remote file
It would be a natural extension to treat the @ syntax as a uri (because, come on, it's curl!) if it contains a ":" but even that gets hairy because the next layer of indirection would be --variable-with-curlrc to allow setting the enormous number of parameters one would want when chasing the interior uri
raldu 8 days ago |
Pausing and resuming downloads, and fetching bytes in splits would turn curl into a download manager like aria2.
benatkin 8 days ago |
This could be used to send a single file in a tar. Nice!
DrewRWx 8 days ago |
Nice! This will make my podcast blog to RSS generator a whole lot easier since I just need to grab some headers from the MP3 in addition to info the the page.
alp1n3_eth 7 days ago |
It's nice to see, but the extension on top of the cURL preexisting syntax is rough to look at.
Ever since stumbling upon Hurl a while ago, I've really enjoyed how easy to read the scripts are. Anyone that partially understands web/http is able to read and tell exactly what's going on in the low -> medium complexity scripts w/o even reading the docs.