1

The command below cats a file full of urls and feeds it into gnu parallel. I would like the output in curlresponse.txt to have also the original request url. How can it be done?

cat urls.txt  | parallel -j 5  curl -IL -k {} > curlresponse.txt

so sample of final output (after all redirects if any) is like this :

OriginalUrl
HTTP/1.1 200
Server: nginx
Date: Mon, 18 Mar 2019 07:35:11 GMT
Content-Type: text/html
Content-Length: 188
Connection: keep-alive

Sometimes the url might be redirected so the final http response should be the one having the original request url.

Improve this question

1 Answer 1

Reset to default
0

Strictly:

cat urls.txt  |
  parallel -j 5 echo {}';'curl -IL -k {} '|' perl -ne '/^HTTP.* 200/ .. 0 and print' > curlresponse.txt

But for situations like this I usually prefer --tag which will prepend every line with the input:

cat urls.txt  |
  parallel -j 5 --tag curl -IL -k {} '|' perl -ne '/^HTTP.* 200/ .. 0 and print' > curlresponse.txt

It often makes further processing easier.

Improve this answer
0

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.